popen question

[Robert Latest]

Hello,

look at this function:

--------------
def test():
child = os.popen('./slow')
for line in child:
print line
-------------

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

Thanks,

robert

 

[Marc 'BlackJack' Rintsch]

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

Both processes have to make their communication ends unbuffered or line
buffered. See the documentation of `os.popen()` for the `bufsize`
argument. And do whatever is needed to output the numbers from ``slow``
unbuffered or line buffered.

Ciao,
Marc 'BlackJack' Rintsch

 

[Robert Latest]

Both processes have to make their communication ends unbuffered or line
buffered.

Yeah, I figured something like that.

And do whatever is needed to output the numbers from ``slow``
unbuffered or line buffered.

Hm, "slow" of course is just a little test program I wrote for this purpose.
In reality I want to call another program whose behavior I can't influence
(well, technically I could because it's open-source, but let's assume it to
be a black box for now).

If 'slow' or some other program does buffered output, how come I can see
its output line-by-line in the shell?

robert

 

[Hrvoje Niksic]

If 'slow' or some other program does buffered output, how come I can
see its output line-by-line in the shell?

stdio uses different buffering strategies depending on the output
type. When the output is a TTY, line buffering is used; when the
output goes to a pipe or file, it is fully buffered.

In reality I want to call another program whose behavior I can't
influence (well, technically I could because it's open-source, but
let's assume it to be a black box for now).

To test whether your black box buffers output to pipe, simply start it
like this:

$ ./slow | cat

If you see lines one by one, you are in luck, and you can fix things
on the Python level simply by avoiding buffering in popen. If not,
you will need to resort to more advanced hackery (e.g. fixing stdio
using LD_PRELOAD).

 

[Robert Latest]

stdio uses different buffering strategies depending on the output
type. When the output is a TTY, line buffering is used; when the
output goes to a pipe or file, it is fully buffered.

Makes sense.

If you see lines one by one, you are in luck, and you can fix things
on the Python level simply by avoiding buffering in popen. If not,
you will need to resort to more advanced hackery (e.g. fixing stdio
using LD_PRELOAD).

Do I really? After all, the shell itself doesn't hack stdio, does it?
Anyway, I'm taking this over to comp.unix.programmer since it really isn't a
python problem.

Thanks,
robert

 

[Robert Latest]

pexpect is the solution. Seems to wrap quite a bit of dirty pseudo-tty
hacking.

robert

 

[Hrvoje Niksic]

Do I really? After all, the shell itself doesn't hack stdio, does
it?

Have you tried the "./slow | cat" test? It's not about the shell
doing anything special, it's about slow's stdio choosing buffering
"appropriate" for the output device. The hackery I referred to would
be necessary to force slow's stdio to use line buffering over
file/pipe output simply because there is no documented way to do that
(that I know of).

Anyway, I'm taking this over to comp.unix.programmer since it really
isn't a python problem.

It's still an often-encountered problem, so I believe a summary of the
solution(s) would be appreciated.

 

[Karthik Gurusamy]

Hello,

look at this function:

--------------
def test():
child = os.popen('./slow')
for line in child:
print line
-------------

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

I've seen this problem and it took me a while to figure out what is
happening.
As other posts, I too first suspected it's a problem related to line/
full buffering on the sender side (./slow here).

It turned out that the "for line in child:" line in the iterator is
the culprit. The iterator does something like a child.readlines()
underneath (in it's __iter__ call) instead of a more logical single
line read.

Change your reading to force line-by-line read
e.g.
While True:
line = child.readline()
if not line: break
print line

Karthik