Something that Perl can do that Python can't?

[Dr. Who]

So here it is: handle unbuffered output from a child process.

Here is the child process script (bufcallee.py):
import time
print 'START'
time.sleep(10)
print 'STOP'

In Perl, I do:
open(FILE, "python bufcallee.py |");
while ($line = <FILE>)
{
print "LINE: $line";
}

in which case I get
LINE: START
followed by a 10 second pause and then
LINE: STOP

The equivalent in Python:
import sys, os

FILE = os.popen('python bufcallee.py')
for line in FILE:
print 'LINE:', line

yields a 10 second pause followed by
LINE: START
LINE: STOP

I have tried the subprocess module, the -u on both the original and
called script, setting bufsize=0 explicitly but to no avail. I also
get the same behavior on Windows and Linux.

If anyone can disprove me or show me what I'm doing wrong, it would be
appreciated.

Jeff

 

[Dr. Who]

Well, I finally managed to solve it myself by looking at some code.
The solution in Python is a little non-intuitive but this is how to get
it:

while 1:
line = stdout.readline()
if not line:
break
print 'LINE:', line,

If anyone can do it the more Pythonic way with some sort of iteration
over stdout, please let me know.

Jeff

 

[Donn Cave]

So here it is: handle unbuffered output from a child process.

Your Perl program works the same for me, on MacOS X,
as your Python program. That's what we would expect,
of course, because the problem is with the (Python)
program on the other end - it's buffering output,
because the output device is not a terminal.

Donn Cave, (e-mail address removed)

 

[George Sakkis]

Well, I finally managed to solve it myself by looking at some code.
The solution in Python is a little non-intuitive but this is how to get
it:

while 1:
line = stdout.readline()
if not line:
break
print 'LINE:', line,

If anyone can do it the more Pythonic way with some sort of iteration
over stdout, please let me know.

Jeff

You can use the sentinel form of iter():

for line in iter(stdout.readline, ''):
print 'LINE:', line,


George

 

[Andrew Dalke]

Well, I finally managed to solve it myself by looking at some code.
The solution in Python is a little non-intuitive but this is how to get
it:

while 1:
line = stdout.readline()
if not line:
break
print 'LINE:', line,

If anyone can do it the more Pythonic way with some sort of iteration
over stdout, please let me know.

Python supports two different but related iterators over
lines of a file. What you show here is the oldest way.
It reads up to the newline (or eof) and returns the line.

The newer way is

for line in stdout:
...

which is equivalent to

_iter = iter(stdout)
while 1:
try:
line = _iter.next()
except StopIteration:
break

...

The file.__iter__() is implemented by doing
a block read and internally breaking the block
into lines. This make the read a lot faster
because it does a single system call for the
block instead of a system call for every
character read. The downside is that the read
can block (err, a different use of "block")
waiting for enough data.

If you want to use the for idiom and have
the guaranteed "no more than a line at a time"
semantics, try this

for line in iter(stdout.readline, ""):
print "LINE:", line
sys.stdout.flush()

Andrew
(e-mail address removed)

 

[jitu]

Your Perl program works the same for me, on MacOS X,
as your Python program. That's what we would expect,
of course, because the problem is with the (Python)
program on the other end - it's buffering output,
because the output device is not a terminal.

Donn Cave, (e-mail address removed)

Yes Donn's right , works the same for me , bufcallee.py may be should
look like
this

import time
import sys

sysout=sys.stdout

sysout.write("START\n")
sysout.flush()
time.sleep(10)
sysout.write("STOP\n")
sysout.flush()

regards
jitu