line-by-line output from a subprocess

[Chermside, Michael]

I am using the subprocess module to invoke a command-line utility and
process the output.

However, I would like to process the output line-by-line as it is
generated rather than
running the subprocess to completion and THEN processing the results.
So, for instance,
I'd like to write code like this:

all_files = []
import subprocess
subp = subprocess.Popen('dir', stdout=subprocess.PIPE)
for line in subp.stdout:
sys.stdout.print(line)
all_files.append(line)

....and have it print the lines to stdout one-by-one until the entire
list is
collected. (Pretend that 'dir' is VERY slow.)

The communicate() method in subprocess blocks until the subprocess has
exited... I'm sure there is some good reason for this behavior. But how
does
one achieve what I am trying to do?

(PS: this is on Windows... although I'd rather use an OS-agnostic
solution if
one exists.)

-- Michael Chermside






*****************************************************************************
This email may contain confidential or privileged information. If you believe
you have received the message in error, please notify the sender and delete
the message without copying or disclosing it.
*****************************************************************************

 

[Simon Percivall]

Okay, so the reason what you're trying to do doesn't work is that the
readahead buffer used by the file iterator is 8192 bytes, which clearly
might be too much. It also might be because the output from the
application you're running is buffered, so you might have to do
something about that as well.

Anyway, if the output from the child application is unbuffered, writing
a generator like this would work:

def iterread(fobj):
stdout = fobj.stdout.read(1) # or what you like
data = ""
while stdout:
data += stdout
while "\n" in data:
line, data = data.split("\n", 1)
yield line
stdout = fobj.stdout.read(1)
if data:
yield data,

 

[Simon Percivall]

Or, doing the same thing, but with less code:

Hmm ... What have I been smoking?