Reading output from a child process non-blockingly

[Yuan HOng]

In my program I have to call an external program and parse its output.
For that I use the os.popen2 function, and then read the output
stream.

But the complexity is that the external program gives back its output
in a piecemeal manner, with long delays between the outputs. In the
main program I want to therefore read the output in a non-blocking
manner, to read as many bytes as the child process is spitting out.

The question is, how can I achieve that?

I tried use select.select on the output stream returned by os.popen2,
but it returns a readable file descriptor only after the whole child
process ends.

Here is a script simulating the external program:

test.py:
import sys, time
print 'hello\n'*500
sys.stdout.flush()
time.sleep(100)
print 'world\n'*500

And here is what I am tring to do in the main program to read its output:

import os, select
cmd = 'python test.py'
pin, pout = os.popen2(cmd)
while not select.select([pout], [], [], some_timeout)[0]:
pass
pout.readline()

I hope to get the first return very soon, before the external program
sleeps, but instead only after the whole program exits do I get any
output.

Can anyone give me a hint?

 

[ilochab]

Yuan HOng ha scritto:

In my program I have to call an external program and parse its output.
For that I use the os.popen2 function, and then read the output
stream.

But the complexity is that the external program gives back its output
in a piecemeal manner, with long delays between the outputs. In the
main program I want to therefore read the output in a non-blocking
manner, to read as many bytes as the child process is spitting out.

The question is, how can I achieve that?

What about using a thread to control the child process?

Licia

 

[Thomas Guettler]

Am Wed, 29 Jun 2005 16:08:54 +0800 schrieb Yuan HOng:

In my program I have to call an external program and parse its output.
For that I use the os.popen2 function, and then read the output
stream. [cut]
I tried use select.select on the output stream returned by os.popen2,
but it returns a readable file descriptor only after the whole child
process ends.

Select is the right module for this. But it only works on file descriptors
on unix.

What happens, if you run the external command on the shell like this:

ext_prog > out.log &

Check out.log with "tail" or "less +F". Do you see the data appear
in small chunks? If not, the application detects that stdout is not
a tty and you only get data if the buffer is full or if the application
ends.

If out.log gets filled in chunks you want to read, you can read this file
as long as the application is running. Remember the size of the file after
each read and use fd.seek() to read only the new data after opening it
again. This should work on windows, too.

HTH,
Thomas

 

[Dan Sommers]

Check out.log with "tail" or "less +F". Do you see the data appear in
small chunks? ...

You'll need "tail -f", I think.

Regards,
Dan