Non-blocking pipes during subprocess handling

[Tom Plunket]

I'm using subprocess to launch, well, sub-processes, but now I'm
stumbling due to blocking I/O.

Is there a way for me to know that there's data on a pipe, and possibly
how much data is there so I can get it? Currently I'm doing this:

process = subprocess.Popen(
args,
bufsize=1,
universal_newlines=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

def ProcessOutput(instream, outstream):
text = instream.readline()
if len(text) > 0:
print >>outstream, text,
return True
else:
return False

while process.poll() is None:
ProcessOutput(process.stdout, sys.stdout)
ProcessOutput(process.stderr, sys.stderr)

# clean up everything to EOF once the process ends.
somethingPrinted = True
while somethingPrinted:
somethingPrinted = ProcessOutput(
process.stdout, sys.stdout)
somethingPrinted |= ProcessOutput(
process.stderr, sys.stderr)


Unfortunately, stream.readline will block 'til it gets a line, and
typically there won't be anything on the stderr stream. The reason for
the redirections in the first place is that I'm launching this script as
a subprocess from a GUI app that catches stdout and stderr and directs
the output to the appropriate windows, but in some cases I don't
actually want the output at all (I've removed that logic though since it
needlessly complicates my example; suffice to say everything below the
process = subprocess.Popen... line is enclosed in a try and then in an
if block.

The documentation on file.read() indicate that there's an option for
"non-blocking" mode, but I'm stumped as to how to even look for how to
enable and use that.

thanks,
-tom!

--

 

[Gabriel Genellina]

I'm using subprocess to launch, well, sub-processes, but now I'm
stumbling due to blocking I/O.

Is there a way for me to know that there's data on a pipe, and possibly
how much data is there so I can get it? Currently I'm doing this:

Using a thread for each stream is the safest way, specially if you
can't control the child process.


--
Gabriel Genellina
Softlab SRL






__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

 

[Nick Craig-Wood]

I'm using subprocess to launch, well, sub-processes, but now I'm
stumbling due to blocking I/O.

Is there a way for me to know that there's data on a pipe, and possibly
how much data is there so I can get it?

You might want to check out this modification to subprocess which does
non-blocking pipes.

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

I personally think something like that should be built into subprocess

 

[Donn Cave]

I'm using subprocess to launch, well, sub-processes, but now I'm
stumbling due to blocking I/O.

Is there a way for me to know that there's data on a pipe, and possibly
how much data is there so I can get it? Currently I'm doing this:

process = subprocess.Popen(
args,
bufsize=1,
universal_newlines=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

def ProcessOutput(instream, outstream):
text = instream.readline()
if len(text) > 0:
print >>outstream, text,
return True
else:
return False

I think it would be fair to say that your problem is
not due to blocking I/O, so much as buffered I/O. Since
you don't appear to need to read one line at a time, you
can detect and read data from the file descriptor without
any buffering. Don't mix with buffered I/O, as this will
throw the select off. From memory - better check, since
it has been a while since I wrote anything real like this
(or for that matter much of anything in Python) --


import select
def ProcessOutput(instream, outstream):
fdr = [instream.fileno()]
(r, w, e) = select.select(fdr, [], [], 0.0)
for fd in r:
text = os.read(fd, 4096)
outstream.write(text)

Donn Cave, (e-mail address removed)