Non-blocking read with popen subprocess

[Dhanesh]

Hi ,

I am trying to use subprocess popen on a windows command line
executable with spits messages on STDOUT as well as STDIN. Code
snippet is as below :-
##########################################################################
sOut=""
sErr=""
javaLoaderPath = os.path.join("c:\\","Program Files","Research In
Motion","BlackBerry JDE 4.7.0","bin","Javaloader.exe")
cmd = [javaLoaderPath,'-u','load','helloworld.jad']
popen = subprocess.Popen
(cmd,bufsize=256,shell=False,stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
while True:
sErr = sErr + popen.stderr.read(64)
sOut = sOut + popen.stdout.read(64)------------------> Blocks
here
if None != popen.poll():
break;
##########################################################################

I observed that python scripts blocks at stdout read on the other hand
when I do not create a child stdout PIPE say " stdout=None" , things
seems to be working fine.

how can I we have a non blocking read ?
And why does stdout block even where data is available to be read
( which seem to be apparent when stdout=None, and logs appear on
parents STDOUT) ?

 

[Jonathan Gardner]

how can I we have a non blocking read ?

See http://docs.python.org/library/popen2.html#flow-control-issues

Note well: In the non-blocking world, you have to use select() or poll
() to get your job done.

You may want to look at "communicate" (http://docs.python.org/library/
subprocess.html#popen-objects) which may be what you need.

 

[Piet van Oostrum]

JG> See http://docs.python.org/library/popen2.html#flow-control-issues

JG> Note well: In the non-blocking world, you have to use select() or poll
JG> () to get your job done.

AFAIK, on Windows these work only on sockets, not on files. I think
pipes are in the files camp.

Alternatively you can read one of the two in a different thread.

 

[Javier Collado]

Hello,

According to my experience and from what I've read in other threads,
subprocess isn't easy to use for interactive tasks. I don't really
know, but maybe it wasn't even designed for that at all.

On the other hand, pexpect seems to work fine for interactive use and
even provides a method for nonblocking reads, so I think that you
should consider to take a look at it:
http://pexpect.sourceforge.net/pexpect.html#spawn-read_nonblocking

Best regards,
Javier

 

[birdsong]

Seehttp://docs.python.org/library/popen2.html#flow-control-issues

Note well: In the non-blocking world, you have to use select() or poll
() to get your job done.

I solved a problem just yesterday regarding select on a Popen.stdout
file descriptor that was created with a 100Mb buffer. I'd love it if
anybody could critique my solution and my understanding of my
findings.

In my case, select would return with my Popen object ready for read,
I'd go off and try to read and then my read would block. After a few
strace's I found that Popen.stdout.read(), which I am assuming is the
same as the read builtin, would make multiple read system calls, in
many cases blocking until it returned my default buffer size.

Sure enough, it's in the docs: http://docs.python.org/library/stdtypes.html?highlight=fread
"""
Note that this method may call the underlying C function fread() more
than once in an effort to acquire as close to size bytes as possible.
Also note that when in non-blocking mode, less data than was requested
may be returned, even if no size parameter was given.
"""

I couldn't find a way to tell subprocess to return my stdout fd in non-
blocking mode, but then I thought about it not being a problem with
the fd and how it was opened as much as a problem with the read
call.

A non-blocking fd will have a read call throw EWOULDBLOCK when asked
for more bytes than is in buffer, a blocking fd would have read just
return fewer bytes ...do I have this this right?

I can handle assembling the data fine, I just don't want to block.

I wrote a little test program to spit out random stdout to test a
Popen object's stdout in os.read. The len() of the result of os.read
was all over the place depending on how quickly i re-ran read() on the
fd. There were slight delays from time to time, but I attributed that
to the interpreter being context switched in, ssh delays ...or the
weather...in general the return was pretty fast.

I switched my program to use os.read and now it's snappy and spends
most of it's time blocking on select exactly where I want it to.

(this is on linux btw, I guess os.read is different everywhere)