subprocess module and long-lived subprocesses

[skip]

I'm converting some os.popen calls to use subprocess.Popen. I had
previously been ignoring stdout and stderr when using os.popen. The primary
motivation to switch to subprocess.Popen now is that I now want to check
stderr, so would have to make code changes to use os.popen[34] anyway.
Might as well go whole hog and switch to the new API.

The library documentation doesn't talk a lot about long-lived subprocesses
other than the possibility of deadlock when using Popen.wait(). Ideally, I
would write to the subprocess's stdin, check for output on stdout and
stderr, then lather, rinse, repeat. Is it safe to assume that if the stdout
and/or stderr pipes have nothing for me the reads on those file objects (I'm
using PIPE for all three std* files) will return immediately with an empty
string for output? They won't block, will they? Will a broken pipe IOError
get raised as for os.popen() or do I have to call Popen.poll() even in error
situations?

Thanks,

 

[Nobody]

The library documentation doesn't talk a lot about long-lived subprocesses
other than the possibility of deadlock when using Popen.wait(). Ideally, I
would write to the subprocess's stdin, check for output on stdout and
stderr, then lather, rinse, repeat. Is it safe to assume that if the stdout
and/or stderr pipes have nothing for me the reads on those file objects (I'm
using PIPE for all three std* files) will return immediately with an empty
string for output? They won't block, will they?

They will. You need to use either threads, select() or non-blocking I/O in
order to avoid deadlock. See the definitions of subprocess._communicate()
(there's one version for Windows which uses threads and another for Unix
using select()).

Will a broken pipe IOError get raised as for os.popen()

IOError(EPIPE) will be raised if you write to the stdin pipe when there
are no readers.

or do I have to call Popen.poll() even in error situations?

Once you're finished with the process, you should close .stdin then
consume all output from .stdout and .stderr until both report EOF, then
call .wait(). That should cover any possible child behaviour (e.g. if
the child explicitly close()s its stdin, getting EPIPE doesn't mean that
you can forget about the process or that .wait() won't deadlock).