subprocess module and blocking

[Robin Becker]

I'm using a polling loop in a thread that looks approximately like this

while 1:
p = find_a_process()
rc = p.poll()
if rc is not None:
out, err = p.communicate()
#deal with output etc
sleep(1)

the process p is opened using

p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, cwd=cwd)

stdin is actually never written to.

I notice that under both win32 and freebsd that things are fine provided
that the subprocess doesn't write too much to stdout/stderr. However,
the subprocess seems to lock often if too much is written (under freebsd
I see a process state of POLL). I assume that the subprocess is filling
up the pipe and then failing to wake up again. I had expected that
subprocess would take care of this for me, but possibly I'm being
utterly clueless and stupid. What should I do to avoid blocking in the
subprocess?

 

[Dieter Maurer]

I'm using a polling loop in a thread that looks approximately like this

while 1:
p = find_a_process()
rc = p.poll()
if rc is not None:
out, err = p.communicate()
#deal with output etc
sleep(1)
....
I notice that under both win32 and freebsd that things are fine
provided that the subprocess doesn't write too much to
stdout/stderr. However, the subprocess seems to lock often if too much
is written (under freebsd I see a process state of POLL). I assume
that the subprocess is filling up the pipe and then failing to wake up
again. I had expected that subprocess would take care of this for me,

You just found out that this is not the case.

The warning attached to "communicate"s docstring might have
you averted: "Note: the data read is buffered in memory...
do not use for large size".

If subprocess would do what you expect, it would need to
buffer the output (to make room in the pipes again).
But, for large data, this could have dramatic consequences.

Thus, you should use "select" on the pipes to find out
when to read data.


Dieter

 

[Robin Becker]

Dieter Maurer wrote:
......

You just found out that this is not the case.

thanks I suppose I was being a moron.

The warning attached to "communicate"s docstring might have
you averted: "Note: the data read is buffered in memory...
do not use for large size".

If subprocess would do what you expect, it would need to
buffer the output (to make room in the pipes again).
But, for large data, this could have dramatic consequences.

Thus, you should use "select" on the pipes to find out
when to read data.

Unfortunately select isn't an option for windows (I think that's only for
sockets). I guess I need to read repeatedly from the stdout etc to get the
output. Clearly the poll call can't return a status until we've finished
reading.