Trouble with popen2

[Rembrandt Q Einstein]

I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

Here is a very well put (by someone else) posting that describes some
background:

http://mail.python.org/pipermail/python-list/2004-July/230837.html

I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires. However,
I need to also read stderr, so I made this sample writer and runner:

-------
#!/usr/bin/python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120

print >>sys.stderr, "hi"
-------
#!/usr/bin/python

import popen2

f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----


This hangs in the childerr.readlines(). Is it blocking? If so, why?
In any case, how can I be sure to read ALL data from BOTH stderr and
stdout and not be in danger of hanging?


PS: I just found that if I swap the order of the readlines() statements,
it works. But I don't want to use that until I understand why. I
suspect it's a race condition and I don't want to rely on that.

 

[Rembrandt Q Einstein]

I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

Here is a very well put (by someone else) posting that describes some
background:

http://mail.python.org/pipermail/python-list/2004-July/230837.html

I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires. However,
I need to also read stderr, so I made this sample writer and runner:

-------
#!/usr/bin/python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120

print >>sys.stderr, "hi"
-------
#!/usr/bin/python

import popen2

f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----


This hangs in the childerr.readlines(). Is it blocking? If so, why? In
any case, how can I be sure to read ALL data from BOTH stderr and stdout
and not be in danger of hanging?


PS: I just found that if I swap the order of the readlines() statements,
it works. But I don't want to use that until I understand why. I
suspect it's a race condition and I don't want to rely on that.

It's possible that when I have child.readlines() first, it consumes all
50000 output lines and then err.readlines() grabs the "hi" and it just
exists. When I have them the other way, err.readlines() blocks (I
thought readlines() was non-blocking, though) and it just hangs.

My real external program isn't nice enough to order the output like
that, so here's a more realistic writer:

-------------
#!/usr/bin/python

import sys
import math

for i in range(0, int(sys.argv[1])):
if math.fmod(i, 100) == 0:
print >>sys.stderr, "hi"
print >>sys.stdout, "X" * 120

 

[Rembrandt Q Einstein]

I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

Here is a very well put (by someone else) posting that describes some
background:

http://mail.python.org/pipermail/python-list/2004-July/230837.html

I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires.
However, I need to also read stderr, so I made this sample writer and
runner:

-------
#!/usr/bin/python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120

print >>sys.stderr, "hi"
-------
#!/usr/bin/python

import popen2

f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----


This hangs in the childerr.readlines(). Is it blocking? If so, why?
In any case, how can I be sure to read ALL data from BOTH stderr and
stdout and not be in danger of hanging?


PS: I just found that if I swap the order of the readlines()
statements, it works. But I don't want to use that until I understand
why. I suspect it's a race condition and I don't want to rely on that.

It's possible that when I have child.readlines() first, it consumes all
50000 output lines and then err.readlines() grabs the "hi" and it just
exists. When I have them the other way, err.readlines() blocks (I
thought readlines() was non-blocking, though) and it just hangs.

My real external program isn't nice enough to order the output like
that, so here's a more realistic writer:

-------------
#!/usr/bin/python

import sys
import math

for i in range(0, int(sys.argv[1])):
if math.fmod(i, 100) == 0:
print >>sys.stderr, "hi"
print >>sys.stdout, "X" * 120

Ah--despite my new nick, I am an idiot. I should just use select() on
fromchild and childerr. Something like this:

while f.poll() == -1:
select([], [f.fromchild, f.childerr],[], 0)
blah

However, I think I might still miss some data this way.

 

[Albert Hofkamp]

The challenge is to write a program that will run this one in a
subprocess and read all of both stderr and stdout.

Ah--despite my new nick, I am an idiot. I should just use select() on
fromchild and childerr. Something like this:

while f.poll() == -1:
select([], [f.fromchild, f.childerr],[], 0)
blah

However, I think I might still miss some data this way.

Nope, this should work, assuming that you catch the output of select,
and act accordingly.
(at least, that is my experience)

Not sure why you need the f.poll() though, I think you can do without.



Maybe an extension to the documentation of popen would be in order, this
post pops up quite often here.

 

[P]

I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

This should be a FAQ if it's not.
Have a go with: http://www.pixelbeat.org/libs/subProcess.py

Pádraig.