Possible problem with popen2 module

[A. Lloyd Flanagan]

OK, I've got a weird one I haven't been able to figure out yet.
Admittedly I haven't had time to dig into the library source, but this
behavior certainly doesn't seem right. Here's a test case:

"""Test program to demonstrate problem with popen2 module."""
import popen2

def main (argv):
mycmd = 'python2.3 -c "for i in range(100000):\n print i"'
p = popen2.Popen3(mycmd)
print 'result code is %d' % p.wait()
for line in p.fromchild.xreadlines():
print line,

if __name__ == '__main__':
import sys
main(sys.argv)

As you can see I'm using the popen2.Popen3 class to run a process
which prints a lot of output.

Here's the problem: for small values of the constant in the range()
call, e.g. 1000, this works as expected. For larger values, e.g.
100,000, the program never returns from the p.wait() call after the
child process completes. It appears tbe waiting forever on the
waitpid() call.

This is occuring on a Sun server:

uname -a

SunOS fred 5.8 Generic_108528-29 sun4u sparc SUNW,Sun-Fire-880

and I've seen the exact same behavior under HP-UX:

uname -a

HP-UX hpserver B.11.11 U 9000/800 2243344530 unlimited-user license

I don't see this behavior with calling os.popen(). I DO see it with
the current implementation of popen5() from the PEP.

Does anyone know why this is occurring? Is this a bona-fide bug? Or
am I just being stupid somehow?
Thanks!

A. Lloyd Flanagan
Contract Programmer
Richmond, VA

 

[Donn Cave]

OK, I've got a weird one I haven't been able to figure out yet.
Admittedly I haven't had time to dig into the library source, but this
behavior certainly doesn't seem right. Here's a test case:

"""Test program to demonstrate problem with popen2 module."""
import popen2

def main (argv):
mycmd = 'python2.3 -c "for i in range(100000):\n print i"'
p = popen2.Popen3(mycmd)
print 'result code is %d' % p.wait()
for line in p.fromchild.xreadlines():
print line,

if __name__ == '__main__':
import sys
main(sys.argv)

As you can see I'm using the popen2.Popen3 class to run a process
which prints a lot of output.

Here's the problem: for small values of the constant in the range()
call, e.g. 1000, this works as expected. For larger values, e.g.
100,000, the program never returns from the p.wait() call after the
child process completes. It appears tbe waiting forever on the
waitpid() call.

I don't see this behavior with calling os.popen(). I DO see it with
the current implementation of popen5() from the PEP.

Does anyone know why this is occurring? Is this a bona-fide bug? Or
am I just being stupid somehow?

Well, I will leave it to you to decide how stupid this was.
Nice job on the problem report though, really covers all the
bases.

Pipes are `slow', fixed size devices. You can write only some
small amount of data to a pipe, and then you have to wait for
the peer process on the other end to read that data and make
more room. Waiting like (and waiting for the peer on reads)
is what makes them slow, which really means interruptible by
signals (just an aside.)

What would work the way you want is a disk file. Redirect
output to a file, wait for the process to exit, and read the
file. Pipes are for processes whose output you want to read
while in progress, and you must do that whether you want to
or not.

You don't have exactly this problem with popen() because you're
not really doing the same thing - it doesn't have a wait(),
just a close(), and close() closes the pipe first, which kills
the process so the wait works.

Donn Cave, (e-mail address removed)

 

[A. Lloyd Flanagan]

Pipes are `slow', fixed size devices. You can write only some
small amount of data to a pipe, and then you have to wait for
the peer process on the other end to read that data and make
more room. Waiting like (and waiting for the peer on reads)
is what makes them slow, which really means interruptible by
signals (just an aside.)

What would work the way you want is a disk file. Redirect

Thanks Donn, it makes sense now. I only feel a little stupid .
I'm certainly learning more about inter-process communcation on this assignment.