Possible problem with popen2 module

Discussion in 'Python' started by A. Lloyd Flanagan, Apr 30, 2004.

  1. OK, I've got a weird one I haven't been able to figure out yet.
    Admittedly I haven't had time to dig into the library source, but this
    behavior certainly doesn't seem right. Here's a test case:

    """Test program to demonstrate problem with popen2 module."""
    import popen2

    def main (argv):
    mycmd = 'python2.3 -c "for i in range(100000):\n print i"'
    p = popen2.Popen3(mycmd)
    print 'result code is %d' % p.wait()
    for line in p.fromchild.xreadlines():
    print line,

    if __name__ == '__main__':
    import sys
    main(sys.argv)

    As you can see I'm using the popen2.Popen3 class to run a process
    which prints a lot of output.

    Here's the problem: for small values of the constant in the range()
    call, e.g. 1000, this works as expected. For larger values, e.g.
    100,000, the program never returns from the p.wait() call after the
    child process completes. It appears tbe waiting forever on the
    waitpid() call.

    This is occuring on a Sun server:
    > uname -a

    SunOS fred 5.8 Generic_108528-29 sun4u sparc SUNW,Sun-Fire-880

    and I've seen the exact same behavior under HP-UX:
    > uname -a

    HP-UX hpserver B.11.11 U 9000/800 2243344530 unlimited-user license

    I don't see this behavior with calling os.popen(). I DO see it with
    the current implementation of popen5() from the PEP.

    Does anyone know why this is occurring? Is this a bona-fide bug? Or
    am I just being stupid somehow?
    Thanks!

    A. Lloyd Flanagan
    Contract Programmer
    Richmond, VA
    A. Lloyd Flanagan, Apr 30, 2004
    #1
    1. Advertising

  2. A. Lloyd Flanagan

    Donn Cave Guest

    In article <>,
    (A. Lloyd Flanagan) wrote:
    > OK, I've got a weird one I haven't been able to figure out yet.
    > Admittedly I haven't had time to dig into the library source, but this
    > behavior certainly doesn't seem right. Here's a test case:
    >
    > """Test program to demonstrate problem with popen2 module."""
    > import popen2
    >
    > def main (argv):
    > mycmd = 'python2.3 -c "for i in range(100000):\n print i"'
    > p = popen2.Popen3(mycmd)
    > print 'result code is %d' % p.wait()
    > for line in p.fromchild.xreadlines():
    > print line,
    >
    > if __name__ == '__main__':
    > import sys
    > main(sys.argv)
    >
    > As you can see I'm using the popen2.Popen3 class to run a process
    > which prints a lot of output.
    >
    > Here's the problem: for small values of the constant in the range()
    > call, e.g. 1000, this works as expected. For larger values, e.g.
    > 100,000, the program never returns from the p.wait() call after the
    > child process completes. It appears tbe waiting forever on the
    > waitpid() call.


    > I don't see this behavior with calling os.popen(). I DO see it with
    > the current implementation of popen5() from the PEP.
    >
    > Does anyone know why this is occurring? Is this a bona-fide bug? Or
    > am I just being stupid somehow?


    Well, I will leave it to you to decide how stupid this was.
    Nice job on the problem report though, really covers all the
    bases.

    Pipes are `slow', fixed size devices. You can write only some
    small amount of data to a pipe, and then you have to wait for
    the peer process on the other end to read that data and make
    more room. Waiting like (and waiting for the peer on reads)
    is what makes them slow, which really means interruptible by
    signals (just an aside.)

    What would work the way you want is a disk file. Redirect
    output to a file, wait for the process to exit, and read the
    file. Pipes are for processes whose output you want to read
    while in progress, and you must do that whether you want to
    or not.

    You don't have exactly this problem with popen() because you're
    not really doing the same thing - it doesn't have a wait(),
    just a close(), and close() closes the pipe first, which kills
    the process so the wait works.

    Donn Cave,
    Donn Cave, Apr 30, 2004
    #2
    1. Advertising

  3. Donn Cave <> wrote in message news:<>...
    > In article <>,
    > (A. Lloyd Flanagan) wrote:


    > Pipes are `slow', fixed size devices. You can write only some
    > small amount of data to a pipe, and then you have to wait for
    > the peer process on the other end to read that data and make
    > more room. Waiting like (and waiting for the peer on reads)
    > is what makes them slow, which really means interruptible by
    > signals (just an aside.)
    >
    > What would work the way you want is a disk file. Redirect


    Thanks Donn, it makes sense now. I only feel a little stupid :).
    I'm certainly learning more about inter-process communcation on this assignment.
    A. Lloyd Flanagan, May 3, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Guy

    popen2

    Guy, Aug 12, 2003, in forum: Python
    Replies:
    1
    Views:
    2,842
    Donn Cave
    Aug 12, 2003
  2. cherico

    popen2 with large input

    cherico, Jan 29, 2004, in forum: Python
    Replies:
    2
    Views:
    322
    Jeff Epler
    Jan 29, 2004
  3. Diez B. Roggisch

    popen2 trouble

    Diez B. Roggisch, Apr 2, 2004, in forum: Python
    Replies:
    2
    Views:
    315
    Diez B. Roggisch
    Apr 5, 2004
  4. Magnus Lycka

    Problem with select.poll and popen2

    Magnus Lycka, Aug 30, 2005, in forum: Python
    Replies:
    1
    Views:
    2,565
    Magnus Lycka
    Aug 30, 2005
  5. diego

    Problem with win32pipe.popen2

    diego, Jan 15, 2007, in forum: Python
    Replies:
    1
    Views:
    332
    Gabriel Genellina
    Jan 16, 2007
Loading...

Share This Page