Trouble with popen2

Discussion in 'Python' started by Rembrandt Q Einstein, Sep 27, 2004.

  1. I am running an external command and I need to know a) when it is done
    and b) what it wrote to both stdout and stderr. After a little
    searching, I found the popen2 module and used the Popen3 class. I'm
    having trouble with it hanging, though.

    Here is a very well put (by someone else) posting that describes some
    background:

    http://mail.python.org/pipermail/python-list/2004-July/230837.html

    I came to a similar conclusion as that poster and his workaround
    (independently discovered by me) does do the job he requires. However,
    I need to also read stderr, so I made this sample writer and runner:

    -------
    #!/usr/bin/python

    import sys

    for line in range(0, int(sys.argv[1])):
    print "X" * 120

    print >>sys.stderr, "hi"
    -------
    #!/usr/bin/python

    import popen2

    f = popen2.Popen3("./writer.py 50000", True)
    outs = []
    errs = []
    while (f.poll() == -1):
    errs += f.childerr.readlines()
    outs += f.fromchild.readlines()
    -----


    This hangs in the childerr.readlines(). Is it blocking? If so, why?
    In any case, how can I be sure to read ALL data from BOTH stderr and
    stdout and not be in danger of hanging?


    PS: I just found that if I swap the order of the readlines() statements,
    it works. But I don't want to use that until I understand why. I
    suspect it's a race condition and I don't want to rely on that.
    Rembrandt Q Einstein, Sep 27, 2004
    #1
    1. Advertising

  2. Rembrandt Q Einstein wrote:
    > I am running an external command and I need to know a) when it is done
    > and b) what it wrote to both stdout and stderr. After a little
    > searching, I found the popen2 module and used the Popen3 class. I'm
    > having trouble with it hanging, though.
    >
    > Here is a very well put (by someone else) posting that describes some
    > background:
    >
    > http://mail.python.org/pipermail/python-list/2004-July/230837.html
    >
    > I came to a similar conclusion as that poster and his workaround
    > (independently discovered by me) does do the job he requires. However,
    > I need to also read stderr, so I made this sample writer and runner:
    >
    > -------
    > #!/usr/bin/python
    >
    > import sys
    >
    > for line in range(0, int(sys.argv[1])):
    > print "X" * 120
    >
    > print >>sys.stderr, "hi"
    > -------
    > #!/usr/bin/python
    >
    > import popen2
    >
    > f = popen2.Popen3("./writer.py 50000", True)
    > outs = []
    > errs = []
    > while (f.poll() == -1):
    > errs += f.childerr.readlines()
    > outs += f.fromchild.readlines()
    > -----
    >
    >
    > This hangs in the childerr.readlines(). Is it blocking? If so, why? In
    > any case, how can I be sure to read ALL data from BOTH stderr and stdout
    > and not be in danger of hanging?
    >
    >
    > PS: I just found that if I swap the order of the readlines() statements,
    > it works. But I don't want to use that until I understand why. I
    > suspect it's a race condition and I don't want to rely on that.


    It's possible that when I have child.readlines() first, it consumes all
    50000 output lines and then err.readlines() grabs the "hi" and it just
    exists. When I have them the other way, err.readlines() blocks (I
    thought readlines() was non-blocking, though) and it just hangs.

    My real external program isn't nice enough to order the output like
    that, so here's a more realistic writer:

    -------------
    #!/usr/bin/python

    import sys
    import math

    for i in range(0, int(sys.argv[1])):
    if math.fmod(i, 100) == 0:
    print >>sys.stderr, "hi"
    print >>sys.stdout, "X" * 120
    ----

    The challenge is to write a program that will run this one in a
    subprocess and read all of both stderr and stdout.
    Rembrandt Q Einstein, Sep 27, 2004
    #2
    1. Advertising

  3. Rembrandt Q Einstein wrote:
    > Rembrandt Q Einstein wrote:
    >
    >> I am running an external command and I need to know a) when it is done
    >> and b) what it wrote to both stdout and stderr. After a little
    >> searching, I found the popen2 module and used the Popen3 class. I'm
    >> having trouble with it hanging, though.
    >>
    >> Here is a very well put (by someone else) posting that describes some
    >> background:
    >>
    >> http://mail.python.org/pipermail/python-list/2004-July/230837.html
    >>
    >> I came to a similar conclusion as that poster and his workaround
    >> (independently discovered by me) does do the job he requires.
    >> However, I need to also read stderr, so I made this sample writer and
    >> runner:
    >>
    >> -------
    >> #!/usr/bin/python
    >>
    >> import sys
    >>
    >> for line in range(0, int(sys.argv[1])):
    >> print "X" * 120
    >>
    >> print >>sys.stderr, "hi"
    >> -------
    >> #!/usr/bin/python
    >>
    >> import popen2
    >>
    >> f = popen2.Popen3("./writer.py 50000", True)
    >> outs = []
    >> errs = []
    >> while (f.poll() == -1):
    >> errs += f.childerr.readlines()
    >> outs += f.fromchild.readlines()
    >> -----
    >>
    >>
    >> This hangs in the childerr.readlines(). Is it blocking? If so, why?
    >> In any case, how can I be sure to read ALL data from BOTH stderr and
    >> stdout and not be in danger of hanging?
    >>
    >>
    >> PS: I just found that if I swap the order of the readlines()
    >> statements, it works. But I don't want to use that until I understand
    >> why. I suspect it's a race condition and I don't want to rely on that.

    >
    >
    > It's possible that when I have child.readlines() first, it consumes all
    > 50000 output lines and then err.readlines() grabs the "hi" and it just
    > exists. When I have them the other way, err.readlines() blocks (I
    > thought readlines() was non-blocking, though) and it just hangs.
    >
    > My real external program isn't nice enough to order the output like
    > that, so here's a more realistic writer:
    >
    > -------------
    > #!/usr/bin/python
    >
    > import sys
    > import math
    >
    > for i in range(0, int(sys.argv[1])):
    > if math.fmod(i, 100) == 0:
    > print >>sys.stderr, "hi"
    > print >>sys.stdout, "X" * 120
    > ----
    >
    > The challenge is to write a program that will run this one in a
    > subprocess and read all of both stderr and stdout.


    Ah--despite my new nick, I am an idiot. I should just use select() on
    fromchild and childerr. Something like this:

    while f.poll() == -1:
    select([], [f.fromchild, f.childerr],[], 0)
    blah

    However, I think I might still miss some data this way.
    Rembrandt Q Einstein, Sep 27, 2004
    #3
  4. On Mon, 27 Sep 2004 15:27:56 -0400, Rembrandt Q Einstein <hercules.rockefeller@springfield.??.us> wrote:
    >> The challenge is to write a program that will run this one in a
    >> subprocess and read all of both stderr and stdout.

    >
    > Ah--despite my new nick, I am an idiot. I should just use select() on
    > fromchild and childerr. Something like this:
    >
    > while f.poll() == -1:
    > select([], [f.fromchild, f.childerr],[], 0)
    > blah
    >
    > However, I think I might still miss some data this way.


    Nope, this should work, assuming that you catch the output of select,
    and act accordingly.
    (at least, that is my experience)

    Not sure why you need the f.poll() though, I think you can do without.



    Maybe an extension to the documentation of popen would be in order, this
    post pops up quite often here.

    --
    Albert
    --
    Unlike popular belief, the .doc format is not an open publically available format.
    Albert Hofkamp, Sep 28, 2004
    #4
  5. Rembrandt Q Einstein

    Guest

    Rembrandt Q Einstein wrote:
    > I am running an external command and I need to know a) when it is done
    > and b) what it wrote to both stdout and stderr. After a little
    > searching, I found the popen2 module and used the Popen3 class. I'm
    > having trouble with it hanging, though.


    This should be a FAQ if it's not.
    Have a go with: http://www.pixelbeat.org/libs/subProcess.py

    Pádraig.
    , Sep 29, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Guy

    popen2

    Guy, Aug 12, 2003, in forum: Python
    Replies:
    1
    Views:
    2,835
    Donn Cave
    Aug 12, 2003
  2. cherico

    popen2 with large input

    cherico, Jan 29, 2004, in forum: Python
    Replies:
    2
    Views:
    317
    Jeff Epler
    Jan 29, 2004
  3. Diez B. Roggisch

    popen2 trouble

    Diez B. Roggisch, Apr 2, 2004, in forum: Python
    Replies:
    2
    Views:
    306
    Diez B. Roggisch
    Apr 5, 2004
  4. A. Lloyd Flanagan

    Possible problem with popen2 module

    A. Lloyd Flanagan, Apr 30, 2004, in forum: Python
    Replies:
    2
    Views:
    303
    A. Lloyd Flanagan
    May 3, 2004
  5. Daniel Klein
    Replies:
    3
    Views:
    284
    Daniel Klein
    Nov 18, 2006
Loading...

Share This Page