get text from rogramms runn by subprocess.Popen immediatetly

Discussion in 'Python' started by Rüdiger Ranft, Apr 16, 2009.

  1. Hi all,

    I need to call some programms and catch their stdout and stderr streams.
    While the Popen class from subprocess handles the call, I get the
    results of the programm not until the programm finishes. Since the
    output of the programm is used to generate a progress indicator, I need
    a way to acces the values written to stdout/stderr as fast as possible.

    Beneath is a test which shows what I did

    TIA
    Rudi

    ----8<-------8<-------8<-- iodummy.cpp -8<-------8<---
    #include <iostream>
    #include <unistd.h>

    int main()
    {
    for( int i = 0; i < 10; i++ )
    {
    std::cerr << i << std::endl;
    sleep(2);
    }
    }

    from subprocess import Popen, PIPE
    from time import sleep

    p = Popen('./iodummy',stdin=PIPE, stdout=PIPE, stderr=PIPE)
    sleep(3)
    # now I expect '0\n1\n' in stderr, but read() blocks until
    # the end of iodummy.
    print p.stderr.read()
    p.wait()

    --
    GPG encrypted mails preferred.
    GPG verschlüsselte Mails bevorzugt.
    ---> http://chaosradio.ccc.de/media/ds/ds085.pdf Seite 20 <----


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.8 (MingW32)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

    iQIcBAEBAwAGBQJJ5xKqAAoJEI+IiZON95WTR5YP/iymEmF37qbnnMSSQZFs629Q
    sSLrcjNytEmuQZvYdvsrJJ2LiR4NHzLTC+zVVP41VsOwfydfAFZhWBuKZTN2pqUT
    jfZAYM1KALoy0f9Z9gD08ehTwrXy5+1ZRmj3aLmdqSmD6Jqjfnz3495VivB2f9JY
    aYT8ZGglikVDeiTO8GekA7Lyfq+DItZ8jXcSxlQqiz6ega0DapL6NHH6c9d5nx+F
    qAc7/JERZsMNd8EMDp6tb8xDVFp/ZIzSS3aYFRmQHfdE8lSp5DG2868gzEVU/h5Y
    0D25gs/tAmuYFyutMq1GulDwLcpZWFqjidibav6M6XabvL45QcsKrsAlDQrLin/x
    JwDct72lgseh2JrPx4Kcdz1/+4uppMuPYbnJ1YMTsGxthMXyP0Wk2RVITrVHoH2P
    j4GwuNPGiEu3rkNutUUer4wkQVUnss/D71jNYXl9SgmoMsq9Ub66p8GO9vepgWlw
    NK/7U5iad96ETHJgcvWhx8o/ymr8V6kOBqQpLlRCNsd/2MRQvoT61tmK0UDdx2lL
    x/LrhZftbAiVN5hzzAC2H6nX02m9bipfqWKbzvp71wbLZufXAo2R8jSmibrdUQff
    16TUVeIhUY4YIuIscG6YyTMit5tEUTtCL2Mi3yTFe8R8uOR3h/1t1DbhDBYRtQAh
    Sa8efc/lkb8gLUJU3nnz
    =xaSm
    -----END PGP SIGNATURE-----
     
    Rüdiger Ranft, Apr 16, 2009
    #1
    1. Advertising

  2. Rüdiger Ranft schrieb:
    > Hi all,
    >
    > I need to call some programms and catch their stdout and stderr streams.
    > While the Popen class from subprocess handles the call, I get the
    > results of the programm not until the programm finishes. Since the
    > output of the programm is used to generate a progress indicator, I need
    > a way to acces the values written to stdout/stderr as fast as possible.


    Use the communicate()-method of Popen-objects.

    Diez
     
    Diez B. Roggisch, Apr 16, 2009
    #2
    1. Advertising

  3. Re: Re: get text from rogramms runn by subprocess.Popen immediatetly

    Diez B. Roggisch schrieb:
    > Rüdiger Ranft schrieb:
    >> Hi all,
    >>
    >> I need to call some programms and catch their stdout and stderr streams.
    >> While the Popen class from subprocess handles the call, I get the
    >> results of the programm not until the programm finishes. Since the
    >> output of the programm is used to generate a progress indicator, I need
    >> a way to acces the values written to stdout/stderr as fast as possible.

    >
    > Use the communicate()-method of Popen-objects.


    It gives the same behavior, the first call to communicate gets all text,
    the second gives a closed handle error :(. I also tried
    p.communicate(''), which gives the same as p.communicate(None) result.

    bye
    Rudi

    cat run.py
    from subprocess import Popen, PIPE
    from time import sleep

    p = Popen('./iodummy',stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=2)
    sleep(3)
    # now I expect '0\n1\n' in stderr
    print 'Point1:', p.communicate(None)
    sleep(3)
    print 'Point2:', p.communicate(None)
    p.wait()

    python run.py
    Point1: ('', '0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n')
    Point2:
    Traceback (most recent call last):
    File "run.py", line 9, in ?
    print 'Point2:', p.communicate('')
    File "/usr/lib/python2.4/subprocess.py", line 1028, in communicate
    self.stdin.flush()
    ValueError: I/O operation on closed file

    --
    GPG encrypted mails preferred.
    GPG verschlüsselte Mails bevorzugt.
    ---> http://chaosradio.ccc.de/media/ds/ds085.pdf Seite 20 <----
     
    Rüdiger Ranft, Apr 16, 2009
    #3
  4. Rüdiger Ranft schrieb:
    > Diez B. Roggisch schrieb:
    >> Rüdiger Ranft schrieb:
    >>> Hi all,
    >>>
    >>> I need to call some programms and catch their stdout and stderr streams.
    >>> While the Popen class from subprocess handles the call, I get the
    >>> results of the programm not until the programm finishes. Since the
    >>> output of the programm is used to generate a progress indicator, I need
    >>> a way to acces the values written to stdout/stderr as fast as possible.

    >> Use the communicate()-method of Popen-objects.

    >
    > It gives the same behavior, the first call to communicate gets all text,
    > the second gives a closed handle error :(. I also tried
    > p.communicate(''), which gives the same as p.communicate(None) result.


    That's likely due to buffering then. You could try & see if flushing on
    the side of the subprocess works (you can't do that from python's side
    of things within this scenario).

    Or you might want to use pexpect that emulates a pseudo-terminal which
    also changes the way data is displayed.

    Diez
     
    Diez B. Roggisch, Apr 16, 2009
    #4
  5. On Apr 16, 4:12 am, Rüdiger Ranft <> wrote:
    > Hi all,
    >
    > I need to call some programms and catch their stdout and stderr streams.
    > While the Popen class from subprocess handles the call, I get the
    > results of the programm not until the programm finishes. Since the
    > output of the programm is used to generate a progress indicator, I need
    > a way to acces the values written to stdout/stderr as fast as possible.
    >
    > Beneath is a test which shows what I did
    >
    > TIA
    > Rudi
    >
    > ----8<-------8<-------8<-- iodummy.cpp -8<-------8<---
    > #include <iostream>
    > #include <unistd.h>
    >
    > int main()
    > {
    > for( int i = 0; i < 10; i++ )
    > {
    > std::cerr << i << std::endl;
    > sleep(2);
    > }
    >
    > }
    >
    > from subprocess import Popen, PIPE
    > from time import sleep
    >
    > p = Popen('./iodummy',stdin=PIPE, stdout=PIPE, stderr=PIPE)
    > sleep(3)
    > # now I expect '0\n1\n' in stderr, but read() blocks until
    > # the end of iodummy.
    > print p.stderr.read()
    > p.wait()
    >


    I'm sure the regulars will correct me, but if are on some kind of *nix
    variant, couldn't you just redirect the stderr to the stdout like the
    following

    [cdalten@localhost oakland]$ more nope2.py
    #!/usr/bin/python

    import os
    os.popen('./iodummy 2>&1', 'w')
    [cdalten@localhost oakland]$ ./nope2.py
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    [cdalten@localhost oakland]$
     
    grocery_stocker, Apr 16, 2009
    #5
  6. >>>>> Rüdiger Ranft <> (RR) wrote:

    >RR> Hi all,
    >RR> I need to call some programms and catch their stdout and stderr streams.
    >RR> While the Popen class from subprocess handles the call, I get the
    >RR> results of the programm not until the programm finishes. Since the
    >RR> output of the programm is used to generate a progress indicator, I need
    >RR> a way to acces the values written to stdout/stderr as fast as possible.


    >RR> print p.stderr.read()


    Use p.stderr.readline() or p.stderr.read(1) in a loop.
    --
    Piet van Oostrum <>
    URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
    Private email:
     
    Piet van Oostrum, Apr 17, 2009
    #6
  7. Rüdiger Ranft

    Aaron Brady Guest

    On Apr 16, 8:02 am, Rüdiger Ranft <> wrote:
    > Diez B. Roggisch schrieb:
    >
    > > Rüdiger Ranft schrieb:
    > >> Hi all,

    >
    > >> I need to call some programms and catch their stdout and stderr streams.
    > >> While the Popen class from subprocess handles the call, I get the
    > >> results of the programm not until the programm finishes. Since the
    > >> output of the programm is used to generate a progress indicator, I need
    > >> a way to acces the values written to stdout/stderr as fast as possible..

    snip

    FWIR from what I recall, you have to create and open read and write
    file descriptors using the 'msvcrt.get_osfhandle' function, and pass
    them to the subprocess on the command line. Or you can pass the
    handles. Then you get a communication channel that is not STDIN.

    I have no idea why the designers did this. I've submitted a patch,
    and received 0 attention on it.
     
    Aaron Brady, Apr 17, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michele Simionato

    subprocess.Popen

    Michele Simionato, Dec 12, 2004, in forum: Python
    Replies:
    5
    Views:
    17,666
  2. Ivo Woltring
    Replies:
    4
    Views:
    3,558
    Jeff Shannon
    Jan 27, 2005
  3. John Abel
    Replies:
    2
    Views:
    3,304
    John Abel
    May 12, 2005
  4. calmar
    Replies:
    1
    Views:
    325
    Fredrik Lundh
    Feb 16, 2006
  5. File.popen/IO.popen

    , May 20, 2006, in forum: Ruby
    Replies:
    1
    Views:
    232
    Robert Klemme
    May 20, 2006
Loading...

Share This Page