Capturing stderr and stdout of a subprocess as a single stream

Discussion in 'Python' started by Fuzzyman, Jan 7, 2007.

  1. Fuzzyman

    Fuzzyman Guest

    Hello all,

    Before I ask the question a couple of notes :

    * This question is for implementing a script inside the Wing IDE. For
    some reason using the subprocess module doesn't work so I need a
    solution that doesn't use this module.
    * The platform is Windows and I'm happy with a Windoze only solution.
    :)

    I would like to execute subprocesses asynchronously and capture stdout
    / stderr as a single stream.

    If I use "os.popen3(executable)" it gives me separate pipes for stdout
    and stderr, but reads from them are blocking.

    The output on stdout and stderr may be interleaved and I would like to
    display them *as* they arrive. That means I can't just read from them
    and output the results.

    The only solution I can think of is to read from both a character at a
    time on two separate threads, putting the data into a queue. A separate
    thread could pull characters off the queue and display them. (I don't
    need to differentiate between stdout and stderr when I display.)

    Can anyone think of a better solution ?

    My current code works, but *doesn't* capture stderr :

    from threading import Thread

    pipe = os.popen(executable)

    def DisplayOutput():
    while True:
    output = pipe.read(1)
    if not output:
    break
    display(output)

    Thread(target=DisplayOutput).start()

    All the best,


    Fuzzyman
    http://www.voidspace.org.uk/python/articles.shtml
    Fuzzyman, Jan 7, 2007
    #1
    1. Advertising

  2. On 7 ene, 16:33, "Fuzzyman" <> wrote:
    > Hello all,
    >
    > Before I ask the question a couple of notes :
    >
    > * This question is for implementing a script inside the Wing IDE. For
    > some reason using the subprocess module doesn't work so I need a
    > solution that doesn't use this module.
    > * The platform is Windows and I'm happy with a Windoze only solution.
    > :)
    >
    > I would like to execute subprocesses asynchronously and capture stdout
    > / stderr as a single stream.
    >
    > If I use "os.popen3(executable)" it gives me separate pipes for stdout
    > and stderr, but reads from them are blocking.
    >
    > The output on stdout and stderr may be interleaved and I would like to
    > display them *as* they arrive. That means I can't just read from them
    > and output the results.
    >
    > The only solution I can think of is to read from both a character at a
    > time on two separate threads, putting the data into a queue. A separate
    > thread could pull characters off the queue and display them. (I don't
    > need to differentiate between stdout and stderr when I display.)
    >
    > Can anyone think of a better solution ?
    >
    > My current code works, but *doesn't* capture stderr :
    >
    > from threading import Thread
    >
    > pipe = os.popen(executable)
    >
    > def DisplayOutput():
    > while True:
    > output = pipe.read(1)
    > if not output:
    > break
    > display(output)
    >
    > Thread(target=DisplayOutput).start()
    >
    > All the best,
    >
    > Fuzzymanhttp://www.voidspace.org.uk/python/articles.shtml


    Try using popen4 instead. But since you already have a thread, it may
    be better to use popen2 and two threads to read from stdout and stderr
    to avoid a potential deadlock; they can put read lines into a Queue,
    and DisplayOutput just get these lines in order. (See the warnings in
    the popen2 module documentation).

    --
    Gabriel Genellina
    Gabriel Genellina, Jan 7, 2007
    #2
    1. Advertising

  3. Fuzzyman

    Fuzzyman Guest

    Gabriel Genellina wrote:
    > On 7 ene, 16:33, "Fuzzyman" <> wrote:

    [snip..]
    > > My current code works, but *doesn't* capture stderr :
    > >
    > > from threading import Thread
    > >
    > > pipe = os.popen(executable)
    > >
    > > def DisplayOutput():
    > > while True:
    > > output = pipe.read(1)
    > > if not output:
    > > break
    > > display(output)
    > >
    > > Thread(target=DisplayOutput).start()
    > >
    > > All the best,
    > >
    > > Fuzzyman
    > > http://www.voidspace.org.uk/python/articles.shtml

    >
    > Try using popen4 instead. But since you already have a thread, it may
    > be better to use popen2 and two threads to read from stdout and stderr
    > to avoid a potential deadlock; they can put read lines into a Queue,
    > and DisplayOutput just get these lines in order. (See the warnings in
    > the popen2 module documentation).
    >


    popen4 works great, I didn't even know it existed.

    Two threads and a queue sounds horrible.

    Thanks

    Fuzzyman
    http://www.voidspace.org.uk/python/articles.shtml

    > --
    > Gabriel Genellina
    Fuzzyman, Jan 7, 2007
    #3
  4. On 7 ene, 17:37, "Fuzzyman" <> wrote:

    > Two threads and a queue sounds horrible.


    But unfortunately it's the only way if you don't control how the child
    process behaves.
    (It's not soooo ugly afterwards... certainly would be worse if you had
    to syncronize both reading threads and the display thread using
    semaphores by hand.)

    --
    Gabriel Genellina
    Gabriel Genellina, Jan 7, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Roman Neuhauser
    Replies:
    0
    Views:
    656
    Roman Neuhauser
    Apr 4, 2005
  2. robert
    Replies:
    4
    Views:
    549
    Stephane Chazelas
    Apr 21, 2006
  3. Christoph Haas
    Replies:
    0
    Views:
    389
    Christoph Haas
    Jun 13, 2006
  4. Chris Withers
    Replies:
    1
    Views:
    434
    Lie Ryan
    Dec 18, 2009
  5. Mike

    capturing stdout and stderr?

    Mike, Aug 9, 2009, in forum: Perl Misc
    Replies:
    0
    Views:
    98
Loading...

Share This Page