Capturing stderr and stdout of a subprocess as a single stream

F

Fuzzyman

Hello all,

Before I ask the question a couple of notes :

* This question is for implementing a script inside the Wing IDE. For
some reason using the subprocess module doesn't work so I need a
solution that doesn't use this module.
* The platform is Windows and I'm happy with a Windoze only solution.
:)

I would like to execute subprocesses asynchronously and capture stdout
/ stderr as a single stream.

If I use "os.popen3(executable)" it gives me separate pipes for stdout
and stderr, but reads from them are blocking.

The output on stdout and stderr may be interleaved and I would like to
display them *as* they arrive. That means I can't just read from them
and output the results.

The only solution I can think of is to read from both a character at a
time on two separate threads, putting the data into a queue. A separate
thread could pull characters off the queue and display them. (I don't
need to differentiate between stdout and stderr when I display.)

Can anyone think of a better solution ?

My current code works, but *doesn't* capture stderr :

from threading import Thread

pipe = os.popen(executable)

def DisplayOutput():
while True:
output = pipe.read(1)
if not output:
break
display(output)

Thread(target=DisplayOutput).start()

All the best,


Fuzzyman
http://www.voidspace.org.uk/python/articles.shtml
 
G

Gabriel Genellina

Hello all,

Before I ask the question a couple of notes :

* This question is for implementing a script inside the Wing IDE. For
some reason using the subprocess module doesn't work so I need a
solution that doesn't use this module.
* The platform is Windows and I'm happy with a Windoze only solution.
:)

I would like to execute subprocesses asynchronously and capture stdout
/ stderr as a single stream.

If I use "os.popen3(executable)" it gives me separate pipes for stdout
and stderr, but reads from them are blocking.

The output on stdout and stderr may be interleaved and I would like to
display them *as* they arrive. That means I can't just read from them
and output the results.

The only solution I can think of is to read from both a character at a
time on two separate threads, putting the data into a queue. A separate
thread could pull characters off the queue and display them. (I don't
need to differentiate between stdout and stderr when I display.)

Can anyone think of a better solution ?

My current code works, but *doesn't* capture stderr :

from threading import Thread

pipe = os.popen(executable)

def DisplayOutput():
while True:
output = pipe.read(1)
if not output:
break
display(output)

Thread(target=DisplayOutput).start()

All the best,

Fuzzymanhttp://www.voidspace.org.uk/python/articles.shtml

Try using popen4 instead. But since you already have a thread, it may
be better to use popen2 and two threads to read from stdout and stderr
to avoid a potential deadlock; they can put read lines into a Queue,
and DisplayOutput just get these lines in order. (See the warnings in
the popen2 module documentation).
 
F

Fuzzyman

Gabriel said:
Try using popen4 instead. But since you already have a thread, it may
be better to use popen2 and two threads to read from stdout and stderr
to avoid a potential deadlock; they can put read lines into a Queue,
and DisplayOutput just get these lines in order. (See the warnings in
the popen2 module documentation).

popen4 works great, I didn't even know it existed.

Two threads and a queue sounds horrible.

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/articles.shtml
 
G

Gabriel Genellina

Two threads and a queue sounds horrible.

But unfortunately it's the only way if you don't control how the child
process behaves.
(It's not soooo ugly afterwards... certainly would be worse if you had
to syncronize both reading threads and the display thread using
semaphores by hand.)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,832
Latest member
GlennSmall

Latest Threads

Top