get text from rogramms runn by subprocess.Popen immediatetly

R

Rüdiger Ranft

Hi all,

I need to call some programms and catch their stdout and stderr streams.
While the Popen class from subprocess handles the call, I get the
results of the programm not until the programm finishes. Since the
output of the programm is used to generate a progress indicator, I need
a way to acces the values written to stdout/stderr as fast as possible.

Beneath is a test which shows what I did

TIA
Rudi

----8<-------8<-------8<-- iodummy.cpp -8<-------8<---
#include <iostream>
#include <unistd.h>

int main()
{
for( int i = 0; i < 10; i++ )
{
std::cerr << i << std::endl;
sleep(2);
}
}

from subprocess import Popen, PIPE
from time import sleep

p = Popen('./iodummy',stdin=PIPE, stdout=PIPE, stderr=PIPE)
sleep(3)
# now I expect '0\n1\n' in stderr, but read() blocks until
# the end of iodummy.
print p.stderr.read()
p.wait()

--
GPG encrypted mails preferred.
GPG verschlüsselte Mails bevorzugt.
---> http://chaosradio.ccc.de/media/ds/ds085.pdf Seite 20 <----


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIcBAEBAwAGBQJJ5xKqAAoJEI+IiZON95WTR5YP/iymEmF37qbnnMSSQZFs629Q
sSLrcjNytEmuQZvYdvsrJJ2LiR4NHzLTC+zVVP41VsOwfydfAFZhWBuKZTN2pqUT
jfZAYM1KALoy0f9Z9gD08ehTwrXy5+1ZRmj3aLmdqSmD6Jqjfnz3495VivB2f9JY
aYT8ZGglikVDeiTO8GekA7Lyfq+DItZ8jXcSxlQqiz6ega0DapL6NHH6c9d5nx+F
qAc7/JERZsMNd8EMDp6tb8xDVFp/ZIzSS3aYFRmQHfdE8lSp5DG2868gzEVU/h5Y
0D25gs/tAmuYFyutMq1GulDwLcpZWFqjidibav6M6XabvL45QcsKrsAlDQrLin/x
JwDct72lgseh2JrPx4Kcdz1/+4uppMuPYbnJ1YMTsGxthMXyP0Wk2RVITrVHoH2P
j4GwuNPGiEu3rkNutUUer4wkQVUnss/D71jNYXl9SgmoMsq9Ub66p8GO9vepgWlw
NK/7U5iad96ETHJgcvWhx8o/ymr8V6kOBqQpLlRCNsd/2MRQvoT61tmK0UDdx2lL
x/LrhZftbAiVN5hzzAC2H6nX02m9bipfqWKbzvp71wbLZufXAo2R8jSmibrdUQff
16TUVeIhUY4YIuIscG6YyTMit5tEUTtCL2Mi3yTFe8R8uOR3h/1t1DbhDBYRtQAh
Sa8efc/lkb8gLUJU3nnz
=xaSm
-----END PGP SIGNATURE-----
 
D

Diez B. Roggisch

Rüdiger Ranft said:
Hi all,

I need to call some programms and catch their stdout and stderr streams.
While the Popen class from subprocess handles the call, I get the
results of the programm not until the programm finishes. Since the
output of the programm is used to generate a progress indicator, I need
a way to acces the values written to stdout/stderr as fast as possible.

Use the communicate()-method of Popen-objects.

Diez
 
R

Rüdiger Ranft

Diez said:
Use the communicate()-method of Popen-objects.

It gives the same behavior, the first call to communicate gets all text,
the second gives a closed handle error :(. I also tried
p.communicate(''), which gives the same as p.communicate(None) result.

bye
Rudi

cat run.py
from subprocess import Popen, PIPE
from time import sleep

p = Popen('./iodummy',stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=2)
sleep(3)
# now I expect '0\n1\n' in stderr
print 'Point1:', p.communicate(None)
sleep(3)
print 'Point2:', p.communicate(None)
p.wait()

python run.py
Point1: ('', '0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n')
Point2:
Traceback (most recent call last):
File "run.py", line 9, in ?
print 'Point2:', p.communicate('')
File "/usr/lib/python2.4/subprocess.py", line 1028, in communicate
self.stdin.flush()
ValueError: I/O operation on closed file
 
D

Diez B. Roggisch

Rüdiger Ranft said:
It gives the same behavior, the first call to communicate gets all text,
the second gives a closed handle error :(. I also tried
p.communicate(''), which gives the same as p.communicate(None) result.

That's likely due to buffering then. You could try & see if flushing on
the side of the subprocess works (you can't do that from python's side
of things within this scenario).

Or you might want to use pexpect that emulates a pseudo-terminal which
also changes the way data is displayed.

Diez
 
G

grocery_stocker

Hi all,

I need to call some programms and catch their stdout and stderr streams.
While the Popen class from subprocess handles the call, I get the
results of the programm not until the programm finishes. Since the
output of the programm is used to generate a progress indicator, I need
a way to acces the values written to stdout/stderr as fast as possible.

Beneath is a test which shows what I did

TIA
Rudi

----8<-------8<-------8<-- iodummy.cpp -8<-------8<---
#include <iostream>
#include <unistd.h>

int main()
{
for( int i = 0; i < 10; i++ )
{
std::cerr << i << std::endl;
sleep(2);
}

}

from subprocess import Popen, PIPE
from time import sleep

p = Popen('./iodummy',stdin=PIPE, stdout=PIPE, stderr=PIPE)
sleep(3)
# now I expect '0\n1\n' in stderr, but read() blocks until
# the end of iodummy.
print p.stderr.read()
p.wait()

I'm sure the regulars will correct me, but if are on some kind of *nix
variant, couldn't you just redirect the stderr to the stdout like the
following

[cdalten@localhost oakland]$ more nope2.py
#!/usr/bin/python

import os
os.popen('./iodummy 2>&1', 'w')
[cdalten@localhost oakland]$ ./nope2.py
0
1
2
3
4
5
6
7
8
9
[cdalten@localhost oakland]$
 
P

Piet van Oostrum

Rüdiger Ranft said:
RR> Hi all,
RR> I need to call some programms and catch their stdout and stderr streams.
RR> While the Popen class from subprocess handles the call, I get the
RR> results of the programm not until the programm finishes. Since the
RR> output of the programm is used to generate a progress indicator, I need
RR> a way to acces the values written to stdout/stderr as fast as possible.
RR> print p.stderr.read()

Use p.stderr.readline() or p.stderr.read(1) in a loop.
 
A

Aaron Brady

snip

FWIR from what I recall, you have to create and open read and write
file descriptors using the 'msvcrt.get_osfhandle' function, and pass
them to the subprocess on the command line. Or you can pass the
handles. Then you get a communication channel that is not STDIN.

I have no idea why the designers did this. I've submitted a patch,
and received 0 attention on it.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top