Interprocess communication woes

[Murali]

Hi Python Gurus,

I am writing a GUI app (on linux) using pygtk which would launch some
external applications and display their stdout and stderr inside the
output window of my application synchronously. I am using the
subprocess module's Popen to launch the external programs and to
capture their stdout and stderr. The problem is that, for some
external programs that I launch inside my interface, I am not able to
capture and display the stdout as the program *runs*.

After some investigation, I found out that this problem had nothing to
do with my GUI app not getting refreshed and I was able to reproduce
this problem with normal python scripts. Here is one such script

#File test.py
from subprocess import Popen
from subprocess import PIPE
import sys
if __name__ == '__main__':
prog = sys.argv[1]
proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
out = proc.stdout
while proc.poll() is None:
print out.readline()

Run this program as follows
$ test.py "ping -c 10 www.google.com"

Now, you would see the responses as if you just launched ping on the
command line. Now, lets look at another program. Here is a simple C++
program that prints numbers 1 to 10 at the passage of every second
(sort of a stopwatch)

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
main ( )
{
timeval t1, t2;
gettimeofday(&t1, NULL);
int prev = -1;
int cur = 0;
while (true)
{
gettimeofday(&t2,NULL);
if(t2.tv_sec - t1.tv_sec > 10)
break;
else
{
cur = t2.tv_sec-t1.tv_sec;
if (cur != prev)
{
printf("%d\r\n",cur);
prev = cur;
}
}
}
}

if you would build this program and call it lets say timer ($ g++ -o
timer timer.cpp) and run it with our python script like this

$python test.py "./timer"

you would see that every time you run the program your results vary
and on top of this the stdout of the timer program gets displayed all
at once presumably when the timer program has completed execution.

Why this discrepancy between the ping and timer programs? Is my
test.py script correct? Is there a better or a preferred method for
doing interprocess communication in Python.

Thanks!
Murali.

 

[Nick Craig-Wood]

After some investigation, I found out that this problem had nothing to
do with my GUI app not getting refreshed and I was able to reproduce
this problem with normal python scripts. Here is one such script

#File test.py
from subprocess import Popen
from subprocess import PIPE
import sys
if __name__ == '__main__':
prog = sys.argv[1]
proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
out = proc.stdout
while proc.poll() is None:
print out.readline()

Run this program as follows
$ test.py "ping -c 10 www.google.com"

Now, you would see the responses as if you just launched ping on the
command line. Now, lets look at another program. Here is a simple C++
program that prints numbers 1 to 10 at the passage of every second
(sort of a stopwatch)

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
main ( )
{
timeval t1, t2;
gettimeofday(&t1, NULL);
int prev = -1;
int cur = 0;
while (true)
{
gettimeofday(&t2,NULL);
if(t2.tv_sec - t1.tv_sec > 10)
break;
else
{
cur = t2.tv_sec-t1.tv_sec;
if (cur != prev)
{
printf("%d\r\n",cur);
prev = cur;
}
}
}
}

if you would build this program and call it lets say timer ($ g++ -o
timer timer.cpp) and run it with our python script like this

$python test.py "./timer"

you would see that every time you run the program your results vary
and on top of this the stdout of the timer program gets displayed all
at once presumably when the timer program has completed execution.

Why this discrepancy between the ping and timer programs? Is my
test.py script correct? Is there a better or a preferred method for
doing interprocess communication in Python.

Buffering is your problem.

If you add a fflush(stdout); after the printf(...); you'll find the
c++ program works as you expect.

It is just a fact of life of the C stdio system. If it is connected
to a terminal then it will turn off buffering. If it is connected
anything else (eg a pipe via subprocess) then it will buffer stuff as
you've seen.

So you can

a) modify the c++ prog to add fflush() in or use setvbuf()
b) use the pexpect module - http://pexpect.sourceforge.net/
c) use the pty module (unix only)

The pexpect module will connect to the subprogram with pseudo-ttys,
fooling the program, and the C library, into thinking that it is
speaking to a terminal and turn off buffering. Pexpect doesn't work
on windows.

The fact that ping works is because it uses fflush() - you can see
this if you "ltrace" it.

 

[Murali]

After some investigation, I found out that this problem had nothing to
do with my GUI app not getting refreshed and I was able to reproduce
this problem with normal python scripts. Here is one such script

#File test.py
from subprocess import Popen
from subprocess import PIPE
import sys
if __name__ == '__main__':
prog = sys.argv[1]
proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
out = proc.stdout
while proc.poll() is None:
print out.readline()

Run this program as follows
$ test.py "ping -c 10www.google.com"

Now, you would see the responses as if you just launched ping on the
command line. Now, lets look at another program. Here is a simple C++
program that prints numbers 1 to 10 at the passage of every second
(sort of a stopwatch)

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
main ( )
{
timeval t1, t2;
gettimeofday(&t1, NULL);
int prev = -1;
int cur = 0;
while (true)
{
gettimeofday(&t2,NULL);
if(t2.tv_sec - t1.tv_sec > 10)
break;
else
{
cur = t2.tv_sec-t1.tv_sec;
if (cur != prev)
{
printf("%d\r\n",cur);
prev = cur;
}
}
}
}

if you would build this program and call it lets say timer ($ g++ -o
timer timer.cpp) and run it with our python script like this

$python test.py "./timer"

you would see that every time you run the program your results vary
and on top of this the stdout of the timer program gets displayed all
at once presumably when the timer program has completed execution.

Why this discrepancy between the ping and timer programs? Is my
test.py script correct? Is there a better or a preferred method for
doing interprocess communication in Python.

Buffering is your problem.

If you add a fflush(stdout); after the printf(...); you'll find the
c++ program works as you expect.

It is just a fact of life of the C stdio system. If it is connected
to a terminal then it will turn off buffering. If it is connected
anything else (eg a pipe via subprocess) then it will buffer stuff as
you've seen.

So you can

Thanks Nick. fflush fixed it. Thanks for your pointers on pexpect and
pty module too.

Murali.