Interprocess communication woes

Discussion in 'Python' started by Murali, Jul 19, 2007.

  1. Murali

    Murali Guest

    Hi Python Gurus,

    I am writing a GUI app (on linux) using pygtk which would launch some
    external applications and display their stdout and stderr inside the
    output window of my application synchronously. I am using the
    subprocess module's Popen to launch the external programs and to
    capture their stdout and stderr. The problem is that, for some
    external programs that I launch inside my interface, I am not able to
    capture and display the stdout as the program *runs*.

    After some investigation, I found out that this problem had nothing to
    do with my GUI app not getting refreshed and I was able to reproduce
    this problem with normal python scripts. Here is one such script

    #File test.py
    from subprocess import Popen
    from subprocess import PIPE
    import sys
    if __name__ == '__main__':
    prog = sys.argv[1]
    proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
    out = proc.stdout
    while proc.poll() is None:
    print out.readline()

    Run this program as follows
    $ test.py "ping -c 10 www.google.com"

    Now, you would see the responses as if you just launched ping on the
    command line. Now, lets look at another program. Here is a simple C++
    program that prints numbers 1 to 10 at the passage of every second
    (sort of a stopwatch)

    #include <stdio.h>
    #include <stdlib.h>
    #include <sys/time.h>
    main ( )
    {
    timeval t1, t2;
    gettimeofday(&t1, NULL);
    int prev = -1;
    int cur = 0;
    while (true)
    {
    gettimeofday(&t2,NULL);
    if(t2.tv_sec - t1.tv_sec > 10)
    break;
    else
    {
    cur = t2.tv_sec-t1.tv_sec;
    if (cur != prev)
    {
    printf("%d\r\n",cur);
    prev = cur;
    }
    }
    }
    }

    if you would build this program and call it lets say timer ($ g++ -o
    timer timer.cpp) and run it with our python script like this

    $python test.py "./timer"

    you would see that every time you run the program your results vary
    and on top of this the stdout of the timer program gets displayed all
    at once presumably when the timer program has completed execution.

    Why this discrepancy between the ping and timer programs? Is my
    test.py script correct? Is there a better or a preferred method for
    doing interprocess communication in Python.

    Thanks!
    Murali.
     
    Murali, Jul 19, 2007
    #1
    1. Advertising

  2. Murali <> wrote:
    > After some investigation, I found out that this problem had nothing to
    > do with my GUI app not getting refreshed and I was able to reproduce
    > this problem with normal python scripts. Here is one such script
    >
    > #File test.py
    > from subprocess import Popen
    > from subprocess import PIPE
    > import sys
    > if __name__ == '__main__':
    > prog = sys.argv[1]
    > proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
    > out = proc.stdout
    > while proc.poll() is None:
    > print out.readline()
    >
    > Run this program as follows
    > $ test.py "ping -c 10 www.google.com"
    >
    > Now, you would see the responses as if you just launched ping on the
    > command line. Now, lets look at another program. Here is a simple C++
    > program that prints numbers 1 to 10 at the passage of every second
    > (sort of a stopwatch)
    >
    > #include <stdio.h>
    > #include <stdlib.h>
    > #include <sys/time.h>
    > main ( )
    > {
    > timeval t1, t2;
    > gettimeofday(&t1, NULL);
    > int prev = -1;
    > int cur = 0;
    > while (true)
    > {
    > gettimeofday(&t2,NULL);
    > if(t2.tv_sec - t1.tv_sec > 10)
    > break;
    > else
    > {
    > cur = t2.tv_sec-t1.tv_sec;
    > if (cur != prev)
    > {
    > printf("%d\r\n",cur);
    > prev = cur;
    > }
    > }
    > }
    > }
    >
    > if you would build this program and call it lets say timer ($ g++ -o
    > timer timer.cpp) and run it with our python script like this
    >
    > $python test.py "./timer"
    >
    > you would see that every time you run the program your results vary
    > and on top of this the stdout of the timer program gets displayed all
    > at once presumably when the timer program has completed execution.
    >
    > Why this discrepancy between the ping and timer programs? Is my
    > test.py script correct? Is there a better or a preferred method for
    > doing interprocess communication in Python.


    Buffering is your problem.

    If you add a fflush(stdout); after the printf(...); you'll find the
    c++ program works as you expect.

    It is just a fact of life of the C stdio system. If it is connected
    to a terminal then it will turn off buffering. If it is connected
    anything else (eg a pipe via subprocess) then it will buffer stuff as
    you've seen.

    So you can

    a) modify the c++ prog to add fflush() in or use setvbuf()
    b) use the pexpect module - http://pexpect.sourceforge.net/
    c) use the pty module (unix only)

    The pexpect module will connect to the subprogram with pseudo-ttys,
    fooling the program, and the C library, into thinking that it is
    speaking to a terminal and turn off buffering. Pexpect doesn't work
    on windows.

    The fact that ping works is because it uses fflush() - you can see
    this if you "ltrace" it.

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
     
    Nick Craig-Wood, Jul 19, 2007
    #2
    1. Advertising

  3. Murali

    Murali Guest

    On Jul 19, 4:30 am, Nick Craig-Wood <> wrote:
    > Murali <> wrote:
    > > After some investigation, I found out that this problem had nothing to
    > > do with my GUI app not getting refreshed and I was able to reproduce
    > > this problem with normal python scripts. Here is one such script

    >
    > > #File test.py
    > > from subprocess import Popen
    > > from subprocess import PIPE
    > > import sys
    > > if __name__ == '__main__':
    > > prog = sys.argv[1]
    > > proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
    > > out = proc.stdout
    > > while proc.poll() is None:
    > > print out.readline()

    >
    > > Run this program as follows
    > > $ test.py "ping -c 10www.google.com"

    >
    > > Now, you would see the responses as if you just launched ping on the
    > > command line. Now, lets look at another program. Here is a simple C++
    > > program that prints numbers 1 to 10 at the passage of every second
    > > (sort of a stopwatch)

    >
    > > #include <stdio.h>
    > > #include <stdlib.h>
    > > #include <sys/time.h>
    > > main ( )
    > > {
    > > timeval t1, t2;
    > > gettimeofday(&t1, NULL);
    > > int prev = -1;
    > > int cur = 0;
    > > while (true)
    > > {
    > > gettimeofday(&t2,NULL);
    > > if(t2.tv_sec - t1.tv_sec > 10)
    > > break;
    > > else
    > > {
    > > cur = t2.tv_sec-t1.tv_sec;
    > > if (cur != prev)
    > > {
    > > printf("%d\r\n",cur);
    > > prev = cur;
    > > }
    > > }
    > > }
    > > }

    >
    > > if you would build this program and call it lets say timer ($ g++ -o
    > > timer timer.cpp) and run it with our python script like this

    >
    > > $python test.py "./timer"

    >
    > > you would see that every time you run the program your results vary
    > > and on top of this the stdout of the timer program gets displayed all
    > > at once presumably when the timer program has completed execution.

    >
    > > Why this discrepancy between the ping and timer programs? Is my
    > > test.py script correct? Is there a better or a preferred method for
    > > doing interprocess communication in Python.

    >
    > Buffering is your problem.
    >
    > If you add a fflush(stdout); after the printf(...); you'll find the
    > c++ program works as you expect.
    >
    > It is just a fact of life of the C stdio system. If it is connected
    > to a terminal then it will turn off buffering. If it is connected
    > anything else (eg a pipe via subprocess) then it will buffer stuff as
    > you've seen.
    >
    > So you can
    >

    Thanks Nick. fflush fixed it. Thanks for your pointers on pexpect and
    pty module too.

    Murali.


    > a) modify the c++ prog to add fflush() in or use setvbuf()
    > b) use the pexpect module -http://pexpect.sourceforge.net/
    > c) use the pty module (unix only)
    >
    > The pexpect module will connect to the subprogram with pseudo-ttys,
    > fooling the program, and the C library, into thinking that it is
    > speaking to a terminal and turn off buffering. Pexpect doesn't work
    > on windows.
    >
    > The fact that ping works is because it uses fflush() - you can see
    > this if you "ltrace" it.
    >
    > --
    > Nick Craig-Wood <> --http://www.craig-wood.com/nick
     
    Murali, Jul 20, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Swapnajit Mittra
    Replies:
    0
    Views:
    452
    Swapnajit Mittra
    Dec 21, 2004
  2. Dave Bartlett

    newbie question: interprocess communication

    Dave Bartlett, May 13, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    497
    DalePres
    May 13, 2004
  3. James  Aguilar
    Replies:
    3
    Views:
    560
    Aguilar, James
    Dec 20, 2005
  4. Michael Butscher
    Replies:
    7
    Views:
    354
    Lawrence D'Oliveiro
    Jul 1, 2006
  5. exhuma.twn
    Replies:
    23
    Views:
    1,047
    Steve Holden
    Feb 18, 2007
Loading...

Share This Page