Starting a child process and getting its stdout?

Discussion in 'Python' started by cypher543, Dec 29, 2006.

  1. cypher543

    cypher543 Guest

    This has been driving me insane for the last hour or so. I have search
    everywhere, and nothing works. I am trying to use the subprocess module
    to run a program and get its output line by line. But, it always waits
    for the process to terminate and then return the output all at once.

    Can someone please show me some code that actually works for this sort
    of thing? It doesn't even have to use the subprocess module. Don't
    worry if the code isn't compatible with Windows. My program is targeted
    at Linux/UNIX users.

    Thanks!
     
    cypher543, Dec 29, 2006
    #1
    1. Advertisements

  2. try:

    Python 2.5c1 (r25c1:51305, Aug 17 2006, 17:07:04)
    [GCC 3.3.6] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    For more info on how to do stdin and other things check out:
    http://docs.python.org/lib/module-subprocess.html

    Hope this helps.

    Adonis
     
    Adonis Vargas, Dec 29, 2006
    #2
    1. Advertisements

  3. cypher543

    Tom Plunket Guest

    Right, you need to use subprocess.PIPE and polling of the process and
    the pipe to get it to do what you want. Hopefully this makes sense, I
    also hope it'll work on Linux but can't imagine why it wouldn't.

    Save this code to a file, name unimportant. Executing the file will
    fire the top clause of the if statement, the bit that runs the "parent"
    part. That starts the script again as a subprocess, which triggers the
    else clause. Hope it makes sense.

    Depending on your environment, you might need 'shell=True' in your Popen
    args. On Windows, that is required if you want to prevent a console
    window from popping up if you're running in a GUI app.

    -tom!

    --

    import subprocess
    import sys
    import time

    if len(sys.argv) == 1:
    # no command line arg means, "we're the parent process."
    print 'starting parent process.'
    myName = sys.argv[0]

    # launch the subprocess with an additional parameter, to trigger
    # else clause below.
    p = subprocess.Popen(
    ( 'python', '-u', myName, 'x' ),
    stdout=subprocess.PIPE
    )

    while p.poll() == None:
    data = p.stdout.readline()
    if data:
    print data,

    print 'process ended with return code', p.returncode

    else:
    # assume the command-line arg means "run the background process"
    print 'starting subprocess'

    for i in range(10):
    time.sleep(.25)
    print i

    sys.exit(14)
     
    Tom Plunket, Dec 29, 2006
    #3
  4. cypher543

    Tom Plunket Guest

    If you change that print to something more decorated, like,

    print 'process said:', data,

    Then it might be more obvious where/how the print statements are getting
    routed.

    Keep in mind that readline() will return one line at a time, and that
    line will end with a newline, although the last line you get (at EOF)
    may not have one.

    again, good luck. Hope this helps,
    -tom!

    --
     
    Tom Plunket, Dec 29, 2006
    #4
  5. cypher543

    cypher543 Guest

    Thank you for the examples, but I have tried all of that before. No
    matter what I do, my program always hangs while it waits for the
    process to exit and then it prints all of the output at once. I've
    tried read(), readline(), readlines(), communicate(), etc and it is
    always the same.

    self.buildPID = subprocess.Popen(["python", "tobeforked.py"], stdout =
    subprocess.PIPE)
    while self.buildPID.poll() == None:
    output = self.buildPID.stdout.readline()
    self.consoleLogBuffer.insert(self.consoleLogBuffer.get_end_iter(),
    output)
    self.consoleLog.scroll_to_mark(self.consoleLogBuffer.get_insert(), 0)

    Keep in mind that I'm not required to use subprocess. But I have also
    tried os.fork and the pty module. They both produce the exact same
    results.
     
    cypher543, Dec 29, 2006
    #5
  6. cypher543

    Tom Plunket Guest

    Did you try my example specifically?
    By default, python will execute in buffered mode if it's attached to a
    pipe. Start it with the '-u' command line option and you may be good to
    go. (`python -h` for more info on this, man pages may also discuss it.)

    -tom!

    --
     
    Tom Plunket, Dec 29, 2006
    #6
  7. Did you *actually* tried what Tom Plunket posted? Two tiny chars make
    a difference.


    --
    Gabriel Genellina
    Softlab SRL






    __________________________________________________
    Preguntá. Respondé. Descubrí.
    Todo lo que querías saber, y lo que ni imaginabas,
    está en Yahoo! Respuestas (Beta).
    ¡Probalo ya!
    http://www.yahoo.com.ar/respuestas
     
    Gabriel Genellina, Dec 30, 2006
    #7
  8. cypher543

    Tom Plunket Guest

    The sad irony is that before taking off for vacation I was struggling at
    work with the same problem in some sense. I couldn't figure out why for
    some processes I got all of the output right away and for others it all
    got queued up 'til the process ended. Working out a simple example for
    this thread showed the light: my calling of my_process.stdout.read() was
    blocking 'til EOF. Oops.

    -tom!

    --
     
    Tom Plunket, Dec 30, 2006
    #8
  9. cypher543

    cypher543 Guest

    Yes, I did try your example. But, after talking on the #python IRC
    channel, I realized that it wasn't the process that was blocking, it
    was my GUI. I had to fire an event using gobject.io_add_watch()
    whenever data was received from the child process. The event then read
    from the process and added a line to my gtk.TextView.

    Thank you for your suggestions, though.
     
    cypher543, Dec 30, 2006
    #9
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.