Non-blocking pipes during subprocess handling

Discussion in 'Python' started by Tom Plunket, Jan 9, 2007.

  1. Tom Plunket

    Tom Plunket Guest

    I'm using subprocess to launch, well, sub-processes, but now I'm
    stumbling due to blocking I/O.

    Is there a way for me to know that there's data on a pipe, and possibly
    how much data is there so I can get it? Currently I'm doing this:

    process = subprocess.Popen(
    args,
    bufsize=1,
    universal_newlines=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE)

    def ProcessOutput(instream, outstream):
    text = instream.readline()
    if len(text) > 0:
    print >>outstream, text,
    return True
    else:
    return False

    while process.poll() is None:
    ProcessOutput(process.stdout, sys.stdout)
    ProcessOutput(process.stderr, sys.stderr)

    # clean up everything to EOF once the process ends.
    somethingPrinted = True
    while somethingPrinted:
    somethingPrinted = ProcessOutput(
    process.stdout, sys.stdout)
    somethingPrinted |= ProcessOutput(
    process.stderr, sys.stderr)


    Unfortunately, stream.readline will block 'til it gets a line, and
    typically there won't be anything on the stderr stream. The reason for
    the redirections in the first place is that I'm launching this script as
    a subprocess from a GUI app that catches stdout and stderr and directs
    the output to the appropriate windows, but in some cases I don't
    actually want the output at all (I've removed that logic though since it
    needlessly complicates my example; suffice to say everything below the
    process = subprocess.Popen... line is enclosed in a try and then in an
    if block.

    The documentation on file.read() indicate that there's an option for
    "non-blocking" mode, but I'm stumped as to how to even look for how to
    enable and use that.

    thanks,
    -tom!

    --
    Tom Plunket, Jan 9, 2007
    #1
    1. Advertising

  2. At Monday 8/1/2007 22:09, Tom Plunket wrote:

    >I'm using subprocess to launch, well, sub-processes, but now I'm
    >stumbling due to blocking I/O.
    >
    >Is there a way for me to know that there's data on a pipe, and possibly
    >how much data is there so I can get it? Currently I'm doing this:


    Using a thread for each stream is the safest way, specially if you
    can't control the child process.


    --
    Gabriel Genellina
    Softlab SRL






    __________________________________________________
    Preguntá. Respondé. Descubrí.
    Todo lo que querías saber, y lo que ni imaginabas,
    está en Yahoo! Respuestas (Beta).
    ¡Probalo ya!
    http://www.yahoo.com.ar/respuestas
    Gabriel Genellina, Jan 9, 2007
    #2
    1. Advertising

  3. Tom Plunket <> wrote:
    > I'm using subprocess to launch, well, sub-processes, but now I'm
    > stumbling due to blocking I/O.
    >
    > Is there a way for me to know that there's data on a pipe, and possibly
    > how much data is there so I can get it?


    You might want to check out this modification to subprocess which does
    non-blocking pipes.

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

    I personally think something like that should be built into subprocess

    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
    Nick Craig-Wood, Jan 9, 2007
    #3
  4. Tom Plunket

    Donn Cave Guest

    In article <>,
    Tom Plunket <> wrote:

    > I'm using subprocess to launch, well, sub-processes, but now I'm
    > stumbling due to blocking I/O.
    >
    > Is there a way for me to know that there's data on a pipe, and possibly
    > how much data is there so I can get it? Currently I'm doing this:
    >
    > process = subprocess.Popen(
    > args,
    > bufsize=1,
    > universal_newlines=True,
    > stdout=subprocess.PIPE,
    > stderr=subprocess.PIPE)
    >
    > def ProcessOutput(instream, outstream):
    > text = instream.readline()
    > if len(text) > 0:
    > print >>outstream, text,
    > return True
    > else:
    > return False



    I think it would be fair to say that your problem is
    not due to blocking I/O, so much as buffered I/O. Since
    you don't appear to need to read one line at a time, you
    can detect and read data from the file descriptor without
    any buffering. Don't mix with buffered I/O, as this will
    throw the select off. From memory - better check, since
    it has been a while since I wrote anything real like this
    (or for that matter much of anything in Python) --


    import select
    def ProcessOutput(instream, outstream):
    fdr = [instream.fileno()]
    (r, w, e) = select.select(fdr, [], [], 0.0)
    for fd in r:
    text = os.read(fd, 4096)
    outstream.write(text)

    Donn Cave,
    Donn Cave, Jan 9, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    12,516
    Allan Herriman
    Apr 8, 2004
  2. Andre Kelmanson

    blocking i/o vs. non blocking i/o (performance)

    Andre Kelmanson, Oct 10, 2003, in forum: C Programming
    Replies:
    3
    Views:
    922
    Valentin Tihomirov
    Oct 12, 2003
  3. nukleus
    Replies:
    14
    Views:
    827
    Chris Uppal
    Jan 22, 2007
  4. Sébastien Cottalorda

    Pipes and non blocking writing

    Sébastien Cottalorda, Apr 20, 2004, in forum: Perl Misc
    Replies:
    2
    Views:
    200
    Sébastien Cottalorda
    Apr 20, 2004
  5. john
    Replies:
    7
    Views:
    244
    Brian McCauley
    Mar 4, 2005
Loading...

Share This Page