Reading stdout & stderr from a pipe with popen3

Discussion in 'Ruby' started by Matt Mencel, Jul 3, 2008.

  1. Matt Mencel

    Matt Mencel Guest

    I originally posted this to a Ruby forum at railsforum.com, but I just found this list so thought I'd try here since I haven't gotten a response at railsforum.

    OK...so I'm using popen3 to run some command line stuff and be able to get back stdout/stderr...


    $cmdin, $cmdout, $cmderr = Open3.popen3("zmprov")

    zmprov opens a new command prompt to which I can send more commands to and get back output...

    $cmdin.puts("sm username")
    $cmdin.puts("sm")
    count = 0
    $cmdout.each |line|
    count += 1 if line.include?("\n")
    puts line
    break if count == 2 # IF I FIND 2 BLANK LINES IN stdout, BREAK THE LOOP
    end

    Which works just fine. However, I've run into a snag. $cmdout and $cmderr are pipes that never close until I send the "quit" command...so in order to break out of my loops I've got to jump through some extra hoops to look for specific patterns in the output. In the above example I have to send an extra "sm" command without the username, and in that output I can find the two blank lines...so I know it's done and can drop out of the loop.

    It's a pain, but I can do that. The real problem becomes what to do when a command returns something in stderr? Because stdout returns nothing...there's nothing to check for to drop out of that loop before I check for stderr....check the example below.

    $cmdin.puts("sm BADUSERNAME") # PRODUCES AN ERROR
    $cmdout.each |line|
    puts line
    end
    $cmderr.each |line|
    puts line
    end

    If the command produces an error....I never get out of the stdout loop to grab the error because stdout is returning me nothing...it's just an open pipe with no data in it until I run a command that produces stdout. So I never get where I can read stderr.

    A coworker suggested I run the original "zmprov" command as "zmprov 2>&1" so that stderr is returned in stdout. Which will work...but I was just wondering if I could get the code to work as designed. If stdout is returning data...read it and do something....if stderr is returning data then do something with that. I'm wanting to keep one from blocking my ability to read the other.


    After some further research and a little testing, I've found that using IO.readpartial (http://www.noobkit.com/show/ruby/ruby/ruby-core/io/readpartial.html) might be what I'm looking for, but I'm not sure. So I can do something like...

    $cmdin.readpartial(4096)

    ...and it will grab all the data in the pipe (up to 4096 bytes I think) and when it runs out of stuff to read it returns control back to the program. What happens though if there is more data waiting in the pipe than I call for with my IO.readpartial(maxlen) argument? Will it still return control or will it keep reading 4096 blocks until it runs out and THEN return control? I think that is how I would want it to handle.

    There is almost no examples out there about IO.readpartial, so I'm hoping someone here can help point me in the right direction.

    Thanks,
    Matt
     
    Matt Mencel, Jul 3, 2008
    #1
    1. Advertising

  2. Matt Mencel

    ara.t.howard Guest

    On Jul 3, 2008, at 2:07 PM, Matt Mencel wrote:

    > A coworker suggested I run the original "zmprov" command as "zmprov
    > 2>&1" so that stderr is returned in stdout. Which will work...but I
    > was just wondering if I could get the code to work as designed. If
    > stdout is returning data...read it and do something....if stderr is
    > returning data then do something with that. I'm wanting to keep one
    > from blocking my ability to read the other.



    the issue is even worse than you describe, the program can easily
    become blocked if it's stdout or stderr pipes get full - to fix the
    situation you need to use threads, one processing both stdout and
    stderr asynchronously where each may have to trigger actions on
    stdin. the general pattern is


    q = Queue.new

    err = Thread.new do
    Thread.current.abort_on_exception = true

    while(( line = stderr.gets ))
    ...
    q.push :somthing if some_condition_on(line)
    end
    q.push :stderr_done
    end

    out = Thread.new do
    Thread.current.abort_on_exception = true

    while(( line = stdout.gets ))
    ...
    q.push :something if some_condition_on(line)
    end
    q.push :stdout_done
    end

    in = Thread.new do
    Thread.current.abort_on_exception

    while(( command = q.pop ))
    ...
    break if stdout_done and stderr_done
    end
    end

    in.join

    so basically have one thread sending commands down stdin. start a
    thread each for stdout and stderr, each doing their own processing, if
    they encounter something which means input needs to be send push it
    onto a queue to allow the stdin thread to do it on their behave. this
    of course ignores exceptional conditions and coordination between the
    stdout and stderr threads, but it's one approach.


    the big conditions any solution needs to handle are having no output
    on either stderr or stdout or being blocked on a write to either due a
    full pipe, which is why this cannot work safely:

    loop do
    handle stdout.gets
    handle stderr.gets
    end

    check out open4 and session for examples of using threads to process
    both stdout and stderr concurrently.

    http://codeforpeople.com/lib/ruby/
    # gem install open4 session

    cheers.


    a @ http://codeforpeople.com/
    --
    we can deny everything, except that we have the possibility of being
    better. simply reflect on that.
    h.h. the 14th dalai lama
     
    ara.t.howard, Jul 7, 2008
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ladygrinningsoul

    Reading stdout and stderr separately

    ladygrinningsoul, Nov 18, 2004, in forum: Perl
    Replies:
    3
    Views:
    1,046
    Alexey A. Kiritchun
    Apr 25, 2005
  2. Elad
    Replies:
    0
    Views:
    415
  3. Christoph Haas
    Replies:
    0
    Views:
    391
    Christoph Haas
    Jun 13, 2006
  4. rantingrick
    Replies:
    3
    Views:
    429
    rantingrick
    Feb 1, 2009
  5. Ivan Novick

    pipe for stderr and stdout

    Ivan Novick, Mar 26, 2008, in forum: Perl Misc
    Replies:
    4
    Views:
    119
    Ted Zlatanov
    Mar 27, 2008
Loading...

Share This Page