Reading stdout & stderr from a pipe with popen3

[Matt Mencel]

I originally posted this to a Ruby forum at railsforum.com, but I just found this list so thought I'd try here since I haven't gotten a response at railsforum.

OK...so I'm using popen3 to run some command line stuff and be able to get back stdout/stderr...


$cmdin, $cmdout, $cmderr = Open3.popen3("zmprov")

zmprov opens a new command prompt to which I can send more commands to and get back output...

$cmdin.puts("sm username")
$cmdin.puts("sm")
count = 0
$cmdout.each |line|
count += 1 if line.include?("\n")
puts line
break if count == 2 # IF I FIND 2 BLANK LINES IN stdout, BREAK THE LOOP
end

Which works just fine. However, I've run into a snag. $cmdout and $cmderr are pipes that never close until I send the "quit" command...so in order to break out of my loops I've got to jump through some extra hoops to look for specific patterns in the output. In the above example I have to send an extra "sm" command without the username, and in that output I can find the two blank lines...so I know it's done and can drop out of the loop.

It's a pain, but I can do that. The real problem becomes what to do when a command returns something in stderr? Because stdout returns nothing...there's nothing to check for to drop out of that loop before I check for stderr....check the example below.

$cmdin.puts("sm BADUSERNAME") # PRODUCES AN ERROR
$cmdout.each |line|
puts line
end
$cmderr.each |line|
puts line
end

If the command produces an error....I never get out of the stdout loop to grab the error because stdout is returning me nothing...it's just an open pipe with no data in it until I run a command that produces stdout. So I never get where I can read stderr.

A coworker suggested I run the original "zmprov" command as "zmprov 2>&1" so that stderr is returned in stdout. Which will work...but I was just wondering if I could get the code to work as designed. If stdout is returning data...read it and do something....if stderr is returning data then do something with that. I'm wanting to keep one from blocking my ability to read the other.


After some further research and a little testing, I've found that using IO.readpartial (http://www.noobkit.com/show/ruby/ruby/ruby-core/io/readpartial.html) might be what I'm looking for, but I'm not sure. So I can do something like...

$cmdin.readpartial(4096)

...and it will grab all the data in the pipe (up to 4096 bytes I think) and when it runs out of stuff to read it returns control back to the program. What happens though if there is more data waiting in the pipe than I call for with my IO.readpartial(maxlen) argument? Will it still return control or will it keep reading 4096 blocks until it runs out and THEN return control? I think that is how I would want it to handle.

There is almost no examples out there about IO.readpartial, so I'm hoping someone here can help point me in the right direction.

Thanks,
Matt

 

[ara.t.howard]

A coworker suggested I run the original "zmprov" command as "zmprov
2>&1" so that stderr is returned in stdout. Which will work...but I
was just wondering if I could get the code to work as designed. If
stdout is returning data...read it and do something....if stderr is
returning data then do something with that. I'm wanting to keep one
from blocking my ability to read the other.

the issue is even worse than you describe, the program can easily
become blocked if it's stdout or stderr pipes get full - to fix the
situation you need to use threads, one processing both stdout and
stderr asynchronously where each may have to trigger actions on
stdin. the general pattern is


q = Queue.new

err = Thread.new do
Thread.current.abort_on_exception = true

while(( line = stderr.gets ))
...
q.push :somthing if some_condition_on(line)
end
q.push :stderr_done
end

out = Thread.new do
Thread.current.abort_on_exception = true

while(( line = stdout.gets ))
...
q.push :something if some_condition_on(line)
end
q.push :stdout_done
end

in = Thread.new do
Thread.current.abort_on_exception

while(( command = q.pop ))
...
break if stdout_done and stderr_done
end
end

in.join

so basically have one thread sending commands down stdin. start a
thread each for stdout and stderr, each doing their own processing, if
they encounter something which means input needs to be send push it
onto a queue to allow the stdin thread to do it on their behave. this
of course ignores exceptional conditions and coordination between the
stdout and stderr threads, but it's one approach.


the big conditions any solution needs to handle are having no output
on either stderr or stdout or being blocked on a write to either due a
full pipe, which is why this cannot work safely:

loop do
handle stdout.gets
handle stderr.gets
end

check out open4 and session for examples of using threads to process
both stdout and stderr concurrently.

http://codeforpeople.com/lib/ruby/
# gem install open4 session

cheers.


a @ http://codeforpeople.com/