System commands with streaming output

[Alex Wayne]

I have a long running system command that I want to print the output
from a ruby script. Basically, this system command runs like a daemon
and prints output. So it I just use:

`./mycommand foo`

The output is completely supressed, and my ruby script simply appears to
hang.

So how can I execute a system command and have the output streamed back
to the terminal before the command finishes?

 

[Tim Hunter]

I have a long running system command that I want to print the output
from a ruby script. Basically, this system command runs like a daemon
and prints output. So it I just use:

`./mycommand foo`

The output is completely supressed, and my ruby script simply appears to
hang.

So how can I execute a system command and have the output streamed back
to the terminal before the command finishes?

ri IO:open

 

[Alex Wayne]

ri IO:open

Hmm, I still can't seem to get this working. I got this:

IO.popen './mycommand foo' do |f|
puts f.gets
end

It's doing its job, but I still get no output on the terminal. And to
be clear, when this command is run on its own I definitely get output.

 

[Tim Hunter]

Hmm, I still can't seem to get this working. I got this:

IO.popen './mycommand foo' do |f|
puts f.gets
end

It's doing its job, but I still get no output on the terminal. And to
be clear, when this command is run on its own I definitely get output.

Possibly the command is writing to stderr instead of stdout. What
happens when you do this?

/mycommand foo >mycommand.stdout 2>mycommand.stderr

Does mycommand.stdout have the output, or mycommand.stderr? If the
latter, then the command is writing to stderr, and you're going to have
to use popen3 instead of plain popen:
http://ruby-doc.org/stdlib/libdoc/open3/rdoc/index.html, or if you don't
need to distinguish between the two, you could simply re-direct stderr
to stdout. I'm doing this from memory but I believe this should work:

IO.popen './mycommand foo >2&1' do...

 

[Alex Wayne]

Possibly the command is writing to stderr instead of stdout. What
happens when you do this?

./mycommand foo >mycommand.stdout 2>mycommand.stderr

pretty sure thats not it. Running that gives me no output at all.

I tried a super simple scenario.

test.rb:
#!/usr/local/bin/ruby
IO.popen('./foo.rb') { |f| puts f.gets }

foo.rb:
#!/usr/local/bin/ruby
10.times do |i|
puts i
sleep 0.5
end

When I run "./foo.rb" I get a slow count to 10, as expected.

When I run "./test.rb" I get a 5 second pause followed by a "0", then it
exits.

So it looks the block passed to popen isnt even getting called until the
command that it invokes exits. Then the variable passed into the block
is the entire output of that command?

 

[Tim Hunter]

So it looks the block passed to popen isnt even getting called until the
command that it invokes exits. Then the variable passed into the block
is the entire output of that command?

It seems possible that there is some buffering going on. Perhaps
somebody else will have a better idea.

 

[Jens Wille]

hi alex!

i'm sorry, i didn't follow the whole thread, but you probably want
to modify your simple example like so:

---- [ foo.rb ] ----
# immediately flush all output to the underlying OS
STDOUT.sync = true

10.times do |i|
puts i
sleep 0.5
end
--------------------

---- [ test.rb ] ----
IO.popen('./foo.rb') { |f|
until f.eof?
puts f.gets
end
}
---------------------

Alex Wayne [2008-04-08 22:29]:

So it looks the block passed to popen isnt even getting called
until the command that it invokes exits.

well, output is buffered first. only when that buffer is filled,
your receiving end of the pipe will see it. which, in this case,
coincides with the program's exit. if you don't want buffering to
happen, you have to set the sync mode to true (see [1]).

Then the variable passed into the block is the entire output of
that command?

it just reads the first line from that output, that's what IO#gets
does (see [2]).

[1] <http://www.ruby-doc.org/core/classes/IO.html#M002288>
[2] <http://www.ruby-doc.org/core/classes/IO.html#M002297>

hth
jens

--
Jens Wille, Dipl.-Bibl. (FH)
prometheus - Das verteilte digitale Bildarchiv für Forschung & Lehre
Kunsthistorisches Institut der Universität zu Köln
Albertus-Magnus-Platz, D-50923 Köln
Tel.: +49 (0)221 470-6668, E-Mail: (e-mail address removed)
http://www.prometheus-bildarchiv.de/

 

[Ezra Zygmuntowicz]

It seems possible that there is some buffering going on. Perhaps
somebody else will have a better idea.

add this to the top of your script:

$stdout.sync = true


That will stop it from buffering and will flush the output every time
you call puts

Cheers-

- Ezra Zygmuntowicz
-- Founder & Software Architect
-- (e-mail address removed)
-- EngineYard.com

 

[Alex Wayne]

Thanks guys. You all rock.