capture output in real time

[Paul Winward]

I'm wondering how to capture data written to stdout (or stderr) in real
time using popen3. The problem occurs with the following sample
scripts:

sleep.rb
--------
puts "before sleep"
sleep 3
puts "after sleep"


capture.rb
---------
require 'open3'

Open3.popen3("ruby sleep.rb") do |stdin, stdout, stderr|

Thread.new do
loop do
out = stdout.gets
puts out if out
end
end

sleep 5
end


capture.rb outputs both "before sleep" and "after sleep" after 3 seconds
have passed. When using stderr, however, output is seen in real time.

Note, I'm avoiding STDOUT.flush in sleep.rb since I won't be able to
change the actual scripts I'll be passing to popen3.

Thanks for any help.
Paul

 

[ara.t.howard]

Note, I'm avoiding STDOUT.flush in sleep.rb since I won't be able to
change the actual scripts I'll be passing to popen3.

popen3 is going to be using fork/exec under the hood and should
inherit the sync setting of STDOUT, so try setting

STDOUT.sync = true

before the popen and see if that helps - untested, but i'm running out
the door....

cheers.

a @ http://codeforpeople.com/

 

[Paul Winward]

Thanks for the suggestion but I'm still having the problem.

Paul

 

[Paul Winward]

Another approach I had tried was to use IO.select, as shown in
http://whynotwiki.com/Ruby_/_Process_management ( attributed to Jamis
Buck). This too, produced the same delay in output, but is shown below:

Thank you for any help

def read_until(pipe, stop_at, verbose = true)
lines = []
line = ""
while result = IO.select([pipe]) #, nil, nil, 10)
next if result.empty?

c = pipe.read(1)
break if c.nil?

line << c
break if line =~ stop_at

# Start a new line?
if line[-1] == ?\n
puts line if verbose
lines << line
line = ""
end
end
lines
end

 

[Paul Winward]

I found a perl solution at http://www.perlmonks.org/?node_id=305812 that
makes use of pseudo tty to get line-buffered output. Unfortunately, I
can find very little documentation on how to use the ruby module PTY,
and it looks like it is no longer supported
(http://www.ruby-forum.com/topic/134771#new). Also, I wonder if a
pseudo tty can capture stderr as well
(http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2003-07/0544.html)

The perl solution is system dependent, but it worked for me on Linux
2.6.22. I did see a debian package libpty-ruby but I don't see any
documentation and am not sure if this is still supported.

The notion of a pseudo tty is new to me and I'd appreciate any help. Is
there a ruby equivalent for the perl solution below?

Thanks

#!/usr/bin/perl -w
use strict;
use IO:ty;

my $pty = new IO:ty;
my $slave = $pty->slave();

my $pid = fork();
die "Couldn't fork: $!" unless defined $pid;

if($pid){ # dup STDOUT to Pty and run external program:
$pty->close_slave();
open(STDOUT, ">&",$pty)||die $!;
system "perl blackbox.pl";
print "\cD"; # send ^d to end

} else { # this is your monitoring process
$pty->make_slave_controlling_terminal();
print "*$_" while <$slave>;
exit;
}
__END__

Another approach I had tried was to use IO.select, as shown in
http://whynotwiki.com/Ruby_/_Process_management ( attributed to Jamis
Buck). This too, produced the same delay in output, but is shown below:

Thank you for any help

def read_until(pipe, stop_at, verbose = true)
lines = []
line = ""
while result = IO.select([pipe]) #, nil, nil, 10)
next if result.empty?

c = pipe.read(1)
break if c.nil?

line << c
break if line =~ stop_at

# Start a new line?
if line[-1] == ?\n
puts line if verbose
lines << line
line = ""
end
end
lines
end

Thanks for the suggestion but I'm still having the problem.

Paul

 

[ara.t.howard]

I found a perl solution at http://www.perlmonks.org/?node_id=305812
that
makes use of pseudo tty to get line-buffered output. Unfortunately, I
can find very little documentation on how to use the ruby module PTY,
and it looks like it is no longer supported
(http://www.ruby-forum.com/topic/134771#new). Also, I wonder if a
pseudo tty can capture stderr as well
(http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2003-07/0544.html
)

The perl solution is system dependent, but it worked for me on Linux
2.6.22. I did see a debian package libpty-ruby but I don't see any
documentation and am not sure if this is still supported.

The notion of a pseudo tty is new to me and I'd appreciate any
help. Is
there a ruby equivalent for the perl solution below?

pty is supported and the dist has a bunch of demo code in it (ext/pty/
*) but you cannot capture stderr akaikt.

a @ http://codeforpeople.com/

 

[ara.t.howard]

Thanks for the suggestion but I'm still having the problem.

if the child program makes a call to setvbuf (as ruby does) there is
little you can do to alter this behaviour. otherwise the 'normal'
approach will work:

cfp:~ > cat a.rb
r, w = IO.pipe

child = fork{ STDOUT.reopen w; w.close; system 'echo 4 && sleep 1 &&
echo 2' }

w.close

while(( char = r.read(1) ))
printf "%f : %s\n", Time.now.to_f, char.inspect
end


cfp:~ > ruby a.rb
1210456103.163009 : "4"
1210456103.163153 : "\n"
1210456104.166967 : "2"
1210456104.167023 : "\n"


i might be wrong, but i really don't think there is anything you can
do with a child program that alters the buffering mode itself.

a @ http://codeforpeople.com/

 

[Paul Winward]

Thanks for the pointers. I wasn't aware of setvbuf. I'll look more into
pty.

 

[Tallak Tveide]

Hi folks.

After trying a lot of things, this worked for me on Linux:

%x{stty -icanon -echo}
key = STDIN.read(1)

I found this at
http://stackoverflow.com/questions/582200/setting-non-canonical-mode-on-stdin-with-ruby

Instead of the %x{...} you could use Termios instead to reset the
'ICANON' flag:

http://devver.net/blog/2009/07/a-command-line-prompt-with-timeout-and-countdown/

 

[Robert Klemme]

popen3 is going to be using fork/exec under the hood and should inherit t= he
sync setting of STDOUT, so try setting

=A0STDOUT.sync =3D true

before the popen and see if that helps - untested, but i'm running out th= e
door....

IMHO that can only work if you use fork with a block. Since Paul
wants to start a new process that cannot possibly inherit buffering
settings that way. Since Ruby decides buffering mode of stdout based
on the type (i.e. if it is a terminal then sync =3D true otherwise not)
and the child process cannot be influenced by Paul it follows that the
only feasible way to use would be to present a tty as stdout to the
child (via some form of pseudo tty mechanism). But that may have
adverse effects because some programs change their output based on the
type of stdout as well (just think about including colorization on
ttys and omitting it on pipes).

This is the place to ask: Paul, what do you need that behavior for?
Maybe there are other mechanism to use.

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/