Asynchronous process to process pipe IO

[Pito Salas]

This is slightly subtle question about how a process that reads from a
pipe that is being written to by another process may run asynchronously.

Here's a code snippet:

def start_processing
open("|-", "r") do |worker|
if worker
# here we are in the parent
i = 0
worker.each_line do |line|
puts "line #{i}
i = i+1
end
else
# here we are in child thread
exec("./iacommand.rb")
end
end
end

iacommand.rb is invoked as a process within ruby. It does a lot of
processing, outputs a line, does a more processing, outputs another
line, and so on.

I would like to have the "puts" calls occur right when the a line is
generated by the script running in the process.

The way it is written above, what happens is that the app waits until
iacommand.rb exits and then calls the series of puts in immediate
succession.

So in other words, how do I asynchronously read from a pipe that is
being fed by a process?

 

[Joel VanderWerf]

This is slightly subtle question about how a process that reads from a
pipe that is being written to by another process may run asynchronously.

Here's a code snippet:

def start_processing
open("|-", "r") do |worker|
if worker
# here we are in the parent
i = 0
worker.each_line do |line|
puts "line #{i}
i = i+1
end
else
# here we are in child thread
exec("./iacommand.rb")
end
end
end

iacommand.rb is invoked as a process within ruby. It does a lot of
processing, outputs a line, does a more processing, outputs another
line, and so on.

I would like to have the "puts" calls occur right when the a line is
generated by the script running in the process.

The way it is written above, what happens is that the app waits until
iacommand.rb exits and then calls the series of puts in immediate
succession.

So in other words, how do I asynchronously read from a pipe that is
being fed by a process?

Hm, the following shows output at 1 sec intervals:

open("|-", "r") do |worker|
if worker
i = 0
worker.each_line do |line|
puts "line #{i}:#{line}"
i = i+1
end
else
exec("ruby -e '3.times {|i| p i; sleep 1}'")
end
end

Does it work that way for you?

 

[Pito Salas]

Hm, the following shows output at 1 sec intervals:

open("|-", "r") do |worker|
if worker
i = 0
worker.each_line do |line|
puts "line #{i}:#{line}"
i = i+1
end
else
exec("ruby -e '3.times {|i| p i; sleep 1}'")
end
end

Does it work that way for you?

Yes it does. Double-hm.

My case is using a second ruby file, iacommand.rb, which is doing much
the same as what your -e is doing. I wonder if that makes the
difference? Or it must be something else.

 

[Pito Salas]

Yes it does. Double-hm.

My case is using a second ruby file, iacommand.rb, which is doing much
the same as what your -e is doing. I wonder if that makes the
difference? Or it must be something else.

It does seem to be the difference between ruby -e and ./iacommand.rb.
Can anyone see what the key is??

Thanks!!

Here's what fails (that is, blocks) (tested both in irb and in eclipse)

open("|-", "r") do |worker|
if worker
i = 0
worker.each_line do |line|
puts "line #{i}:#{line}"
i = i+1
end
else
exec("./iacommand.rb", "-t")
end
end


And here's the exact text of iacommand.rb:


require 'rubygems'
require 'getoptlong'

parser = GetoptLong.new
parser.set_options(
["-h", "--help", GetoptLong::NO_ARGUMENT],
["-t", "--test", GetoptLong::NO_ARGUMENT],
["-v", "--version", GetoptLong::NO_ARGUMENT])

valid = false
loop do

begin
opt, arg = parser.get
break if not opt
case opt
when "-h"
puts "Usage: ..."
valid = true
break
when "-t"
valid = true
break
when "-v"
puts "Version 0.0"
valid = true
break
end
end
end

if valid
puts "start"
i = 0
5.times do
puts "ballot #{i}"
delay = rand(5).to_i
sleep delay
i = i+1
end
puts "exit"
else
puts "invalid parameters for iacommand"
end

 

[Robert Klemme]

Hm, the following shows output at 1 sec intervals:

open("|-", "r") do |worker|
if worker
i = 0
worker.each_line do |line|
puts "line #{i}:#{line}"
i = i+1
end
else
exec("ruby -e '3.times {|i| p i; sleep 1}'")
end
end

Does it work that way for you?

Why are you guys using such a complex construction? If you just want to
see the output when it comes why not just:

system "./iacommand.rb"

Or maybe this, if you need to process the output:

IO.popen "./iacommand.rb" do |io|
io.each_line do |line|
puts line
end
end

Note, you may have to do $stdout.sync = true in "iacommand.rb".

Kind regards

robert

 

[Pito Salas]

Why are you guys using such a complex construction? If you just want to
see the output when it comes why not just:

system "./iacommand.rb"

Or maybe this, if you need to process the output:

IO.popen "./iacommand.rb" do |io|
io.each_line do |line|
puts line
end
end

Note, you may have to do $stdout.sync = true in "iacommand.rb".

Kind regards

robert

Robert,

Thanks, the $stdout.sync = true was the magic bullet.

Now, I am not exactly clear (reading the doc) about the difference
between exec and system... I do need to be able to send command line
arguments to iacommand.rb and I do indeed need to process the output 'as
it appears.'

Thanks!!

Pito

 

[Gary Wright]

Now, I am not exactly clear (reading the doc) about the difference
between exec and system... I do need to be able to send command line
arguments to iacommand.rb and I do indeed need to process the output
'as
it appears.'

exec() causes a new program to be executed within the current process.
Unless exec() fails (e.g. program not found), exec() will never 'return'
because the current program will be discarded (i.e. the Ruby
interpreter)
in favor of the newly exec'ed one.

system() causes a new (child) process to be created and for the new
program to be executed (i.e. exec'ed) within the new child process.
The current process will wait for the child process to terminate before
proceeding.

The difference is whether the new program replaces the currently
running program or not. Note that 'program' here means the Ruby
interpreter and not the particular Ruby script that is running.

Note: I'm answering from the perspective of a Unix/Posix environment.
My windows-fu isn't sufficient to say if my description is correct
for that environment, but it is probably pretty close.

Gary Wright

 

[Mike Barsalou]

exec() causes a new program to be executed within the current process.
Unless exec() fails (e.g. program not found), exec() will never 'return'
because the current program will be discarded (i.e. the Ruby  
interpreter)
in favor of the newly exec'ed one.

system() causes a new (child) process to be created and for the new
program to be executed (i.e. exec'ed) within the new child process.
The current process will wait for the child process to terminate before
proceeding.

The difference is whether the new program replaces the currently
running program or not. Note that 'program' here means the Ruby
interpreter and not the particular Ruby script that is running.

Note: I'm answering from the perspective of a Unix/Posix environment.
My windows-fu isn't sufficient to say if my description is correct
for that environment, but it is probably pretty close.

Gary Wright

Would Ruby's fork behave exactly like system here, or would it have a
third path? Likely fork wouldn't be an option for the OP because he
wants to process the output?

Mike B.

 

[Gary Wright]

Would Ruby's fork behave exactly like system here, or would it have a
third path? Likely fork wouldn't be an option for the OP because he
wants to process the output?

fork and exec are the 'primitive' operations.

system is basically implemented by forking and having the child
process exec the new program while the parent waits for it to
complete. System() is just a convenience method for this
common pattern.

Capturing the output of a program basically means using fork/exec
along with rearranging the standout output of the exec'ed
program. For simple cases this is handled by Ruby's backtick syntax:

output = `program_to_run`

Gary Wright

 

[Robert Klemme]

Robert,

Thanks, the $stdout.sync = true was the magic bullet.

Now, I am not exactly clear (reading the doc) about the difference
between exec and system... I do need to be able to send command line
arguments to iacommand.rb and I do indeed need to process the output 'as
it appears.'

Then you'll want to use one of the popen family of functions. Even with
IO.popen you can pass arguments:

IO.popen ["./iacommand.rb", "arg", "arg"] do |io|
...
end

Note, if you also want to write to the process's stdin, you need to
provide a file mode as second argument. See docs.

Kind regards

robert

 

[Pito Salas]

Then you'll want to use one of the popen family of functions. Even with
IO.popen you can pass arguments:

IO.popen ["./iacommand.rb", "arg", "arg"] do |io|
...
end

Note, if you also want to write to the process's stdin, you need to
provide a file mode as second argument. See docs.

This is what I originally had. Pretty similar...

open("|-", "r") do |worker|
if worker
...
else
exec("./iacommand.rb", "arg", "arg")
end
end

How do you think that the idea you show above is different/better? This
gets pretty subtle... Thanks again for the great thread about Threads

 

[Robert Klemme]

Then you'll want to use one of the popen family of functions. =A0Even wi= th
IO.popen you can pass arguments:

IO.popen ["./iacommand.rb", "arg", "arg"] do |io|
=A0 =A0...
end

Note, if you also want to write to the process's stdin, you need to
provide a file mode as second argument. =A0See docs.

This is what I originally had. Pretty similar...

open("|-", "r") do |worker|
=A0 =A0if worker
...
=A0 =A0else
=A0 =A0 =A0exec("./iacommand.rb", "arg", "arg")
=A0 =A0end
=A0end

How do you think that the idea you show above is different/better?

I would prefer IO.popen as it is shorter - and it does not need
cryptic (aka perlish) file names:

IO.popen ["./iacommand.rb", "arg", "arg"], "rw" do |io|
# no if here either
io.puts "start"
io.close_write

io.each_line do |line|
puts "GOT: #{line}"
end
end

This
gets pretty subtle... Thanks again for the great thread about Threads

Sorry, I do not find it that subtle. IMHO it is pretty obvious that
IO.popen is superfior for the job you want to get done. That is, of
course, unless I am missing something...

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/