Wrapping applications in Ruby

[Jp Hastings-spital]

Variants on this theme have been posted time and again, but I can't find
a solution for this specific problem:

Say I have a program I need to use that performs a task for a given
amount of time then finishes, all the while providing a 'percent
complete' style output on STDOUT. eg:

/counting.rb
[RUBY]
#!/usr/bin/env ruby
n = 0

while n <= 10
$stdout.puts "Completed: #{n*10}%"
n = n + 1
end
[/RUBY]

Now, I'd like to build a ruby script that will call this script, wait
until completion and then continue with its tasks BUT all the while
updating a variable with the percentage complete. eg:

/script.rb
[RUBY]
def updatePercentage(pc)
p "The percentage is now #{pc}"
end

Thread.new {
# -- Problem area

# Attempt 1
`./counting.rb`
# This will wait until after counting.rb is complete before
# "All Done!" is printed, but I can't get to its stdout stream
# before the program has finished running

# Attempt 2
Open3.popen3('./counting.rb') { |stdin, stdout, stderr|
p stdout.readpartial(1)

# With a view to doing something like:
updatePercentage(stdout.readline.gsub(/^.*([0-9]+)%.*$/,"$1"))
}
# This doesn't work, counting.rb never begins executing
# (tested by adding a line: open('output.txt','w') {|file| file.puts
"Percentage: #{n*10}%" }
# to counting.rb and watching the file externally)

p "All done!"
}

p "Irrelevant things"
sleep
[/RUBY]

Any ideas anyone? Essentially I'm looking for ways to wrap C programs
into ruby.

Cheers, JP

 

[Dylan Evans]

[Note: parts of this message were removed to make it a legal post.]

I won't go into details but you should look into pipes in this case. By
creating a process which does the work your wrapper code can just wait on
the pipe.

Variants on this theme have been posted time and again, but I can't find
a solution for this specific problem:

Say I have a program I need to use that performs a task for a given
amount of time then finishes, all the while providing a 'percent
complete' style output on STDOUT. eg:

./counting.rb
[RUBY]
#!/usr/bin/env ruby
n = 0

while n <= 10
$stdout.puts "Completed: #{n*10}%"
n = n + 1
end
[/RUBY]

Now, I'd like to build a ruby script that will call this script, wait
until completion and then continue with its tasks BUT all the while
updating a variable with the percentage complete. eg:

./script.rb
[RUBY]
def updatePercentage(pc)
p "The percentage is now #{pc}"
end

Thread.new {
# -- Problem area

# Attempt 1
`./counting.rb`
# This will wait until after counting.rb is complete before
# "All Done!" is printed, but I can't get to its stdout stream
# before the program has finished running

# Attempt 2
Open3.popen3('./counting.rb') { |stdin, stdout, stderr|
p stdout.readpartial(1)

# With a view to doing something like:
updatePercentage(stdout.readline.gsub(/^.*([0-9]+)%.*$/,"$1"))
}
# This doesn't work, counting.rb never begins executing
# (tested by adding a line: open('output.txt','w') {|file| file.puts
"Percentage: #{n*10}%" }
# to counting.rb and watching the file externally)

p "All done!"
}

p "Irrelevant things"
sleep
[/RUBY]

Any ideas anyone? Essentially I'm looking for ways to wrap C programs
into ruby.

Cheers, JP

 

[Jp Hastings-spital]

I won't go into details but you should look into pipes in this case.

Thanks for the prompt response - I'm assuming you mean Ruby pipes not
BASH pipes or equivalent. (http://www.ruby-doc.org/core/classes/IO.html)

I'll post here if I find a solution.

 

[Robert Klemme]

Thanks for the prompt response - I'm assuming you mean Ruby pipes not
BASH pipes or equivalent. (http://www.ruby-doc.org/core/classes/IO.html)

I'll post here if I find a solution.

I have no idea what you mean by "bash pipes" but the underlying
mechanism is the same. It's a feature of the operating system. One
solution is to use IO.popen. Your code does not read all the lines,
that might be one reason for your issue.

robert@fussel /c/Temp
$ ./run-counter.rb
Got 0
Got 10
Got 20
Got 30
Got 40
Got 50
Got 60
Got 70
Got 80
Got 90
Got 100

robert@fussel /c/Temp
$ cat run-counter.rb
#!/usr/bin/env ruby19

IO.popen "./counting.rb" do |io|
io.each do |line|
pct = line[%r{:\s*(\d+)%}, 1].to_i
puts "Got #{pct}"
end
end

robert@fussel /c/Temp
$ cat counting.rb
#!/usr/bin/env ruby19

$stdout.sync = true

11.times do |n|
puts "Completed: #{n*10}%"
end

robert@fussel /c/Temp
$

Similar with Open3, only that you then get multiple pipes.

Cheers

robert

 

[Jp Hastings-spital]

Robert,
Thanks very much for your reply - its helped no end! However I'm still
having problems:

Firstly (though really this isn't vitally important, as I only *need*
stdout):
Open3.popen3 doesn't appear to work when I drop it in place of IO.popen
(where |stdout| is replaced with |stdin,stdout,stderr|).
The called program is definitely running, however even the following
code outputs nothing:

Open3.popen3 "./counting.rb" do |stdin,stdout,stderr|
p stdout.read
end

But as I say, I only really need the stdout pipe. The problem here is
that the program I'm calling doesn't appear to send the \n character at
all (its the output from HandBrake's CLI), and as such the io block only
gets called once, at the end of the program's execution. Here's some
code:

$ HandBrakeCLI {options} 2> /dev/null
\rEncoding: task 1 of 1, 0.09 %\rEncoding: task 1 of 1, 0.13
%\rEncoding: task 1 of 1, 0.18 %\rEncoding: task 1 of 1, 0.23 %

$ cat wrapper.rb
input = "input.avi"
output = "output.mp4"
IO.popen("/usr/local/bin/HandBrakeCLI -i \"#{input}\" -o \"#{output}\"")
{ |stdout|
p stdout.read
}

This prints nothing! Am I right in thinking that the popen block gets
called every time a \n character is captured by the pipe? (Your
'counting.rb' uses \n rather than \r - and works) Any ideas (other than
say, passing the output through sed or something similar)?

Thanks again, JP

 

[Robert Klemme]

Robert,
Thanks very much for your reply - its helped no end! However I'm still
having problems:

Firstly (though really this isn't vitally important, as I only *need*
stdout):
Open3.popen3 doesn't appear to work when I drop it in place of IO.popen
(where |stdout| is replaced with |stdin,stdout,stderr|).
The called program is definitely running, however even the following
code outputs nothing:

Open3.popen3 "./counting.rb" do |stdin,stdout,stderr|
p stdout.read
end

No idea what you're doing, but it works for me

robert@fussel /c/Temp
$ ./run-counter.rb
Got 0
Got 10
Got 20
Got 30
Got 40
Got 50
Got 60
Got 70
Got 80
Got 90
Got 100
Got3 0
Got3 10
Got3 20
Got3 30
Got3 40
Got3 50
Got3 60
Got3 70
Got3 80
Got3 90
Got3 100

robert@fussel /c/Temp
$ cat run-counter.rb
#!/usr/bin/env ruby19

require 'open3'

IO.popen "./counting.rb" do |io|
io.each do |line|
pct = line[%r{:\s*(\d+)%}, 1].to_i
puts "Got #{pct}"
end
end

Open3.popen3 "./counting.rb" do |sin,sout,serr|
sin.close
th = Thread.new { serr.read }

sout.each do |line|
pct = line[%r{:\s*(\d+)%}, 1].to_i
puts "Got3 #{pct}"
end

th.join
end

robert@fussel /c/Temp
$

But as I say, I only really need the stdout pipe. The problem here is
that the program I'm calling doesn't appear to send the \n character at
all (its the output from HandBrake's CLI), and as such the io block only
gets called once, at the end of the program's execution. Here's some
code:

$ HandBrakeCLI {options} 2> /dev/null
\rEncoding: task 1 of 1, 0.09 %\rEncoding: task 1 of 1, 0.13
%\rEncoding: task 1 of 1, 0.18 %\rEncoding: task 1 of 1, 0.23 %

$ cat wrapper.rb
input = "input.avi"
output = "output.mp4"
IO.popen("/usr/local/bin/HandBrakeCLI -i \"#{input}\" -o \"#{output}\"")
{ |stdout|
p stdout.read
}

This prints nothing! Am I right in thinking that the popen block gets
called every time a \n character is captured by the pipe?

No. The block is called once per execution of the other process.

(Your
'counting.rb' uses \n rather than \r - and works) Any ideas (other than
say, passing the output through sed or something similar)?

Could be that your problem is IO buffering on the sender side (i.e. the
program you invoke). Alternatively, if you do not close stdin of the
subprogram that might be waiting for some input.

Cheers

robert

 

[Robert Klemme]

Few more remarks:

Robert,
Thanks very much for your reply - its helped no end! However I'm still
having problems:

Firstly (though really this isn't vitally important, as I only *need*
stdout):
Open3.popen3 doesn't appear to work when I drop it in place of IO.popen
(where |stdout| is replaced with |stdin,stdout,stderr|).
The called program is definitely running, however even the following
code outputs nothing:

Open3.popen3 "./counting.rb" do |stdin,stdout,stderr|
p stdout.read

This will read the whole stream to the end! Which means, it will return
only after the process died.

end

But as I say, I only really need the stdout pipe. The problem here is
that the program I'm calling doesn't appear to send the \n character at
all (its the output from HandBrake's CLI), and as such the io block only
gets called once, at the end of the program's execution. Here's some
code:

$ HandBrakeCLI {options} 2> /dev/null
\rEncoding: task 1 of 1, 0.09 %\rEncoding: task 1 of 1, 0.13
%\rEncoding: task 1 of 1, 0.18 %\rEncoding: task 1 of 1, 0.23 %

If you have "\r" as line terminator, you can do the reading like this

io.each "\r" do |line|
# do whatever with line
end

e.g.

robert@fussel ~
$ ruby -e '3.times {|i| printf "%03d\r", i}' | ruby -e
'$stdin.each("\r") {|l| p l}'
"000\r"
"001\r"
"002\r"

robert@fussel ~
$

$ cat wrapper.rb
input = "input.avi"
output = "output.mp4"
IO.popen("/usr/local/bin/HandBrakeCLI -i \"#{input}\" -o \"#{output}\"")
{ |stdout|
p stdout.read

See above.

}

This prints nothing! Am I right in thinking that the popen block gets
called every time a \n character is captured by the pipe? (Your
'counting.rb' uses \n rather than \r - and works) Any ideas (other than
say, passing the output through sed or something similar)?

Thanks again, JP

Note that the proper line reading (as suggested above) still does not
help if the other program internally buffers IO.

Kind regards

robert

 

[Jp Hastings-spital]

IO.popen method with io.each "\r" works spot on! Thanks so much!

Haven't tried working with Open3.popen3 yet, I may try reinstalling the
gem.

Thanks for your replies, you've been patient and very helpful!

If you have "\r" as line terminator, you can do the reading like this

io.each "\r" do |line|
# do whatever with line
end

This makes it work beautifully!

 

[Robert Klemme]

IO.popen method with io.each "\r" works spot on! Thanks so much!

Haven't tried working with Open3.popen3 yet, I may try reinstalling the
gem.

Thanks for your replies, you've been patient and very helpful!

You're welcome. Good to hear that your issue is fixed now!

Kind regards

robert