Get back data from a child (with exec)

[Lawrence Oluyede]

What I'm trying to do is to pass some data
to a child process that execute an external process
and get back data from this external process.

---
PATH = "/usr/bin/highlight"
ARGS = "-f -l -t 8 -S %s"
LANG = "rb"

rd, wr = IO.pipe

if fork
# parent
rd.close
$stdout.reopen(wr)
wr.close

fd = File.open("/usr/lib/ruby/1.8/thread.rb")
str = fd.read
$stdout.write str
else
# child
wr.close
$stdin.reopen(rd)
rd.close

params = ARGS % LANG
cmd = "%s %s" % [PATH, params]
exec cmd
end

 

[Ara.T.Howard]

What I'm trying to do is to pass some data
to a child process that execute an external process
and get back data from this external process.

---
PATH = "/usr/bin/highlight"
ARGS = "-f -l -t 8 -S %s"
LANG = "rb"

rd, wr = IO.pipe

if fork
# parent
rd.close
$stdout.reopen(wr)
wr.close

fd = File.open("/usr/lib/ruby/1.8/thread.rb")
str = fd.read
$stdout.write str
else
# child
wr.close
$stdin.reopen(rd)
rd.close

params = ARGS % LANG
cmd = "%s %s" % [PATH, params]
exec cmd
end

harp:~ > cat a.rb
IO:open('-') do |pipe|
if pipe
stdout = pipe.read
puts "parent got <#{ stdout }> from child"
else
exec 'echo', '-n', '42'
end
end

harp:~ > ruby a.rb
parent got <42> from child


you can also

STDERR.reopen '/dev/null'

in the child the shutup stderr - otherwise look at the code for open3 (it's
short) to see how to get a handle on both stdout and stderr of the child.

hth.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| renunciation is not getting rid of the things of this world, but accepting
| that they pass away. --aitken roshi
===============================================================================

 

[Lawrence Oluyede]

harp:~ > cat a.rb
IO:open('-') do |pipe|
if pipe
stdout = pipe.read
puts "parent got <#{ stdout }> from child"
else
exec 'echo', '-n', '42'
end
end

harp:~ > ruby a.rb
parent got <42> from child

Works, but there's a problem. If I open the pipe in read/write
mode and exec a program that reads from stdin it doesn't work

 

[Ara.T.Howard]

Works, but there's a problem. If I open the pipe in read/write mode and exec
a program that reads from stdin it doesn't work

harp:~ > cat a.rb
IO:open('-', 'r+') do |pipe|
if pipe
pipe.puts '42'
pipe.close_write
stdout = pipe.read.strip
#pipe.close
puts "parent got <#{ stdout }> from child"
else
exec 'cat'
end
end

harp:~ > ruby a.rb
parent got <42> from child


you aren't trying to run a program that requires a tty (like top) are you?

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| renunciation is not getting rid of the things of this world, but accepting
| that they pass away. --aitken roshi
===============================================================================

 

[Lawrence Oluyede]

pipe.close_write

That's what I missed!

you aren't trying to run a program that requires a tty (like top) are you?

No no...

Thanks a lot!

 

[Lawrence Oluyede]

How can I deal with large inputs to an external process
popen driven? With 5k of data sent to its standard
input blocks.

 

[Ara.T.Howard]

How can I deal with large inputs to an external process
popen driven? With 5k of data sent to its standard
input blocks.

send the data down in a thread

data = IO::read 'huge_file'
sender = Thread::new(data){data.each{|line| pipe.puts line}}

you'd probably then want a reader to load a thread safe queue with the
returned data.

why don't you want to block?

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| renunciation is not getting rid of the things of this world, but accepting
| that they pass away. --aitken roshi
===============================================================================

 

[Lawrence Oluyede]

send the data down in a thread

data = IO::read 'huge_file'
sender = Thread::new(data){data.each{|line| pipe.puts line}}

you'd probably then want a reader to load a thread safe queue with the
returned data.

That's what I was doing before with popen3, wrap the function with a Thread.
Let me try again

why don't you want to block?

Cause it hangs the UI (a webpage)

 

[Ilmari Heikkinen]

How can I deal with large inputs to an external process
popen driven? With 5k of data sent to its standard
input blocks.

You need to send the data in blocks lequal to the pipe buffer size.
Something like pipe.write(input.read(4096)) until input.eof?

Also, if the external process outputs to its stdout, it will block once
it
has written pipe buffer bytes, so you need to read from the pipe aswell.

So, one thread for writing, one thread for reading.


Btw, maybe popened IOs should do this automatically under covers?

test_popen.rb

d = ' '*32000
IO.popen('cat','r+'){|c|
Thread.new{ c.write(d); c.close_write }
puts c.read.size
}

Here's a way to fix the hang:

class IO
class << self
alias_method :real_popen, open
def popen(*args,&block)
io = real_popen(*args)
def io.write(data)
i = wb = 0
(wb += super(data[i,4096]); i+=4096) while i < data.size
wb
end
if block_given?
block.call(io)
else
io
end
end
end
end

 

[Ara.T.Howard]

That's what I was doing before with popen3, wrap the function with a Thread.
Let me try again


Cause it hangs the UI (a webpage)

ah, why didn't you say so? ;-) that's __exactly__ why i designed my session
lib - to NOT hang tk apps. it's thread safe so you can do this:

class UI
def initialize
@session = Session::new
@stdout_widget = SomeWidget::new
@stderr_widget = SomeWidget::new
end

def button_pressed
command = get_system_command

Thread::new(session, command) do |s, c|
s.execute(c) do |stdout, stderr|
@stdout_widget.update stdout if stdout
@stderr_widget.update stderr if stderr
end
end
end
end

and this runs in the background. the block for stdout and stderr are handled
as output is produced. session runs commands in the shell and so you don't
have a handle on the stdin in the process, but this would be easy to solve via

require 'tempfile'

def run_background_command command, input
tmp = Tempfile::new rand.to_s
begin
tmp.write input
tmp.close

command = "#{ command } < #{ tmp.path }"

Thread::new do
@session.execute(command) do |stdout, stderr|
async_handle stdout if stdout
async_handle stderr if stderr
end
end
ensure
tmp.close! if tmp
end
end
...
...
dont_forget_to_join_this_thread = run_background_command 'ui_hanger.exe', 'abc'
...
...
dont_forget_to_join_this_thread.join

make sense?

http://www.codeforpeople.com/lib/ruby/session/
http://raa.ruby-lang.org/project/session/

hth.

-a
--
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| renunciation is not getting rid of the things of this world, but accepting
| that they pass away. --aitken roshi
===============================================================================

 

[Lawrence Oluyede]

ah, why didn't you say so? ;-) that's __exactly__ why i designed my session
lib - to NOT hang tk apps. it's thread safe so you can do this:

Seems cool but I don't get it work with my own
example. Popen/forks/threads/hangs are causing me a big headache in those
days Everything (popen and my own fork based code) seems to work except
when it's behind a web app.

I think I should write a Ruby replacement of the external process

 

[Lawrence Oluyede]

You need to send the data in blocks lequal to the pipe buffer size.
Something like pipe.write(input.read(4096)) until input.eof?

Tried send data line by line but it still hangs

Btw, maybe popened IOs should do this automatically under covers?

I think it does but my code works behind a Rails/cgi app and it doesn't work.
I'm planning to write a Ruby code replacement for the external process.

Thanks anyway

 

[Lawrence Oluyede]

I think I should write a Ruby replacement of the external process

I think I can wait

Ara you are great, the 2.4.0 version of your session lib resolves
all my problems

Thanks a lot