reading from an external process with IO.popen

[Robert Citek]

Hello all,

I'm trying to wrap my head around IO.popen with some simple examples
that send data to and read data from an
external process. =C2=A0I've create a sample case in the shell like this:

$ { echo hello ; sleep 2 ; echo world; } | cat
hello
world

I've written the same in ruby like so, which works:

$ cat foo.rb
#!/usr/bin/env ruby
if $0 =3D=3D __FILE__
=C2=A0cat =3D IO.popen("cat", "w+") ;
=C2=A0cat.puts("hello, ") ;
=C2=A0puts(cat.gets) ;
=C2=A0sleep 2 ;
=C2=A0cat.puts("world") ;
=C2=A0puts(cat.gets) ;
end

$ ./foo.rb
hello
world

However, if I change the cat command to a sed command, the ruby
version no longer works. =C2=A0The command-line equivalent does work, but
the ruby version waits forever and has to be interrupted:

$ { echo hello ; sleep 2 ; echo world; } | sed -ne p
hello
world

$ cat foo.rb
#!/usr/bin/env ruby
if $0 =3D=3D __FILE__
=C2=A0cat =3D IO.popen("sed -ne p", "w+") ;
=C2=A0cat.puts("hello, ") ;
=C2=A0puts(cat.gets) ;
=C2=A0sleep 2 ;
=C2=A0cat.puts("world") ;
=C2=A0puts(cat.gets) ;
end

$ ./foo.rb
/foo.rb:6:in `gets': Interrupt
=C2=A0 =C2=A0 =C2=A0 =C2=A0from ./foo.rb:6

Why does ruby work in the first case but wait forever in the second?

Using this version of ruby:

$ ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i486-linux]

Thanks in advance for any pointers to references.

Regards,
- Robert

 

[Robert Klemme]

Hello all,

I'm trying to wrap my head around IO.popen with some simple examples
that send data to and read data from an
external process. =A0I've create a sample case in the shell like this:

$ { echo hello ; sleep 2 ; echo world; } | cat
hello
world

I've written the same in ruby like so, which works:

$ cat foo.rb
#!/usr/bin/env ruby
if $0 =3D=3D __FILE__
=A0cat =3D IO.popen("cat", "w+") ;
=A0cat.puts("hello, ") ;
=A0puts(cat.gets) ;
=A0sleep 2 ;
=A0cat.puts("world") ;
=A0puts(cat.gets) ;
end

$ ./foo.rb
hello
world

However, if I change the cat command to a sed command, the ruby
version no longer works. =A0The command-line equivalent does work, but
the ruby version waits forever and has to be interrupted:

That's probably because you do not close the write end of the pipe in
Ruby code. Also, it's better to place the reading portion in a
separate thread in order to prevent deadlocks. And, please use the
block form of IO.popen which is more robust.

Try this pattern:

IO.popen("cat", "w+") do |cat|
# background output
t =3D Thread.new { cat.each {|l| puts l} }

# main work
cat.puts "hello, "
sleep 2
cat.puts "world"

# terminate processing:
cat.close_write
t.join
end

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Robert Citek]

That's probably because you do not close the write end of the pipe in
Ruby code.

Perhaps, but what if I don't want to close the pipe? That is, I would
like to keep the pipe open so that I can send some data, read some
data and work on it, send some more data, read some more data and work
on it, etc. much like the process was a service, e.g. database. I am
trying to code the equivalent of a Call and Response. My examples
using cat and sed are just stand-ins for the real program.

BTW, the cat example works as expected, but the using sed doesn't
work. That is, there is no output from sed until the pipe closes.
There seems to be some buffering going on. I'm guessing it's from the
Ruby side since I don't see this when run from the shell. But that's
just a guess.

Of course, it's entirely possible that IO.popen is not the "right" way
to tackle this and I have not discovered the Ruby way, yet.

Again, any pointers in the right direction are greatly appreciated.

Regards,
- Robert

 

[Robert Klemme]

Perhaps, but what if I don't want to close the pipe? That is, I would
like to keep the pipe open so that I can send some data, read some
data and work on it, send some more data, read some more data and work
on it, etc. much like the process was a service, e.g. database. I am
trying to code the equivalent of a Call and Response. My examples
using cat and sed are just stand-ins for the real program.

If the program you are using does not cooperate you're out of luck. For
example, if it assigns a huge read buffer then you might have to send
hundreds of lines before it even starts processing the first one. I
have no idea how the implementation of sed that you are using does it
but if you for example think of sort you _cannot_ get any output before
the last line has been written and the write end of the pipe has been
closed.

BTW, the cat example works as expected, but the using sed doesn't
work. That is, there is no output from sed until the pipe closes.
There seems to be some buffering going on. I'm guessing it's from the
Ruby side since I don't see this when run from the shell. But that's
just a guess.

The shell closes the pipe as well. It is sed that is doing the
buffering and you have no control over it unless it provides an option
to control this.

Of course, it's entirely possible that IO.popen is not the "right" way
to tackle this and I have not discovered the Ruby way, yet.

No, it's the right way but your expectations cannot be met in all cases.

Kind regards

robert

 

[Robert Citek]

The shell closes the pipe as well. =C2=A0It is sed that is doing the buff= ering
and you have no control over it unless it provides an option to control
this.

Yes, it appears that the external program is controlling the
buffering. When I tried the same process with the program I really
wanted to use, IO.popen worked pretty much the way I wanted it to.
The pattern was this:

foo =3D io.popen("external_program", "w+")
while data =3D gets
prepare data
foo.puts(data)
while not end of record
newdata +=3D foo.readlines
end
process newdata
end
foo.close

Turns out that the program I used has a signal to signify the end of a
chunk of data. So the program knows when I am finished sending data
and it can start crunching away. And I know when I can stop reading
data from the pipe and begin processing it. This saves the time of
repeatedly having to open and close the pipe.

No, it's the right way but your expectations cannot be met in all cases.

It's nice to know that I'm at least on the right track, or on one of
many possible right tracks.

Thanks for your help.

Regards,
- Robert