reading from an external process with IO.popen

Discussion in 'Ruby' started by Robert Citek, Nov 5, 2009.

  1. Robert Citek

    Robert Citek Guest

    Hello all,

    I'm trying to wrap my head around IO.popen with some simple examples
    that send data to and read data from an
    external process. =C2=A0I've create a sample case in the shell like this:

    $ { echo hello ; sleep 2 ; echo world; } | cat
    hello
    world

    I've written the same in ruby like so, which works:

    $ cat foo.rb
    #!/usr/bin/env ruby
    if $0 =3D=3D __FILE__
    =C2=A0cat =3D IO.popen("cat", "w+") ;
    =C2=A0cat.puts("hello, ") ;
    =C2=A0puts(cat.gets) ;
    =C2=A0sleep 2 ;
    =C2=A0cat.puts("world") ;
    =C2=A0puts(cat.gets) ;
    end

    $ ./foo.rb
    hello
    world

    However, if I change the cat command to a sed command, the ruby
    version no longer works. =C2=A0The command-line equivalent does work, but
    the ruby version waits forever and has to be interrupted:

    $ { echo hello ; sleep 2 ; echo world; } | sed -ne p
    hello
    world

    $ cat foo.rb
    #!/usr/bin/env ruby
    if $0 =3D=3D __FILE__
    =C2=A0cat =3D IO.popen("sed -ne p", "w+") ;
    =C2=A0cat.puts("hello, ") ;
    =C2=A0puts(cat.gets) ;
    =C2=A0sleep 2 ;
    =C2=A0cat.puts("world") ;
    =C2=A0puts(cat.gets) ;
    end

    $ ./foo.rb
    /foo.rb:6:in `gets': Interrupt
    =C2=A0 =C2=A0 =C2=A0 =C2=A0from ./foo.rb:6

    Why does ruby work in the first case but wait forever in the second?

    Using this version of ruby:

    $ ruby -v
    ruby 1.8.6 (2007-09-24 patchlevel 111) [i486-linux]

    Thanks in advance for any pointers to references.

    Regards,
    - Robert
    Robert Citek, Nov 5, 2009
    #1
    1. Advertising

  2. 2009/11/5 Robert Citek <>:
    > Hello all,
    >
    > I'm trying to wrap my head around IO.popen with some simple examples
    > that send data to and read data from an
    > external process. =A0I've create a sample case in the shell like this:
    >
    > $ { echo hello ; sleep 2 ; echo world; } | cat
    > hello
    > world
    >
    > I've written the same in ruby like so, which works:
    >
    > $ cat foo.rb
    > #!/usr/bin/env ruby
    > if $0 =3D=3D __FILE__
    > =A0cat =3D IO.popen("cat", "w+") ;
    > =A0cat.puts("hello, ") ;
    > =A0puts(cat.gets) ;
    > =A0sleep 2 ;
    > =A0cat.puts("world") ;
    > =A0puts(cat.gets) ;
    > end
    >
    > $ ./foo.rb
    > hello
    > world
    >
    > However, if I change the cat command to a sed command, the ruby
    > version no longer works. =A0The command-line equivalent does work, but
    > the ruby version waits forever and has to be interrupted:


    That's probably because you do not close the write end of the pipe in
    Ruby code. Also, it's better to place the reading portion in a
    separate thread in order to prevent deadlocks. And, please use the
    block form of IO.popen which is more robust.

    Try this pattern:

    IO.popen("cat", "w+") do |cat|
    # background output
    t =3D Thread.new { cat.each {|l| puts l} }

    # main work
    cat.puts "hello, "
    sleep 2
    cat.puts "world"

    # terminate processing:
    cat.close_write
    t.join
    end

    Kind regards

    robert

    --=20
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
    Robert Klemme, Nov 6, 2009
    #2
    1. Advertising

  3. Robert Citek

    Robert Citek Guest

    On Fri, Nov 6, 2009 at 5:15 AM, Robert Klemme
    <> wrote:
    > 2009/11/5 Robert Citek <>:
    >> However, if I change the cat command to a sed command, the ruby
    >> version no longer works. =C2=A0The command-line equivalent does work, bu=

    t
    >> the ruby version waits forever and has to be interrupted:

    >
    > That's probably because you do not close the write end of the pipe in
    > Ruby code.


    Perhaps, but what if I don't want to close the pipe? That is, I would
    like to keep the pipe open so that I can send some data, read some
    data and work on it, send some more data, read some more data and work
    on it, etc. much like the process was a service, e.g. database. I am
    trying to code the equivalent of a Call and Response. My examples
    using cat and sed are just stand-ins for the real program.

    BTW, the cat example works as expected, but the using sed doesn't
    work. That is, there is no output from sed until the pipe closes.
    There seems to be some buffering going on. I'm guessing it's from the
    Ruby side since I don't see this when run from the shell. But that's
    just a guess.

    Of course, it's entirely possible that IO.popen is not the "right" way
    to tackle this and I have not discovered the Ruby way, yet.

    Again, any pointers in the right direction are greatly appreciated.

    Regards,
    - Robert
    Robert Citek, Nov 6, 2009
    #3
  4. On 11/06/2009 04:01 PM, Robert Citek wrote:
    > On Fri, Nov 6, 2009 at 5:15 AM, Robert Klemme
    > <> wrote:
    >> 2009/11/5 Robert Citek <>:
    >>> However, if I change the cat command to a sed command, the ruby
    >>> version no longer works. The command-line equivalent does work, but
    >>> the ruby version waits forever and has to be interrupted:

    >> That's probably because you do not close the write end of the pipe in
    >> Ruby code.

    >
    > Perhaps, but what if I don't want to close the pipe? That is, I would
    > like to keep the pipe open so that I can send some data, read some
    > data and work on it, send some more data, read some more data and work
    > on it, etc. much like the process was a service, e.g. database. I am
    > trying to code the equivalent of a Call and Response. My examples
    > using cat and sed are just stand-ins for the real program.


    If the program you are using does not cooperate you're out of luck. For
    example, if it assigns a huge read buffer then you might have to send
    hundreds of lines before it even starts processing the first one. I
    have no idea how the implementation of sed that you are using does it
    but if you for example think of sort you _cannot_ get any output before
    the last line has been written and the write end of the pipe has been
    closed.

    > BTW, the cat example works as expected, but the using sed doesn't
    > work. That is, there is no output from sed until the pipe closes.
    > There seems to be some buffering going on. I'm guessing it's from the
    > Ruby side since I don't see this when run from the shell. But that's
    > just a guess.


    The shell closes the pipe as well. It is sed that is doing the
    buffering and you have no control over it unless it provides an option
    to control this.

    > Of course, it's entirely possible that IO.popen is not the "right" way
    > to tackle this and I have not discovered the Ruby way, yet.


    No, it's the right way but your expectations cannot be met in all cases.

    Kind regards

    robert


    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
    Robert Klemme, Nov 6, 2009
    #4
  5. Robert Citek

    Robert Citek Guest

    On Fri, Nov 6, 2009 at 12:50 PM, Robert Klemme
    <> wrote:
    > On 11/06/2009 04:01 PM, Robert Citek wrote:
    >> BTW, the cat example works as expected, but the using sed doesn't
    >> work. =C2=A0That is, there is no output from sed until the pipe closes.
    >> There seems to be some buffering going on. =C2=A0I'm guessing it's from =

    the
    >> Ruby side since I don't see this when run from the shell. =C2=A0But that=

    's
    >> just a guess.

    >
    > The shell closes the pipe as well. =C2=A0It is sed that is doing the buff=

    ering
    > and you have no control over it unless it provides an option to control
    > this.


    Yes, it appears that the external program is controlling the
    buffering. When I tried the same process with the program I really
    wanted to use, IO.popen worked pretty much the way I wanted it to.
    The pattern was this:

    foo =3D io.popen("external_program", "w+")
    while data =3D gets
    prepare data
    foo.puts(data)
    while not end of record
    newdata +=3D foo.readlines
    end
    process newdata
    end
    foo.close

    Turns out that the program I used has a signal to signify the end of a
    chunk of data. So the program knows when I am finished sending data
    and it can start crunching away. And I know when I can stop reading
    data from the pipe and begin processing it. This saves the time of
    repeatedly having to open and close the pipe.

    >> Of course, it's entirely possible that IO.popen is not the "right" way
    >> to tackle this and I have not discovered the Ruby way, yet.

    >
    > No, it's the right way but your expectations cannot be met in all cases.


    It's nice to know that I'm at least on the right track, or on one of
    many possible right tracks.

    Thanks for your help.

    Regards,
    - Robert
    Robert Citek, Nov 6, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Brian Elmegaard
    Replies:
    2
    Views:
    974
    Robert Amesz
    Sep 10, 2003
  2. ckkart
    Replies:
    4
    Views:
    937
    ckkart
    Nov 27, 2008
  3. P.S.
    Replies:
    0
    Views:
    307
  4. Miki Tebeka
    Replies:
    2
    Views:
    581
    Chris Angelico
    Apr 8, 2011
  5. File.popen/IO.popen

    , May 20, 2006, in forum: Ruby
    Replies:
    1
    Views:
    213
    Robert Klemme
    May 20, 2006
Loading...

Share This Page