Thread reading from a pipe blocks other threads, why?

Discussion in 'Ruby' started by Anders Lindgren, Mar 6, 2007.

  1. Hi!

    I'm implementing an application where the main Ruby program spawns off
    a number of external programs using pipes. Each pipe is handled by its
    own thread. Well, so far so good... Unfortunately, I'm having problems
    since when one thread calls "pipe.each_line" the other (!) threads
    hang. Well, if I add a "Process.waitpid" call before the call to
    pipe.each_line then the external command will sooner or later hang
    since noone consumes anything in the pipe.

    To exemplify, below are two ruby programs. The first, lotsoflines.rb,
    will print so much output that the pipe will be filles (in the real
    world this is typically not written in Ruby). The second, test.rb,
    without the "waitpid" will block the thread printing "BG Thread"; with
    the "waitpid" line the background process will never terminate.

    So, what I would need is a way to read from the pipe *without blocking
    other threads*, or a wait until there is data to read, or similar. All
    suggestions are welcome!

    Oh, btw, I need this to work both under win32 and unix.

    -- Anders Lindgren

    ------------------------------------ lotsoflines.rb
    ARGV[0].to_i.times do puts "A LINE" end
    sleep 2
    ------------------------------------ test.rb
    cmd = "ruby lotsoflines.rb 300"

    Thread.new do
    while true
    sleep 0.1
    puts "BG Thread"
    end
    end

    puts "Before popen"

    IO.popen(cmd) do |pipe|

    puts "Inside popen"

    # With this line, the background thread will be blocked, with this
    # line the process at the other end of the pipe will hang...

    Process.waitpid(pipe.pid)

    pipe.each_line do |x|
    puts "XXX:" + x
    end
    end

    puts "After popen"
    Anders Lindgren, Mar 6, 2007
    #1
    1. Advertising

  2. Anders Lindgren

    Guest

    On Tue, 6 Mar 2007, Anders Lindgren wrote:

    > Hi!
    >
    > I'm implementing an application where the main Ruby program spawns off a
    > number of external programs using pipes. Each pipe is handled by its own
    > thread. Well, so far so good... Unfortunately, I'm having problems since
    > when one thread calls "pipe.each_line" the other (!) threads hang. Well, if
    > I add a "Process.waitpid" call before the call to pipe.each_line then the
    > external command will sooner or later hang since noone consumes anything in
    > the pipe.
    >
    > To exemplify, below are two ruby programs. The first, lotsoflines.rb, will
    > print so much output that the pipe will be filles (in the real world this is
    > typically not written in Ruby). The second, test.rb, without the "waitpid"
    > will block the thread printing "BG Thread"; with the "waitpid" line the
    > background process will never terminate.
    >
    > So, what I would need is a way to read from the pipe *without blocking other
    > threads*, or a wait until there is data to read, or similar. All suggestions
    > are welcome!
    >
    > Oh, btw, I need this to work both under win32 and unix.


    it can't be done using popen and threads with current ruby. search the
    archives.

    this may help

    http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README

    gem install systemu


    >
    > -- Anders Lindgren
    >
    > ------------------------------------ lotsoflines.rb
    > ARGV[0].to_i.times do puts "A LINE" end
    > sleep 2
    > ------------------------------------ test.rb
    > cmd = "ruby lotsoflines.rb 300"
    >
    > Thread.new do
    > while true
    > sleep 0.1
    > puts "BG Thread"
    > end
    > end
    >
    > puts "Before popen"
    >
    > IO.popen(cmd) do |pipe|
    >
    > puts "Inside popen"
    >
    > # With this line, the background thread will be blocked, with this
    > # line the process at the other end of the pipe will hang...
    >
    > Process.waitpid(pipe.pid)
    >
    > pipe.each_line do |x|
    > puts "XXX:" + x
    > end
    > end
    >
    > puts "After popen"
    >
    >


    -a
    --
    be kind whenever possible... it is always possible.
    - the dalai lama
    , Mar 6, 2007
    #2
    1. Advertising

  3. Thanks for the reply!

    > it can't be done using popen and threads with current ruby. search the
    > archives.


    Well, I did search the archive, but I didn't get a straight answer on
    this. I couldn't imagine that it would be impossible to read data from
    more than one source at the same time.

    You said "current ruby" -- well, maybe there is hope for the future!

    I just noticed that in the weekly newsletter it was announced that
    Pragmatic Programmers will publish a book on Erlang. This is a
    language I worked on many years ago and I haven't really missed it --
    until today, that is. The model it uses for building concurrent
    applications is really a winner. In that environment you simply
    specify the start function (the "main" of the thread), communication
    is typically performed by passing messages. The "fork" paradigm and
    shared variables used by Ruby is something that really should be
    declared as deprecated, since it really makes the program difficult to
    write (not to mention impossible to read).

    I cross my fingers that the next major version of Ruby uses a non-toy
    concurrent solution.


    > this may help
    >
    > http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README
    >
    > gem install systemu


    Thanks, I will take a look at it!

    -- Anders Lindgren
    Anders Lindgren, Mar 6, 2007
    #3
  4. Anders Lindgren

    MenTaLguY Guest

    On Wed, 7 Mar 2007 00:40:06 +0900, "Anders Lindgren" <> wrote:

    > I cross my fingers that the next major version of Ruby uses a non-toy
    > concurrent solution.


    I'm working on an implementation of actors for Ruby (as part of an omnibus concurrency library). While it doesn't eliminate shared state from the language, in principle it makes it possible to avoid in many cases.

    -mental
    MenTaLguY, Mar 6, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mr. SweatyFinger
    Replies:
    2
    Views:
    1,816
    Smokey Grindel
    Dec 2, 2006
  2. bill
    Replies:
    4
    Views:
    433
    Barry Schwarz
    Nov 9, 2005
  3. Dmitry Teslenko
    Replies:
    5
    Views:
    499
    Dmitry Teslenko
    Dec 30, 2009
  4. matt
    Replies:
    1
    Views:
    255
    George Ogata
    Aug 6, 2004
  5. Replies:
    1
    Views:
    222
    Ben Morrow
    Jun 2, 2004
Loading...

Share This Page