Thread reading from a pipe blocks other threads, why?

[Anders Lindgren]

Hi!

I'm implementing an application where the main Ruby program spawns off
a number of external programs using pipes. Each pipe is handled by its
own thread. Well, so far so good... Unfortunately, I'm having problems
since when one thread calls "pipe.each_line" the other (!) threads
hang. Well, if I add a "Process.waitpid" call before the call to
pipe.each_line then the external command will sooner or later hang
since noone consumes anything in the pipe.

To exemplify, below are two ruby programs. The first, lotsoflines.rb,
will print so much output that the pipe will be filles (in the real
world this is typically not written in Ruby). The second, test.rb,
without the "waitpid" will block the thread printing "BG Thread"; with
the "waitpid" line the background process will never terminate.

So, what I would need is a way to read from the pipe *without blocking
other threads*, or a wait until there is data to read, or similar. All
suggestions are welcome!

Oh, btw, I need this to work both under win32 and unix.

-- Anders Lindgren

------------------------------------ lotsoflines.rb
ARGV[0].to_i.times do puts "A LINE" end
sleep 2
------------------------------------ test.rb
cmd = "ruby lotsoflines.rb 300"

Thread.new do
while true
sleep 0.1
puts "BG Thread"
end
end

puts "Before popen"

IO.popen(cmd) do |pipe|

puts "Inside popen"

# With this line, the background thread will be blocked, with this
# line the process at the other end of the pipe will hang...

Process.waitpid(pipe.pid)

pipe.each_line do |x|
puts "XXX:" + x
end
end

puts "After popen"

 

[ara.t.howard]

Hi!

I'm implementing an application where the main Ruby program spawns off a
number of external programs using pipes. Each pipe is handled by its own
thread. Well, so far so good... Unfortunately, I'm having problems since
when one thread calls "pipe.each_line" the other (!) threads hang. Well, if
I add a "Process.waitpid" call before the call to pipe.each_line then the
external command will sooner or later hang since noone consumes anything in
the pipe.

To exemplify, below are two ruby programs. The first, lotsoflines.rb, will
print so much output that the pipe will be filles (in the real world this is
typically not written in Ruby). The second, test.rb, without the "waitpid"
will block the thread printing "BG Thread"; with the "waitpid" line the
background process will never terminate.

So, what I would need is a way to read from the pipe *without blocking other
threads*, or a wait until there is data to read, or similar. All suggestions
are welcome!

Oh, btw, I need this to work both under win32 and unix.

it can't be done using popen and threads with current ruby. search the
archives.

this may help

http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README

gem install systemu

-- Anders Lindgren

------------------------------------ lotsoflines.rb
ARGV[0].to_i.times do puts "A LINE" end
sleep 2
------------------------------------ test.rb
cmd = "ruby lotsoflines.rb 300"

Thread.new do
while true
sleep 0.1
puts "BG Thread"
end
end

puts "Before popen"

IO.popen(cmd) do |pipe|

puts "Inside popen"

# With this line, the background thread will be blocked, with this
# line the process at the other end of the pipe will hang...

Process.waitpid(pipe.pid)

pipe.each_line do |x|
puts "XXX:" + x
end
end

puts "After popen"

-a

 

[Anders Lindgren]

Thanks for the reply!

it can't be done using popen and threads with current ruby. search the
archives.

Well, I did search the archive, but I didn't get a straight answer on
this. I couldn't imagine that it would be impossible to read data from
more than one source at the same time.

You said "current ruby" -- well, maybe there is hope for the future!

I just noticed that in the weekly newsletter it was announced that
Pragmatic Programmers will publish a book on Erlang. This is a
language I worked on many years ago and I haven't really missed it --
until today, that is. The model it uses for building concurrent
applications is really a winner. In that environment you simply
specify the start function (the "main" of the thread), communication
is typically performed by passing messages. The "fork" paradigm and
shared variables used by Ruby is something that really should be
declared as deprecated, since it really makes the program difficult to
write (not to mention impossible to read).

I cross my fingers that the next major version of Ruby uses a non-toy
concurrent solution.

this may help

http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README

gem install systemu

Thanks, I will take a look at it!

-- Anders Lindgren

 

[MenTaLguY]

I cross my fingers that the next major version of Ruby uses a non-toy
concurrent solution.

I'm working on an implementation of actors for Ruby (as part of an omnibus concurrency library). While it doesn't eliminate shared state from the language, in principle it makes it possible to avoid in many cases.

-mental