$stdin, thread blocking and DRb (for lack of a better subject :)

[Ken Hilton]

Greetings friends in Ruby-land,



I've crafted a Ruby application and its giving me a little trouble and I'm
hoping someone out there can give me some insight into the problem.



The app is composed of 3 threads (other than the primary thread, which waits
for the others to finish once its created them): the first thread is bound
to a code block which reads email via NET:OP3; the 2nd thread is that of a
simple DRb server; and the third thread is bound to a code block that
implements a simple command shell which reads lines from $stdin.



The application works wonderfully but w/the following problem: threads 1 and
2 block when the command shell thread reads from $stdin. Adjusting thread
priorities had no affect on the problem. When the command shell thread is
stopped all works as expected (email is read in the background and in
parallel w/the DRb server accepting requests from an test harness running as
a separate ruby session.) I've checked all the Ruby books and docs I have
and can't find an explanation as to why this should be happening. My
suspicion is that because $stdin belongs to the ruby application main thread
and is being used by the command shell thread, that for some reason the
blocking read of $stdin is holding off the other threads. This speculation
is supported by the fact that entering a command to the shell allows all 3
threads to run in parallel until the command is completed and the next read
of $stdin occurs.



Any insight ya'll can lend will be much appreciated.



Ken.

 

[Ara.T.Howard]

Greetings friends in Ruby-land,



I've crafted a Ruby application and its giving me a little trouble and I'm
hoping someone out there can give me some insight into the problem.


The app is composed of 3 threads (other than the primary thread, which waits
for the others to finish once its created them): the first thread is bound
to a code block which reads email via NET:OP3; the 2nd thread is that of a
simple DRb server; and the third thread is bound to a code block that
implements a simple command shell which reads lines from $stdin.


The application works wonderfully but w/the following problem: threads 1 and
2 block when the command shell thread reads from $stdin. Adjusting thread
priorities had no affect on the problem. When the command shell thread is
stopped all works as expected (email is read in the background and in
parallel w/the DRb server accepting requests from an test harness running as
a separate ruby session.) I've checked all the Ruby books and docs I have
and can't find an explanation as to why this should be happening. My
suspicion is that because $stdin belongs to the ruby application main thread
and is being used by the command shell thread, that for some reason the
blocking read of $stdin is holding off the other threads. This speculation
is supported by the fact that entering a command to the shell allows all 3
threads to run in parallel until the command is completed and the next read
of $stdin occurs.

Any insight ya'll can lend will be much appreciated.

Ken.

in ruby, if the process goes to sleep all threads stop because they are
non-native threads...

this program, however, seems to work in 1.6.8 or 1.8.0 so perhaps you have
another problem though?

~ > cat a.rb
#!/usr/bin/env ruby
require 'thread'

begin
require 'readline'
def prompt s
Readline::readline s, true
end
rescue
STDOUT.sync = true
def prompt
print s
gets
end
end

threads = []
data = {}
mutex = Mutex.new

3.times do
threads <<
Thread.new(threads.size) do |tid|
loop{mutex.synchronize{(data[tid] ||= []) << [tid, Time.now]}; sleep 1}
end
end


loop do
cmd = prompt "select a thread #{ (0...threads.size).to_a.join ',' } > "
case cmd
when /exit/io
exit
when /\d+/io
values = mutex.synchronize{data[cmd.to_i]}
(values || []).each{|val| p val}
end
end

~ > ruby a.rb
select a thread 0,1,2 > 1
[1, Tue Apr 27 15:46:46 MDT 2004]
[1, Tue Apr 27 15:46:47 MDT 2004]
select a thread 0,1,2 > 1
[1, Tue Apr 27 15:46:46 MDT 2004]
[1, Tue Apr 27 15:46:47 MDT 2004]
[1, Tue Apr 27 15:46:48 MDT 2004]
[1, Tue Apr 27 15:46:49 MDT 2004]
select a thread 0,1,2 > exit


so, how do you KNOW that all three threads are not running, if you run this
program you'll see that they do seem to - even with the blocking read... i
think ruby tries very hard not to block the process...

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL :: http://www.ngdc.noaa.gov/stp/
| TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
===============================================================================

 

[Ken Hilton]

Greetings friends in Ruby-land,



I've crafted a Ruby application and its giving me a little trouble and I'm
hoping someone out there can give me some insight into the problem.


The app is composed of 3 threads (other than the primary thread, which waits
for the others to finish once its created them): the first thread is bound
to a code block which reads email via NET:OP3; the 2nd thread is that of a
simple DRb server; and the third thread is bound to a code block that
implements a simple command shell which reads lines from $stdin.


The application works wonderfully but w/the following problem: threads 1 and
2 block when the command shell thread reads from $stdin. Adjusting thread
priorities had no affect on the problem. When the command shell thread is
stopped all works as expected (email is read in the background and in
parallel w/the DRb server accepting requests from an test harness running as
a separate ruby session.) I've checked all the Ruby books and docs I have
and can't find an explanation as to why this should be happening. My
suspicion is that because $stdin belongs to the ruby application main thread
and is being used by the command shell thread, that for some reason the
blocking read of $stdin is holding off the other threads. This speculation
is supported by the fact that entering a command to the shell allows all 3
threads to run in parallel until the command is completed and the next read
of $stdin occurs.

Any insight ya'll can lend will be much appreciated.

Ken.

in ruby, if the process goes to sleep all threads stop because they are
non-native threads...

this program, however, seems to work in 1.6.8 or 1.8.0 so perhaps you have
another problem though?

~ > cat a.rb
#!/usr/bin/env ruby
require 'thread'

begin
require 'readline'
def prompt s
Readline::readline s, true
end
rescue
STDOUT.sync = true
def prompt
print s
gets
end
end

threads = []
data = {}
mutex = Mutex.new

3.times do
threads <<
Thread.new(threads.size) do |tid|
loop{mutex.synchronize{(data[tid] ||= []) << [tid, Time.now]}; sleep 1}
end
end


loop do
cmd = prompt "select a thread #{ (0...threads.size).to_a.join ',' } > "
case cmd
when /exit/io
exit
when /\d+/io
values = mutex.synchronize{data[cmd.to_i]}
(values || []).each{|val| p val}
end
end

~ > ruby a.rb
select a thread 0,1,2 > 1
[1, Tue Apr 27 15:46:46 MDT 2004]
[1, Tue Apr 27 15:46:47 MDT 2004]
select a thread 0,1,2 > 1
[1, Tue Apr 27 15:46:46 MDT 2004]
[1, Tue Apr 27 15:46:47 MDT 2004]
[1, Tue Apr 27 15:46:48 MDT 2004]
[1, Tue Apr 27 15:46:49 MDT 2004]
select a thread 0,1,2 > exit


so, how do you KNOW that all three threads are not running, if you run this
program you'll see that they do seem to - even with the blocking read... i
think ruby tries very hard not to block the process...

-a
--
============================================================================
===
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL :: http://www.ngdc.noaa.gov/stp/
| TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
============================================================================
===

Thanks Ara,

I can't try your program due to not having "readline.rb" I that part of the
Ruby 1.8.1 distribution? If not could you provide it to me? As for how (I
believe) I know that my threads are not running, they log events at a very
fine level to a shared log file (mutex protected.) As soon as the command
shell calls $stdin.gets the threads stop logging and the DRb thread stops
responding to remote method calls (they seem to be queued.) As soon as I
press [Enter] on my command shell and gets() returns the log file gets
updated, my POP3 mail fetch completes and DRb responds immediately. And, if
I simply comment out the gets() call and add a hard coded command string
that is processed repeatedly, all of the threads run in parallel. Hmmm....

 

[Ara.T.Howard]

Thanks Ara,

I can't try your program due to not having "readline.rb" I that part of the
Ruby 1.8.1 distribution?

oops.. should have been

'rescue LoadError'

that way it'll work w/o readline....

If not could you provide it to me? As for how (I believe) I know that my
threads are not running, they log events at a very fine level to a shared
log file (mutex protected.) As soon as the command shell calls $stdin.gets
the threads stop logging and the DRb thread stops responding to remote
method calls (they seem to be queued.) As soon as I press [Enter] on my
command shell and gets() returns the log file gets updated, my POP3 mail
fetch completes and DRb responds immediately. And, if I simply comment out
the gets() call and add a hard coded command string
that is processed repeatedly, all of the threads run in parallel. Hmmm....

sounds tough...

- does the stdin read have the mutex while trying to gets?

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL :: http://www.ngdc.noaa.gov/stp/
| TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
===============================================================================