Thread deadlock

C

castillo.bryan

I wanted to create a thread pool. I know I could have used a
SizedQueue in the thread pool, but I wanted to later on change the
thread queue to a priority queue.

With ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-cygwin] I get a
deadlock error. ruby 1.8.5 (2006-08-25) [i386-mswin32] is not giving
me that error. I think that my code is written correctly and that
this may be a bug in the version/build of ruby I am running on. Does
anybody see a problem with this code? Thanks.

require 'thread'

class ThreadPool

def initialize(thread_size=10, queue_size=100)
@mutex = Mutex.new
@cv = ConditionVariable.new
@queue = []
@max_queue_size = queue_size
@threads = []
thread_size.times { @threads << Thread.new { start_worker } }
end

def add_work(*args, &callback)
push_task(Task.new(*args, &callback))
end

def push_task(task)
puts "#{Thread.current}|push_task|sync on @mutex"
@mutex.synchronize do
while @max_queue_size > 0 && @queue.size >= @max_queue_size do
puts "#{Thread.current}|push_task|wait on @mutex"
@cv.wait(@mutex)
end
@queue.push(task)
puts "#{Thread.current}|execute|broadcast on @mutex"
@cv.broadcast
end
puts "#{Thread.current}|push_task|done with sync on @mutex"
task
end

def pop_task
task = nil
puts "#{Thread.current}|pop_task|sync on @mutex"
@mutex.synchronize do
while @queue.size == 0 do
puts "#{Thread.current}|pop_task|wait on @mutex"
@cv.wait(@mutex)
end
task = @queue.shift
puts "#{Thread.current}|execute|broadcast on @mutex"
@cv.broadcast
end
puts "#{Thread.current}|pop_task|done with sync on @mutex"
task
end

def start_worker
puts "#{Thread.current} running worker"
while true
task = pop_task
return if task == :stop
task.execute
end
end

class Task

attr_reader :result, :exception

def initialize(*args, &callback)
@args = args
@callback = callback
@done = false
@result = nil
@exception = nil
@mutex = Mutex.new
@cv = ConditionVariable.new
end

def execute
begin
@result = @callback.call(*@args)
rescue Exception => e
@exception = e
STDERR.puts "Error in thread #{Thread.current} - #{e}"
e.backtrace.each { |element| STDERR.puts(element) }
end
puts "#{Thread.current}|execute|sync on @mutex"
@mutex.synchronize do
@done = true
puts "#{Thread.current}|execute|broadcast on @mutex"
@cv.broadcast
end
puts "#{Thread.current}|execute|done with sync on @mutex"
end

def join
puts "#{Thread.current}|join|sync on @mutex"
@mutex.synchronize do
while !@done
puts "#{Thread.current}|join|wait on @mutex"
@cv.wait(@mutex)
end
end
puts "#{Thread.current}|join|done with sync on @mutex"
end

end

end

tasks = []
tp = ThreadPool.new(3, 10)
sleep(1)
100.times do |id|
STDERR.puts "adding work"
tasks << tp.add_work do
puts "Running #{id} #{Thread.current}"
sleep 2
puts "Ending #{id} #{Thread.current}"
Time.now
end
end

puts "Waiting for tasks to complete"
tasks.each do |task|
task.join
if !task.exception.nil?
puts "Failed task - #{task.exception}"
else
puts "Result - #{task.result}"
end
end


output:
--------------------------
$ ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-cygwin]
[/users/bcastill/tpool]
$ ruby thread_pool.rb
#<Thread:0x1002b63c> running worker
#<Thread:0x1002b63c>|pop_task|sync on @mutex
#<Thread:0x1002b63c>|pop_task|wait on @mutex
#<Thread:0x1002b4fc> running worker
#<Thread:0x1002b4fc>|pop_task|sync on @mutex
#<Thread:0x1002b4fc>|pop_task|wait on @mutex
#<Thread:0x1002b3d0> running worker
#<Thread:0x1002b3d0>|pop_task|sync on @mutex
#<Thread:0x1002b3d0>|pop_task|wait on @mutex
adding work
#<Thread:0x1003c964>|push_task|sync on @mutex
#<Thread:0x1003c964>|execute|broadcast on @mutex
#<Thread:0x1002b3d0>|execute|broadcast on @mutex
#<Thread:0x1003c964>|push_task|done with sync on @mutex
adding work
#<Thread:0x1003c964>|push_task|sync on @mutex
#<Thread:0x1002b4fc>|pop_task|wait on @mutex
#<Thread:0x1002b3d0>|pop_task|done with sync on @mutex
Running 0 #<Thread:0x1002b3d0>
Ending 0 #<Thread:0x1002b3d0>
#<Thread:0x1002b3d0>|execute|sync on @mutex
#<Thread:0x1002b3d0>|execute|broadcast on @mutex
#<Thread:0x1002b3d0>|execute|done with sync on @mutex
#<Thread:0x1002b3d0>|pop_task|sync on @mutex
#<Thread:0x1002b3d0>|pop_task|wait on @mutex
deadlock 0x1002b4fc: sleep:- - thread_pool.rb:39
deadlock 0x1002b63c: sleep:- - thread_pool.rb:39
deadlock 0x1003c964: sleep:- (main) - thread_pool.rb:20
deadlock 0x1002b3d0: sleep:- - thread_pool.rb:39
thread_pool.rb:39:in `push_task': Thread(0x1002b3d0): deadlock (fatal)
from thread_pool.rb:15:in `add_work'
from thread_pool.rb:109
from thread_pool.rb:107:in `times'
from thread_pool.rb:107
 
C

castillo.bryan

I wanted to create a thread pool. I know I could have used a
SizedQueue in the thread pool, but I wanted to later on change the
thread queue to a priority queue.

With ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-cygwin] I get a
deadlock error. ruby 1.8.5 (2006-08-25) [i386-mswin32] is not giving
me that error. I think that my code is written correctly and that
this may be a bug in the version/build of ruby I am running on. Does
anybody see a problem with this code? Thanks.
Click to expand...

I've tried using some different version of ruby to see if the code
works or not:

Failed versions:
------------------------
ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-linux]
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-cygwin]
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]

Working versions:
------------------------
ruby 1.8.5 (2006-08-25) [i386-mswin32]
ruby 1.8.5 (2007-08-27 rev 4201) [x86-jruby1.0.1]
ruby 1.8.3 (2005-09-21) [i686-linux]
ruby 1.9.0 (2007-09-15 patchlevel 0) [i386-cygwin]


It seems to me like there is something wrong with the latest stable
version of ruby.
 
M

MenTaLguY

It seems to me like there is something wrong with the latest stable
version of ruby.
Click to expand...

1.8.6-p0 is broken, but it is not the most recent stable release.

The latest stable release is 1.8.6-p36.

-mental
 
M

MenTaLguY

I received the error with the stable recommended version from
http://www.ruby-lang.org/en/downloads/.
Click to expand...

Hmm. It looks like that is different from p36 (the file size is different):

ftp://ftp.ruby-lang.org/pub/ruby/1.8/

Try the -p36 tarball and see if it makes a difference (it should, based
on others' experience).

Perhaps someone running the site needs to be more diligent about what is
offered for download.

-mental
 
C

castillo.bryan

Hmm. It looks like that is different from p36 (the file size is different):

ftp://ftp.ruby-lang.org/pub/ruby/1.8/

Try the -p36 tarball and see if it makes a difference (it should, based
on others' experience).
Click to expand...

Yes, p36 works fine. I should have done a search for deadlocks on this
group before posting.

http://groups.google.com/group/comp...gst&q=thread+deadlock&rnum=3#9f80264e115a1235


Perhaps someone running the site needs to be more diligent about what is
offered for download.
Click to expand...

That does concern me. The windows binary on http://www.ruby-lang.org/en/downloads/
points to p0 as well.

Is it a problem with updating the web page or should those links be
soft links to the latest 1.8.6 version?

Here is the directory listing from the ftp server in /pub/ruby

lrwxrwxrwx 1 1014 100 26 Aug 02 12:16 ruby-1.8.6-
p36.tar.bz2 -> 1.8/ruby-1.8.6-p36.tar.bz2
lrwxrwxrwx 1 1014 100 25 Aug 02 12:16 ruby-1.8.6-
p36.tar.gz -> 1.8/ruby-1.8.6-p36.tar.gz
lrwxrwxrwx 1 1014 100 22 Aug 02 12:16 ruby-1.8.6-
p36.zip -> 1.8/ruby-1.8.6-p36.zip
lrwxrwxrwx 1 1000 100 22 Mar 12 2007
ruby-1.8.6.tar.bz2 -> 1.8/ruby-1.8.6.tar.bz2
lrwxrwxrwx 1 1000 100 21 Mar 12 2007
ruby-1.8.6.tar.gz -> 1.8/ruby-1.8.6.tar.gz
lrwxrwxrwx 1 1000 100 18 Mar 12 2007 ruby-1.8.6.zip
-> 1.8/ruby-1.8.6.zip
 
C

castillo.bryan

Yes, p36 works fine. I should have done a search for deadlocks on this
group before posting.

http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/ec...


That does concern me. The windows binary onhttp://www.ruby-lang.org/en/downloads/
points to p0 as well.
Click to expand...

I sent an email to the webmaster at www.ruby-lang.org about having p0
for the main download links, while p36 seems to be the stable one.

My Email
-----------------------
On the downloads page http://www.ruby-lang.org/en/downloads/, there
are links to 1.8.6 recommended versions of ruby that may not be the
latest stable version for 1.8.6. I recently ran into a bug with the
versions of ruby currently linked to on the download page, that was
fixed in 1.8.6-p36. I was wondering if the download page should be
updated to the URLS listed below.

Ruby 1.8.6 Source:
ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.6-p36.tar.gz

Windows 1.8.6 binary:
ftp://ftp.ruby-lang.org/pub/ruby/binaries/mswin32/ruby-1.8.6-p36-i386-mswin32.zip


Here is a link to a thread in comp.lang.ruby, where someone pointed
out I should be using 1.8.6-p36.

http://groups.google.com/group/comp...f0/43e0ed11f8117add?lnk=raot#43e0ed11f8117add
-----------------------------




Here is the response from the web master, James.

Response
-------------------------------
Does any one know who put these versions up? I don't want to undo
something we've done. Do we know what releases these actually are?
Thanks for the information.
-------------------------------




So, I'd like to put his question out to to the mailing list. Should
1.8.6-p36 be the recommended stable release for ruby?
 
C

castillo.bryan

1.8.6-p0 is broken, but it is not the most recent stable release.

The latest stable release is 1.8.6-p36.

-mental
Click to expand...

Even though 1.8.6-p0 was not working with Mutex/ConditionVariable, I
changed the code to use the Monitor class. I did not receive the
error even in 1.8.6-p0.

I found this post about the differences between Mutex and Monitor.

http://groups.google.com/group/comp...st&q=Monitor+vs+Mutex&rnum=1#0412d7952937abd8

I think I will stick to using Monitor for future code.


Here is the code that works on 1.8.6-p0:


require 'thread'
require 'monitor'

class ThreadPool

class PoolStopped < Exception; end

def initialize(thread_size=10, queue_size=100)
@mutex = Monitor.new
@cv = @mutex.new_cond
@Queue = []
@max_queue_size = queue_size
@threads = []
@stopped = false
thread_size.times { @threads << Thread.new { start_worker } }
end

def add_work(*args, &callback)
push_task(Task.new(*args, &callback))
end

def push_task(task)
@mutex.synchronize do
raise PoolStopped.new if @stopped
@cv.wait_while { @max_queue_size > 0 && @queue.size >=
@max_queue_size }
@queue.push(task)
@cv.broadcast
end
task
end

def pop_task
task = nil
@mutex.synchronize do
@cv.wait_while { @queue.size == 0 }
task = @queue.shift
@cv.broadcast
end
task
end

def shutdown
@mutex.synchronize do
@stopped = true
@threads.each { @queue.push:)stop) }
@cv.broadcast
end
@threads.each { |thread| thread.join }
end

def start_worker
while true
task = pop_task
return if task == :stop
task.execute
end
end

# wait for current work to complete
def sync
tasks = @mutex.synchronize { @queue.dup }
tasks.each { |task| task.join }
end

class Task

attr_reader :result, :exception

def initialize(*args, &callback)
@args = args
@callback = callback
@done = false
@result = nil
@exception = nil
@mutex = Monitor.new
@cv = @mutex.new_cond
end

def execute
begin
@result = @callback.call(*@args)
rescue Exception => e
@exception = e
STDERR.puts "Error in thread #{Thread.current} - #{e}"
e.backtrace.each { |element| STDERR.puts(element) }
end
@mutex.synchronize do
@done = true
@cv.broadcast
end
end

def join
@mutex.synchronize { @cv.wait_until { @done } }
end

end

end

tasks = []
tp = ThreadPool.new(10, 1000)
sleep(1)
100.times do |id|
STDERR.puts "adding work"
tasks << tp.add_work do
puts "Running #{id} #{Thread.current}"
sleep 5
puts "Ending #{id} #{Thread.current}"
end
end

puts "Waiting for shutdown"
tp.shutdown
puts "done"
 
M

MenTaLguY

--=-6CgKHFwW9Cz82GXD/AG3
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

I think I will stick to using Monitor for future code.
Click to expand...

I'd strongly recommend against using Monitor -- it is slow and has some
subtle bugs. Have you considered using fastthread?

-mental

--=-6CgKHFwW9Cz82GXD/AG3
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBG80VhSuZBmZzm14ERAsxAAKDdJFAu/I4t8n9oyKKuZHgdTXC87QCfc4r8
vLJ3nBQ+k6yYpRTRDYmq/ZQ=
=7RB/
-----END PGP SIGNATURE-----

--=-6CgKHFwW9Cz82GXD/AG3--
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Mechanize MySQL and threads - deadlock? 1
deadlock in ThreadPool using backtick 15
Multi Threading 5
Thread deadlock problem 5
Deadlock falsely detected when using condition variables 3
Mutex 7
Thread and Mutex in Ruby 1.8.7 2
Thread Deadlock on Ruby 1.8.7 5

Members online

Total: 35 (members: 3, guests: 32)
Robots: 343

Forum statistics

Threads
473,769
Messages
2,569,582
Members
45,071
Latest member
MetabolicSolutionsKeto

Latest Threads

Top