Ruby Threading question Creating 4 threads

[tristin.colby]

Newbie Question:

I have some code where I create 4 threads that pop elements off an
array and process them. The problem is that each thread only pops off
one element, processes it, then returns; they don't continue to
process until the entire array is consumed.

<code>
$jobs = %w[a b c d e f g]
def simulate_cleanup

puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
threads = []

mu = Mutex.new
total = 0

1.upto(4) do |c|
threads << Thread.new do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and return if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end
threads.each { |t| t.join }

puts "Reduced by #{total} files"
end

Example of output:
Thread 1 created
Thread 2 created
Thread 3 created
Thread 4 created
Checking for a
Checking for b
Checking for c
Checking for d

 

[Louis-Philippe]

[Note: parts of this message were removed to make it a legal post.]

what if you put your thread body in an infinite loop, something like:

Thread.new do
loop do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and Thread.exit if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end

and as you may have noticed, I replaced the 'return' statement inside your
thread with a 'Thread.exit' method.
I think it would be the right way to do it...

 

[Louis-Philippe]

[Note: parts of this message were removed to make it a legal post.]

and looking at it more, I think it would be better to leave the less stuff
in the mutex block as possible,
making it something something like:

mu.synchronize do
job = $jobs.pop
end

puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and Thread.exit if job.nil?

I guess the Thread.exit could have caused a deadlock if inside the mutex,
and since the mutex locks your other threads you better keep it as small as
possible for efficiency sake.

what if you put your thread body in an infinite loop, something like:

Thread.new do
loop do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and Thread.exit if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end

and as you may have noticed, I replaced the 'return' statement inside your
thread with a 'Thread.exit' method.
I think it would be the right way to do it...

2009/2/16 <[email protected]>

Newbie Question:

I have some code where I create 4 threads that pop elements off an
array and process them. The problem is that each thread only pops off
one element, processes it, then returns; they don't continue to
process until the entire array is consumed.

<code>
$jobs = %w[a b c d e f g]
def simulate_cleanup

puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
threads = []

mu = Mutex.new
total = 0

1.upto(4) do |c|
threads << Thread.new do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and return if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end
threads.each { |t| t.join }

puts "Reduced by #{total} files"
end

Example of output:
Thread 1 created
Thread 2 created
Thread 3 created
Thread 4 created
Checking for a
Checking for b
Checking for c
Checking for d

 

[Julian Leviston]

Put a while around the pop and block of code to run.

Blog: http://random8.zenunit.com/
Learn rails: http://sensei.zenunit.com/

 

[Julian Leviston]

That won't ever exit the thread because puts returns nil.

Blog: http://random8.zenunit.com/
Learn rails: http://sensei.zenunit.com/

what if you put your thread body in an infinite loop, something like:

Thread.new do
loop do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and Thread.exit if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end

and as you may have noticed, I replaced the 'return' statement
inside your
thread with a 'Thread.exit' method.
I think it would be the right way to do it...

Newbie Question:

I have some code where I create 4 threads that pop elements off an
array and process them. The problem is that each thread only pops
off
one element, processes it, then returns; they don't continue to
process until the entire array is consumed.

<code>
$jobs = %w[a b c d e f g]
def simulate_cleanup

puts "Checking for #{$jobs.length} jobs in /mnt/gen1/focal/temp"
threads = []

mu = Mutex.new
total = 0

1.upto(4) do |c|
threads << Thread.new do
puts "Thread #{c} created"
job = nil

mu.synchronize do
job = $jobs.pop
puts "Checking for #{job}"
#Exit our thread if we have no more jobs to check
puts "No more jobs" and return if job.nil?
end

t = count_files(job)
mu.synchronize { total += t }
end
end
threads.each { |t| t.join }

puts "Reduced by #{total} files"
end

Example of output:
Thread 1 created
Thread 2 created
Thread 3 created
Thread 4 created
Checking for a
Checking for b
Checking for c
Checking for d

 

[Robert Klemme]

Put a while around the pop and block of code to run.

Also class Queue has thread safety built in already.

Kind regards

robert

 

[7stud --]

Also class Queue has thread safety built in already.

Kind regards

robert

...an example of using a Queue:

require 'thread'

arr = %w[a b c d e f g h i j]
num_threads = 4
q = Queue.new

workers = []

num_threads.times do |i|
workers << Thread.new("worker#{i+1}") do |name|
loop do
data = q.pop #pops data off bottom of q, blocks if no data
puts "#{name} retrieved data: #{data}"
break if data == "*END_OF_DATA_FLAG*"
end
end
end

arr.each {|item| q.push(item)}
num_threads.times do
q.push("*END_OF_DATA_FLAG*")
end

workers.each {|thr| thr.join}
puts "Workers are all done working."


--output:--
worker1 retrieved data: a
worker2 retrieved data: b
worker3 retrieved data: c
worker4 retrieved data: d
worker1 retrieved data: e
worker2 retrieved data: f
worker3 retrieved data: g
worker4 retrieved data: h
worker1 retrieved data: i
worker2 retrieved data: j
worker3 retrieved data: *END_OF_DATA_FLAG*
worker4 retrieved data: *END_OF_DATA_FLAG*
worker1 retrieved data: *END_OF_DATA_FLAG*
worker2 retrieved data: *END_OF_DATA_FLAG*
Workers are all done working.