How to control thread number

Li Chen · Sep 15, 2008

Hi all,

I copy the following script from Ruby programming language(bundling
with Ruby). If the size for pages (array size) is small,thread works
very well. If the array size is big (such 2000) the thread doesn't work
at or very slow. I wonder what is the best means to control thread
number based on this script, read a small trunk of array at a time using
each_slice method?

Thanks,

Li

##################### script#####################
pages = %w( www.rubycentral.com
www.awl.com
www.pragmaticprogrammer.com
)

threads = []

for page in pages
threads << Thread.new(page) { |myPage|

h = Net::HTTP.new(myPage, 80)
puts "Fetching: #{myPage}"
resp, data = h.get('/', nil )
puts "Got #{myPage}: #{resp.message}"
}
end

ara.t.howard · Sep 15, 2008

Hi all,

I copy the following script from Ruby programming language(bundling
with Ruby). If the size for pages (array size) is small,thread works
very well. If the array size is big (such 2000) the thread doesn't
work
at or very slow. I wonder what is the best means to control thread
number based on this script, read a small trunk of array at a time
using
each_slice method?

Thanks,

Li

##################### script#####################
pages = %w( www.rubycentral.com
www.awl.com
www.pragmaticprogrammer.com
)

threads = []

for page in pages
threads << Thread.new(page) { |myPage|

h = Net::HTTP.new(myPage, 80)
puts "Fetching: #{myPage}"
resp, data = h.get('/', nil )
puts "Got #{myPage}: #{resp.message}"
}
end

this is exactly what threadify does

cfp:~ > cat a.rb
require 'open-uri'

require 'rubygems'
require 'threadify'

# gem install threadify
#
# http://codeforpeople.com/lib/ruby/threadify/threadify-0.0.3/README

uris =
%w(
http://www.ruby-lang.com
http://www.awl.com
http://www.pragmaticprogrammer.com
)

uris.threadify(2) do |uri|
tid = Thread.current.object_id
response = open(uri){|fd| fd.read}
puts "#{ tid } : #{ uri } => #{ response[0,42].inspect }"
end

cfp:~ > ruby a.rb
1867720 : http://www.ruby-lang.com => "<html>\n<head>\n<title>Ruby
Land</title>\n<me"
1864060 : http://www.awl.com => "<!DOCTYPE html PUBLIC \"-//W3C//DTD
XHTML 1"
1867720 : http://www.pragmaticprogrammer.com => "<!DOCTYPE html PUBLIC
\"-//W3C//DTD XHTML 1"

this implementation is very short, you can read it so see one
technique - which uses sized queues with a fixed number of producers
and consumers

http://codeforpeople.com/lib/ruby/threadify/threadify-0.0.3/lib/threadify.rb

cheers.

a @ http://codeforpeople.com/

Lex Williams · Sep 15, 2008

are you calling join on the threads?

Erik Veenstra · Sep 15, 2008

That's exactly what ThreadLimiter is designed for:

http://www.erikveen.dds.nl/threadlimiter/doc/index.html

gegroet,
Erik V.

Thread error "undefined method `keys' for nil:NilClass"	0	Oct 29, 2009
How to call Ruby from non-interpreter native thread?	1	Apr 1, 2011
Thread pool issue	1	Mar 18, 2010
How to make net-ping thread safe?	14	Mar 15, 2008
Ruby Threads question	4	Sep 6, 2007
Ruby thread-dump Gem	4	Jul 20, 2007
how to translate base 10 number into base 2 number	6	Jan 14, 2007
How to use open uri or net/http class	3	Jun 2, 2005

How to control thread number

Li Chen

ara.t.howard

Lex Williams

Erik Veenstra

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads