Newbie on Threads

[Nabs Kahn]

I'm creating a screen scraping software and I want to have X (let's say
10 for example) threads running simultaneously doing the scraping. The
program will access a text file containing an unknown number of URLs and
then scrape.

My question is how do I setup the threads so that once a thread finishes
execution it picks up another URL and starts executing again.

Thanks,

Nabs

 

[Robert Klemme]

I'm creating a screen scraping software and I want to have X (let's say
10 for example) threads running simultaneously doing the scraping. The
program will access a text file containing an unknown number of URLs and
then scrape.

My question is how do I setup the threads so that once a thread finishes
execution it picks up another URL and starts executing again.

Create a Queue (require 'thread') and have your worker threads read
URL's from it.

Kind regards

robert

 

[Nabs Kahn]

Thanks for the quick response, this is what I wrote, but it doesn't seem
to work, no errors, but it just finishes the program without doing
anything. Where did I go wrong?

require 'thread'

buffer = SizedQueue.new(10)

producer = Thread.new do
File.open("urls.txt").each do |url|
buffer << url
end
end

consumer = Thread.new do
while buffer.num_waiting != 0
url = buffer.pop
#do screen scraping with url here
end
end

consumer.join

 

[Robert Klemme]

Thanks for the quick response, this is what I wrote, but it doesn't seem
to work, no errors, but it just finishes the program without doing
anything. Where did I go wrong?

I am not sure what you expect: your program does not do anything with
the url. Did you try printing it?

require 'thread'

buffer =3D SizedQueue.new(10)

producer =3D Thread.new do
=A0File.open("urls.txt").each do |url|
=A0 =A0buffer << url

For better readability I suggest using buffer.enq.

=A0end
end

consumer =3D Thread.new do
=A0while buffer.num_waiting !=3D 0

Rather use buffer.deq which is a blocking call. Also, in your case,
the consumer will stop working as soon as the queu has run empty
*once*. You rather want a more reliable termination detection.
Usually I put a special value into the queue (more precisely, as many
special values as there are threads). In your case a symbol would
work.

=A0 =A0url =3D buffer.pop
=A0 =A0#do screen scraping with url here
=A0end
end

consumer.join

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Nabs Kahn]

I am not sure what you expect: your program does not do anything with

the url. Did you try printing it?

I left that part out, figured it would make the example unnecessarily
complex.

Thanks for the tips.

Nabs

 

[Thomas B.]

require 'thread'

buffer = SizedQueue.new(10)

producer = Thread.new do
File.open("urls.txt").each do |url|
buffer << url
end
end

consumer = Thread.new do
while buffer.num_waiting != 0
url = buffer.pop
#do screen scraping with url here
end
end

consumer.join

Hello. You said that you want X threads, but in your example you have
only one scraping thread. I think this is more what you intended (not
tested):

require 'thread'

buffer = SizedQueue.new(10)

producer = Thread.new do
File.open("urls.txt").each do |url|
buffer << url
end
end

consumers = Array::new(x){
Thread.new do
while url=buffer.deq
#do screen scraping with url here
end
end
}

Now if you want the program to stop after the producer finishes, you
should add

producer.join
consumers.each{|c| c.join}

to make sure all processing is finished.

 

[Nabs Kahn]

This is what I ended up doing, similar to what was suggested.
(definition of screenScrape method not included)

bufferSize = 10
buffer = SizedQueue.new(bufferSize)
threads = []

producer = Thread.new do
File.open("urls.txt").each do |url|
buffer.enq url
end
bufferSize.times {buffer.enqEND_OF_WORK)}
end

bufferSize.times do
threads << Thread.new do
url = nil
while(url != :END_OF_WORK)
url = buffer.deq
screenScrape(url)
end
end
end

producer.join
threads.each do |thr|
thr.join
end

 

[Robert Klemme]

This is what I ended up doing, similar to what was suggested.
(definition of screenScrape method not included)

Thanks for the update!

bufferSize =3D 10
buffer =3D SizedQueue.new(bufferSize)
threads =3D []

producer =3D Thread.new do
=A0File.open("urls.txt").each do |url|
=A0 =A0buffer.enq url
=A0end
=A0bufferSize.times {buffer.enqEND_OF_WORK)}
end

bufferSize.times do
=A0threads << Thread.new do
=A0 =A0url =3D nil
=A0 =A0while(url !=3D :END_OF_WORK)
=A0 =A0 =A0url =3D buffer.deq
=A0 =A0 =A0screenScrape(url)
=A0 =A0end
=A0end
end

The loop above does not work properly because you will hand off
:END_OF_WORK to screenScrape(). Rather do

threads << Thread.new do
while ((url =3D buffer.deq) !=3D :END_OF_WORK)
screenScrape(url)
end
end

or

threads << Thread.new do
until ((url =3D buffer.deq) =3D=3D :END_OF_WORK)
screenScrape(url)
end
end

producer.join
threads.each do |thr|
=A0thr.join
end

Btw, you do not need a separate producer thread. You can simply do
that in the main thread. But of course you must start worker threads
before you start to fill the queue.

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Nabs Kahn]

Btw, you do not need a separate producer thread. You can simply do

that in the main thread. But of course you must start worker threads
before you start to fill the queue.

Oh, I thought it would cause the main thread to pause until there was
room in the queue since there would be more urls in the file, but only
10 in the queue, so I created a new thread for the producer thread.
Thanks for the information.

 

[Robert Klemme]

Oh, I thought it would cause the main thread to pause until there was
room in the queue since there would be more urls in the file, but only
10 in the queue, so I created a new thread for the producer thread.

Well, it will. But so will your producer thread and eventually the main
thread since it joins on the producer. The producer thread brings you
only advantages if you need to do other things in the main thread. But
since you don't in the code you presented you can as well do the queue
filling in the main thread.

Btw, I just notice one thing: you don't chomp the lines read from the
file. So your URL's will still contain the trailing line feed.

Thanks for the information.

You're welcome!

Kind regards

robert

 

[Nabs Kahn]

Well, it will. But so will your producer thread and eventually the main

thread since it joins on the producer. The producer thread brings you
only advantages if you need to do other things in the main thread. But
since you don't in the code you presented you can as well do the queue
filling in the main thread.

I see, that makes sense.

Btw, I just notice one thing: you don't chomp the lines read from the
file. So your URL's will still contain the trailing line feed.

I am actually chomping them inside the screenScraper method.

Nabs

 

[Robert Klemme]

I see, that makes sense.



I am actually chomping them inside the screenScraper method.

I wouldn't do that. Preparation of the input should be done outside of
your scraping method. You put too much knowledge about the environment
into the method.

Cheers

robert

 

[Nabs Kahn]

I wouldn't do that. Preparation of the input should be done outside of

your scraping method. You put too much knowledge about the environment
into the method.

Right, I see your point, I'll change it. Thanks again.

Nabs