error handling with exceptions in multi-thread download

Pen Ttt · Aug 27, 2010

there are three program fragments,all of them have same problems,
all of them can run ,but when i downloaded 30% data,an error
ocurr
undefined local variable or method `web' for Data:Class (NameError)
cthreaddown is an array which contains many web.
would you kind to tell me how to fix it?
p1

Code:

for page_to_fetch in cthreaddown
threads<<Thread.new(page_to_fetch) do |web|
datafile="/home/pt/stock/"
open(datafile,'a+'){|refile|
open(web){|webfile|
refile.puts  webfile.read
puts  "#{web}  is over"
}}
end
end
threads.each {|thr|
begin
thr.join
rescue Timeout::Error => e
puts    "#{web}  failed"
rescue NameError => e
puts    "#{web}  failed"
rescue => e
puts    "#{web}  failed"
end
}

p2

Code:

for page_to_fetch in cthreaddown
threads<<Thread.new(page_to_fetch) do |web|
datafile="/home/pt/stock/"
open(datafile,'a+'){|refile|
begin
open(web){|webfile|
refile.puts  webfile.read
puts "#{web}  is over"
}}
rescue NameError => e
puts    "#{web}  failed"
rescue => e
puts  "#{web}  wrong"
end
end
end
threads.each {|thr|  thr.join }

p3

Code:

for page_to_fetch in cthreaddown
threads<<Thread.new(page_to_fetch) do |web|
datafile="/home/pt/stock/"
open(datafile,'a+'){|refile|
begin
open(web){|webfile|
refile.puts  webfile.read
puts "#{web}  is over"
}}
rescue NameError => e
puts    "#{web}  failed"
rescue => e
puts  "#{web}  wrong"
end
end
end
threads.each {|thr|
begin
thr.join
rescue Timeout::Error => e
puts    "#{web}  failed"
rescue NameError => e
puts    "#{web}  failed"
rescue => e
puts    "#{web}  failed"
end
}

LAMBEAU Bernard · Aug 27, 2010

[Note: parts of this message were removed to make it a legal post.]

You probably have a TimeOut error. And the errors actually comes from:

rescue Timeout::Error => e
puts "#{web} failed"

because web is not defined there.

B

Pen Ttt · Aug 27, 2010

i have omitted many lines of code,no TimeOut error,
for page_to_fetch in cthreaddown
threads<<Thread.new(page_to_fetch) do |web|
i have got an array--- cthreaddown before
it can run , can get 30% data ,and then stop,

undefined local variable or method `web' for Data:Class (NameError)

Robert Klemme · Aug 27, 2010

2010/8/27 Pen Ttt said:
=A0 =A0there are three =A0program fragments,all of them =A0have same prob= lems,
=A0 =A0all =A0of them =A0can run ,but when i downloaded =A030% data,an er= ror
ocurr
=A0 =A0undefined local variable or method `web' for Data:Class (NameError=

)

You need to look into class methods of class Data. Somewhere there
you will find usage of "web" which is undefined. Unfortunately you
did not post that class so we cannot tell you where your error is.

=A0 =A0cthreaddown =A0is an =A0array =A0which contains many web.
=A0 =A0would you kind to tell =A0me how to fix =A0it?
=A0 =A0p1
=A0 =A0

Code:

=A0 =A0 for page_to_fetch in cthreaddown
=A0 =A0 threads<<Thread.new(page_to_fetch) do |web|
=A0 =A0 =A0 =A0 datafile=3D"/home/pt/stock/"
=A0 =A0 =A0 =A0 open(datafile,'a+'){|refile|
=A0 =A0 =A0 =A0 open(web){|webfile|
=A0 =A0 =A0 =A0 =A0 =A0 =A0refile.puts =A0webfile.read[/QUOTE]

You should use #write with #read and not #puts, because #puts does not
write out arguments unmodified.  Also note that if webfile is large
this approach will break.  You should better use blocked reading.

buffer =3D ""
while webfile.read(1024, buffer)
refile.write buffer
end
[QUOTE]
=A0 =A0 =A0 =A0 =A0 =A0 =A0puts =A0"#{web} =A0is over"
=A0 =A0 =A0 =A0 }}
=A0 =A0 =A0 =A0end
=A0 =A0 =A0end
=A0 =A0 threads.each {|thr|
=A0 =A0 =A0 =A0begin
=A0 =A0 =A0 =A0 thr.join
=A0 =A0 =A0 =A0 rescue Timeout::Error =3D> e
=A0 =A0 =A0 =A0 =A0 puts =A0 =A0"#{web} =A0failed"
=A0 =A0 =A0 =A0 rescue NameError =3D> e
=A0 =A0 =A0 =A0 =A0 puts =A0 =A0"#{web} =A0failed"
=A0 =A0 =A0 =A0 rescue =3D> e
=A0 =A0 =A0 =A0 =A0 puts =A0 =A0"#{web} =A0failed"
=A0 =A0 =A0 =A0end
=A0 =A0 =A0 }
=A0 =A0 =A0

Cheers

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

Pen Ttt · Aug 27, 2010

require 'rubygems'
require 'open-uri'
require 'nokogiri'
require 'mysql'
require 'date'

class Data
def self.getdaystock()
time0=Time.now
i=0
cdown=[]
threads=[]
cthreaddown=[]
open("/home/pt/usa/stocklist","r"){|item|
while line=item.gets
cdown<<line
end}
a=(fromdate.split("-")[1].to_i-1).to_s
b=fromdate.split("-")[2].to_s
c=fromdate.split("-")[0].to_s
d=Time.now.mon.to_s
e=Time.now.day.to_s
f=Time.now.year.to_s
cdown=cdown.map {|item|
item='http://ichart.finance.yahoo.com/table.csv?s='+item.to_s+'&a='+a+'&b='+b+'&c='+c+\
'&d='+d+'&e='+e+'&f='+f+'&g=d&ignore=.csv'}
cdown.each.with_index {|item,idx|
cthreaddown<< item
i=i+1
if i==100 or idx==cdown.size-1 then
cthreaddown=cthreaddown.uniq
for page_to_fetch in cthreaddown
threads<<Thread.new(page_to_fetch) do |web|
datafile="/home/pt/usa/stock/"+web.split("=")[1].to_s.gsub("&a","")
begin
open(datafile,'a+'){|refile|
open(web){|webfile|
refile.puts webfile.read
puts "#{web} over" }}
rescue Timeout::Error => e
puts "#{web} failed"
rescue NameError => e
puts "#{web} failed"
rescue => e
puts "#{web} wrong"
end
end
end
threads.each {|thr|
begin
thr.join
rescue Timeout::Error => e
puts "#{web} failed"
rescue NameError => e
puts "#{web} failed"
rescue => e
puts "#{web} failed"
end }
i=0
cthreaddown=[]
threads = []
sleep 0.01
else
end
}
end
end

i have opened 100 threads. there are 6000 companies in
/home/pt/usa/stocklist .

Pen Ttt · Aug 27, 2010

sorry for that: def self.getdaystock(fromdate)

Robert Klemme · Aug 27, 2010

require 'rubygems'
require 'open-uri'
require 'nokogiri'
require 'mysql'
require 'date'

class Data
def self.getdaystock()
time0=Time.now
i=0
cdown=[]
threads=[]
cthreaddown=[]
open("/home/pt/usa/stocklist","r"){|item|
while line=item.gets
cdown<<line
end}
a=(fromdate.split("-")[1].to_i-1).to_s
b=fromdate.split("-")[2].to_s
c=fromdate.split("-")[0].to_s
d=Time.now.mon.to_s
e=Time.now.day.to_s
f=Time.now.year.to_s
cdown=cdown.map {|item|
item='http://ichart.finance.yahoo.com/table.csv?s='+item.to_s+'&a='+a+'&b='+b+'&c='+c+\
'&d='+d+'&e='+e+'&f='+f+'&g=d&ignore=.csv'}
cdown.each.with_index {|item,idx|
cthreaddown<< item
i=i+1
if i==100 or idx==cdown.size-1 then
cthreaddown=cthreaddown.uniq
for page_to_fetch in cthreaddown
threads<<Thread.new(page_to_fetch) do |web|
datafile="/home/pt/usa/stock/"+web.split("=")[1].to_s.gsub("&a","")
begin
open(datafile,'a+'){|refile|
open(web){|webfile|
refile.puts webfile.read
puts "#{web} over" }}
rescue Timeout::Error => e
puts "#{web} failed"
rescue NameError => e
puts "#{web} failed"
rescue => e
puts "#{web} wrong"
end
end
end
threads.each {|thr|
begin
thr.join
rescue Timeout::Error => e
puts "#{web} failed"
rescue NameError => e
puts "#{web} failed"
rescue => e
puts "#{web} failed"
end }
i=0
cthreaddown=[]
threads = []
sleep 0.01
else
end
}
end
end

i have opened 100 threads. there are 6000 companies in
/home/pt/usa/stocklist .

Well, there you have it. The error is in your exception handling code.

Btw, the whole code could use a bit of commenting and also be simplified
in areas. For example: you store Time.now initially and then call it
over and over again to fetch individual fields. This is not only
inefficient but also error prone because you will not get all values
from the same point in time.

Also, it's a bad die to hardcode the URL in the middle of the code.
It's much better to pass this as parameter or at least store it in some
constant for easier reference and documentation purposes.

Cheers

robert

synchronize between download and save in multi-threads?	1	Aug 14, 2010
Multithreading problem	7	Aug 12, 2010
how many thread?!	1	Jul 31, 2009
Timeouts with Thread#join and Net:HTTP	2	Mar 18, 2009
Handling exceptions in Threads	3	Jul 15, 2008
thread exec	3	Jul 30, 2009
Thread leak?	1	Oct 13, 2005
trouble with threads	5	Feb 21, 2008

error handling with exceptions in multi-thread download

Pen Ttt

LAMBEAU Bernard

Pen Ttt

Robert Klemme

Pen Ttt

Pen Ttt

Robert Klemme

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads