open uri buf read overflow

C

Chuck Dawit

I'm using open-uri to open approx 3500 web pages to check to see if they
have active contect. I'm using open uri to do it, but after about 2346
pages have been open I get a buf_read error and the program crashes.
After looking in the open-uri documentation I couldn't find anything
that closes the connection and dumps the memory after the open method is
called. Does anyone know how to dump the memory after I make each open()
call?
begin
while (line = f.readline)
line = line.split(/\n/)
url = "http://www." + line[0] + ".com"
#puts url
begin
open(url) { #|page| page_content = page.read()
puts url + " exists"
nf.puts url
}
rescue
puts"URL does not exist"
#retry
end
end
rescue EOFError
f.close
nf.close
end
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
474,432
Messages
2,571,681
Members
48,796
Latest member
Greg L.

Latest Threads

Top