Web Crawling/Threading and Things That Go Bump in the Night

R

Remarkable

Hello all

I am trying to write a reliable web-crawler. I tried to write my own
using recursion and found I quickly hit the "too many sockets" open
problem. So I looked for a threaded version that I could easily extend.

The simplest/most reliable I found was called Spider.py (see attached).

At this stage I want a spider that I can point at a site, let it do
it's thing, and reliable get a callback of sorts... including the html
(for me to parse), the url of the page in question (so I can log it)
and the urls-found-on-that-page (so I can strip out any ones I really
don't want and add them to the "seen-list".


Now, this is my question.

The code above ALMOST works fine. The crawler crawls, I get the data I
need BUT... every now and again the code just pauses, I hit control-C
and it reports an error as if it has hit an exception and then carries
on!!! I like the fact that my spider_usage.py file has the minimum
amount of spider stuff in it... really just a main() and handle()
handler.

How does this happen... is a thread being killed and then a new one is
made or what? I suspect it may have something to do with sockets timing
out, but I have no idea...

By the way on small sites (100s of pages) it never gets to the stall,
it's on larger sites such as Amazon that it "fails"

This is my other question

It would be great to know, when the code is stalled, if it is doing
anything... is there any way to even print a full stop to screen?

This is my last question

Given python's suitability for this sort of thing (isn't google written
in it?) I can't believe that that there isn't a kick ass crawler
already out there...

regards

tom

http://www.theotherblog.com/Articles/2006/08/04/python-web-crawler-spider/
 
S

sp1d3rx

Rem, what OS are you trying this on? Windows XP SP2 has a limit of
around 40 tcp connections per second...
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top