crawlers in python with graphing?

[bryan rasmussen]

Hi

I'm wondering if there is a toolkit in python anywhere for doing at a
high level web crawling, dumping links to a data set that could be
imported into R relatively easy or used in python natively to generate
a graph over the website.


It should hopefully be as high level as Wget, not download the pages
but just follow the links, and output graphs.



Cheers
Bryan Rasmussen

 

[Marc 'BlackJack' Rintsch]

It should hopefully be as high level as Wget, not download the pages
but just follow the links, and output graphs.

How do you get at the links without downloading the page!?

Ciao,
Marc 'BlackJack' Rintsch

 

[bryan rasmussen]

Hi,

Sorry, was imprecise, I meant not save the downloaded page locally.
There probably isn't one though, so I should build one myself.
Probably just need a good crawler that can be set to dump all links
into dataset that I can analyse with R.

Cheers,
Bryan Rasmussen

 

[George Sakkis]

Hi,

Sorry, was imprecise, I meant not save the downloaded page locally.
There probably isn't one though, so I should build one myself.
Probably just need a good crawler that can be set to dump all links
into dataset that I can analyse with R.

Cheers,
Bryan Rasmussen

Harvestman (http://harvestman.freezope.org/) is your best bet.

George

 

[gene tani]

Hi,

Sorry, was imprecise, I meant not save the downloaded page locally.
There probably isn't one though, so I should build one myself.
Probably just need a good crawler that can be set to dump all links
into dataset that I can analyse with R.

Cheers,
Bryan Rasmussen

There's quite a few already: webchecker, Orchid, mechanize, mygale:

http://codesnipers.com/?q=node/223&&title=Detecting-Dead-Links

http://pxr.openlook.org/pxr/source/Tools/webchecker/

http://sig.levillage.org/?p=599
http://www.robertblum.com/articles/2005/11/21/challenge-map-i-python-web-scraping
http://www.rexx.com/~dkuhlman/quixote_htmlscraping.html