Web crawler on python

[sonich]

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

 

[Mr.SpOOn]

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

What about BeautifulSoup?

http://www.crummy.com/software/BeautifulSoup/

 

[James Mills]

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/examples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

 

[Support Desk]

-----Original Message-----
From: James Mills [mailto[email protected]]
Sent: Sunday, October 26, 2008 5:26 PM
To: sonich
Cc: (e-mail address removed)
Subject: Re: Web crawler on python

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/e
xamples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

 

[Alex]

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

You should try Orchid http://pypi.python.org/pypi/Orchid/1.1
or you can have a look at my project on launchpad
https://code.launchpad.net/~esaurito/jazz-crawler/experimental.
It's a single site crawler but you can easily modified it.

Bye.

Alex