Writing a Web Robot in Python

[mudgen]

I'm working on writing a web robot (for searching web pages) in Python
so I've been browsing around the web looking for data about building
web robots.

I'm surprised to find that I didn't find much data. Seems like robots
would be kind of a common thing to build. I know there is this: "The
Web Robots Pages" (http://www.robotstxt.org/wc/robots.html) but that
doesn't have a lot of data and seems pretty out of date.

I checked amazon.com for books on building web robots and I didn't find
very much. Seems kind of weird that I don't find much data on this
subject. Anybody know some good places to go to get information on
building a web robots?

 

[Rene Pijlman]

(e-mail address removed):

Anybody know some good places to go to get information on
building a web robots?

Some pointers:
http://safari.oreilly.com/0596005776
http://www.python.org/doc/2.4.2/lib/module-HTMLParser.html
http://www.crummy.com/software/BeautifulSoup/
http://www.python.org/doc/2.4.2/lib/module-robotparser.html
http://cheeseshop.python.org/pypi/spider.py/0.5

 

[WEINHANDL Herbert]

I'm working on writing a web robot (for searching web pages) in Python
so I've been browsing around the web looking for data about building
web robots.

I'm surprised to find that I didn't find much data. Seems like robots
would be kind of a common thing to build. I know there is this: "The
Web Robots Pages" (http://www.robotstxt.org/wc/robots.html) but that
doesn't have a lot of data and seems pretty out of date.

I checked amazon.com for books on building web robots and I didn't find
very much. Seems kind of weird that I don't find much data on this
subject. Anybody know some good places to go to get information on
building a web robots?

maybe one of the following packages will fullfill your needs

http://www.idyll.org/~t/www-tools/twill/

http://wwwsearch.sourceforge.net/mechanize/

http://python.org/pypi/mechanoid

happy pythoning

herbert

 

[George Sakkis]

You'd probably get more results by searching for "web crawlers" or "web
spiders". You may start from here:

http://harvestman.freezope.org/faq.html

HTH,
George