A Question on URLLIB

[joy99]

Dear Group,
I am trying to construct a web based crawler with Python and for that
I am using the URLLIB module, and by doing
import urllib and then trying with urllib.urlopen("url).
Am I going fine?
If some one can kindly highlight if I am doing any mistake.
Best Regards,
Subhabrata Banerjee.

 

[joy99]

You are doing fine so far

However, urllib is quite low level module. Do you know mechanizehttp://wwwsearch.sourceforge.net/mechanize/? I have used it in my
crawler. If you want to crawl a specific site for known data then
Scrapyhttp://scrapy.org/could be useful too.

David Marek
(e-mail address removed)://davidmarek.cz

Thanks David,I'll surely try to have a check.