Trying to make a spider using mechanize

[tedpottel]

Hi,

I can read the home page using the mechanize lib. Is there a way to
load in web pages using filename.html instad of servername/
filename.html. Lots of time the links just have the file name. I'm
trying to read in the links name and then vsit those pages.

here is the sample code I am ussing.


import ClientForm
import mechanize


#get home page
request = mechanize.Request("http://www.activetechconsulting.com")
response = mechanize.urlopen(request)
print response.read()

#sub page (this does note work)
request = mechanize.Request("service.html")
response = mechanize.urlopen(request)
print response.read-Ted

 

[James Mills]

Hi,

Perhaps you might want to
try out using a sample spider
I wrote and base your code of
this ?

See: http://hg.shortcircuit.net.au/index.wsgi/pymills/file/b9936ae2525c/examples/spider.py

cheers
James