Web-crawling

[John Bradbury]

I am trying to develop a special putpose crawler using htmllib & urllib.
How do you tell the server application that you are a modern browser and can
handle frames?

Thanks,

john Bradbury

 

[Rene Pijlman]

John Bradbury:

I am trying to develop a special putpose crawler using htmllib & urllib.
How do you tell the server application that you are a modern browser and can
handle frames?

I don't know of any "I can handle frames" header and I don't see why the
server would care, but you could mimic the User-agent header sent by a
modern browser.

 

[John Bradbury]

I don't know what is causing the problem, but the site I am accessing is
sending out forms for a browser that has a low resolution and does not
support frames. Excuse my ignorance, but where do you set up the User-agent
header you suggested.

Many thanks for your prompt reply.

John Bradbury

 

[Rene Pijlman]

John Bradbury:

where do you set up the User-agent header you suggested.

Its an HTTP header:
http://www.ietf.org/rfc/rfc2616.txt

See also:
http://www.google.com/search?q=urllib+python+"user-agent"

which leads you to... tada... the documentation!
http://www.python.org/doc/current/lib/module-urllib.html

 

[John J. Lee]

John Bradbury:

I am trying to develop a special putpose crawler using htmllib & urllib.
How do you tell the server application that you are a modern browser
and can handle frames?

[...]
server would care, but you could mimic the User-agent header sent by a

[...]
I don't know what is causing the problem, but the site I am accessing is
sending out forms for a browser that has a low resolution and does not
support frames. Excuse my ignorance, but where do you set up the
User-agent header you suggested.

For urllib2 (well, almost):

http://wwwsearch.sourceforge.net/ClientCookie/doc.html#headers


John