mysteries of urllib/urllib2

[Adrian Smith]

I'm trying to use urllib2 to download a page (I'd rather use urllib,
but I need to change the User-Agent header to look like a browser or
G**gle won't send it to me, the big meanies). The following (pinched
from Dive Into Python) seems to work perfectly in Idle, but falls at
the final hurdle when run as a cgi script - can anyone suggest
anything I may have overlooked?

request = urllib2.Request(some_URL)
request.add_header('User-Agent', 'some_plausible_string')
opener = urllib2.build_opener()
data = opener.open(request).read()

 

[Ben Cartwright]

The following (pinched
from Dive Into Python) seems to work perfectly in Idle, but falls at
the final hurdle when run as a cgi script - can anyone suggest
anything I may have overlooked?

request = urllib2.Request(some_URL)
request.add_header('User-Agent', 'some_plausible_string')
opener = urllib2.build_opener()
data = opener.open(request).read()

Most likely the account that cgi script is running as does not have
permissions to access the net. Check the traceback to be sure. Put
this at the top of your cgi script:

import cgitb; cgitb.enable()

--Ben

 

[Adrian Smith]

Most likely the account that cgi script is running as does not
have permissions to access the net. Check the traceback to be
sure. Put this at the top of your cgi script:

import cgitb; cgitb.enable()

Well, it worked with urllib (resulting in a G**gle 403 your-client-
does-not-have-permission-to-get-urlX page), so I think it must have
some access. Apparently there's a way to change the user-agent string
by subclassing urllib's URLopener class, but that's beyond my comfort
zone at present.

 

[Ben Cartwright]

Did you even try this? Asking for Python help without posting the
traceback is like phoning your mechanic and saying, "My car is making
a generic rattling noise, can you tell me what the problem is without
looking under the hood?"

Apparently there's a way to change the user-agent string
by subclassing urllib's URLopener class, but that's beyond my comfort
zone at present.

Untested:

import urllib
url = 'http://groups.google.com/group/Google-AJAX-Search-API/
browse_thread/thread/a0eb87ad13b11762'
opener = urllib.FancyURLopener()
opener.addheaders = [('User-Agent', 'Fauxzilla 4.0')]
data = opener.open(url).read()

Hope that helps,
--Ben

 

[John Nagle]

I'm trying to use urllib2 to download a page (I'd rather use urllib,
but I need to change the User-Agent header to look like a browser or
G**gle won't send it to me, the big meanies). The following (pinched
from Dive Into Python) seems to work perfectly in Idle, but falls at
the final hurdle when run as a cgi script - can anyone suggest
anything I may have overlooked?

request = urllib2.Request(some_URL)
request.add_header('User-Agent', 'some_plausible_string')
opener = urllib2.build_opener()
data = opener.open(request).read()

I doubt that's the problem here, but don't use a USER-AGENT string
that ends in "m" without a preceding "m" when the USER-AGENT
string is the last element of the header. Coyote Point load balancers
will drop the packet.

(Coyote Point uses regular expressions to parse HTTP headers, and
I think somebody wrote "\m" where they meant "\n".)

John Nagle

 

[O.R.Senthil Kumaran]

some access. Apparently there's a way to change the user-agent string
by subclassing urllib's URLopener class, but that's beyond my comfort
zone at present.

Read the urllib2 how-to located at ActiveState Documentation pages.
That gives the concise snippets as how you will set the USER-AGENT string.
<snip>
import urllib
import urllib2

url = 'http://www.someserver.com/cgi-bin/register.cgi'
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
headers = { 'User-Agent' : user_agent }

data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
</snip>

 

[Adrian Smith]

Did you even try this? Asking for Python help without posting the
traceback is like phoning your mechanic and saying, "My car is
making a generic rattling noise, can you tell me what the problem
is without looking under the hood?"

Sorry, I thought as the cgi did appear to have web access it wasn't
applicable, and it's amazing what some mechanics can infer from engine
noise. cgitb certainly does send back an impressive amount of
information, I'll be sure to use it in future.

Apparently there's a way to change the user-agent string
by subclassing urllib's URLopener class, but that's beyond my
comfort zone at present.

Untested:

import urllib
url = 'http://groups.google.com/group/Google-AJAX-Search-API/
browse_thread/thread/a0eb87ad13b11762'
opener = urllib.FancyURLopener()
opener.addheaders = [('User-Agent', 'Fauxzilla 4.0')]
data = opener.open(url).read()

That works a treat, thanks!