Error Codes From a URLError

[Fuzzyman]

I've written a CGI proxy that remotely fetches web pages. I use the
urlopen function in CLientCookie which replaces (and ultimately uses)
the urlopen function in urllib2. What I'd like to do is give a more
useful message to the user when the page fetch fails.

My understanding was that a URLError exception had a 'code' attribute
which was the error code.

I trap the error with a simple : (excuse the horrible indentation,
this is from the middle of a program)

try:
req = urllib2.Request(theurl, txdata, txheaders)
if data['name'] and data['pass']:
import base64
base64string = base64.encodestring('%s:%s' %
(data['name'], data['pass']))[:-1]
req.add_header("Authorization", "Basic %s" %
base64string)
u = openfun(req)
......
except IOError, e: # an error in fetching
the page will raise a URLError which is a subclass of IOError
if not cdone:
print "Content-type: text/html" # this is the
header to the server
print # so is this
blank line
if not hasattr(e, 'code'): # If it's not
an http error - raise it as an error
raise

elif e.code == 401: #
authorisation
print authmess % (theurl, data['mod'])
print authend

elif e.code in errorlist: # standard
http errors
the_err = errorlist[e.code]
print err_mes % (the_err[0], the_err[0], the_err[1])

else: # any others
raise

openfun is urlopen function from ClientCookie (if available) or from
urllib2.

What ought to happen in my limied understanding, is that the exception
(e) should have the attribute 'code' if it a URLError - and that value
ought to tell me what type of error it is... (the error message is
then ina dictionary called errorlist which I nicked from
BaseHTTPServer).

*However*, when I use it to try to fetch a non-existent page
(expecting a 404 error) - it falls at the first raise statement.
Evidently the exception *is* an IOError, but doesn't have the code
attribute.


*However* - you see the explicit test for a 401 (authentication)
error. I have code that can handle authentication... if I go to a page
that requires authentication it picks it up fine...... (so the
URLError in that case does have the code attribute).

The exact error traceback is below - and you can see where urllib2
raises the URLError - because of what it sees as a socket problem...
but it obviously isn't returning the error code properly. Any
suggestions ?

URLError Python 2.2.2: /usr/bin/python
Fri Jul 23 11:06:39 2004

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

/home/buildin/public_html/cgi-bin/approx.py
280 print "Content-type: text/html" # this
is the header to the server

281 print # so is
this blank line

282 if not hasattr(e, 'code'): # If
it's not an http error - raise it as an error

283 raise

284

hasattr undefined, e = <urllib2.URLError instance>


/home/buildin/public_html/cgi-bin/ClientCookie/_urllib2_support.py in
urlopen(url=<urllib2.Request instance>, data=None)
969 finally:

970 urlopen_lock.release()

971 return _opener.open(url, data)

972

973 def urlretrieve(url, filename=None, reporthook=None,
data=None):

global _opener = <ClientCookie._urllib2_support.OpenerDirector
instance>, _opener.open = <bound method OpenerDirector.open of
<ClientCookie._urllib2_support.OpenerDirector instance>>, url =
<urllib2.Request instance>, data = None


/home/buildin/public_html/cgi-bin/ClientCookie/_urllib2_support.py in
open(self=<ClientCookie._urllib2_support.OpenerDirector instance>,
fullurl=<urllib2.Request instance>, data=None)
567 req = meth(req)

568

569 response = urllib2.OpenerDirector.open(self, req,
data)

570

571 # post-process response

response undefined, global urllib2 = <module 'urllib2' from
'/usr/lib/python2.2/urllib2.pyc'>, urllib2.OpenerDirector = <class
urllib2.OpenerDirector>, urllib2.OpenerDirector.open = <unbound method
OpenerDirector.open>, self =
<ClientCookie._urllib2_support.OpenerDirector instance>, req =
<ClientCookie._urllib2_support.Request instance>, data = None


/usr/lib/python2.2/urllib2.py in
open(self=<ClientCookie._urllib2_support.OpenerDirector instance>,
fullurl=<ClientCookie._urllib2_support.Request instance>, data=None)
320 type_ = req.get_type()

321 result = self._call_chain(self.handle_open, type_, type_
+ \

322 '_open', req)

323 if result:

324 return result

req = <ClientCookie._urllib2_support.Request instance>


/usr/lib/python2.2/urllib2.py in
_call_chain(self=<ClientCookie._urllib2_support.OpenerDirector
instance>, chain={'file': [<urllib2.FileHandler instance>], 'ftp':
[<urllib2.FTPHandler instance>], 'http':
[<ClientCookie._urllib2_support.HTTPHandler instance>], 'https':
[<ClientCookie._urllib2_support.HTTPSHandler instance>], 'unknown':
[<urllib2.UnknownHandler instance>]}, kind='http',
meth_name='http_open', *args=(<ClientCookie._urllib2_support.Request
instance>,))
299 func = getattr(handler, meth_name)

300

301 result = func(*args)

302 if result is not None:

303 return result

result undefined, func = <bound method HTTPHandler.http_open of
<ClientCookie._urllib2_support.HTTPHandler instance>>, args =
(<ClientCookie._urllib2_support.Request instance>,)


/home/buildin/public_html/cgi-bin/ClientCookie/_urllib2_support.py in
http_open(self=<ClientCookie._urllib2_support.HTTPHandler instance>,
req=<ClientCookie._urllib2_support.Request instance>)
883 class HTTPHandler(AbstractHTTPHandler):

884 def http_open(self, req):

885 return self.do_open(httplib.HTTP, req)

886

887 http_request = AbstractHTTPHandler.do_request_

self = <ClientCookie._urllib2_support.HTTPHandler instance>,
self.do_open = <bound method HTTPHandler.do_open of
<ClientCookie._urllib2_support.HTTPHandler instance>>, global httplib
= <module 'httplib' from '/usr/lib/python2.2/httplib.pyc'>,
httplib.HTTP = <class httplib.HTTP>, req =
<ClientCookie._urllib2_support.Request instance>


/home/buildin/public_html/cgi-bin/ClientCookie/_urllib2_support.py in
do_open(self=<ClientCookie._urllib2_support.HTTPHandler instance>,
http_class=<class httplib.HTTP>,
req=<ClientCookie._urllib2_support.Request instance>)
728 h.endheaders()

729 except socket.error, err:

730 raise URLError(err)

731 if req.has_data():

732 h.send(req.get_data())

global URLError = <class urllib2.URLError>, err = <socket.gaierror
instance>


URLError:
__doc__ = None
__getitem__ = <bound method URLError.__getitem__ of
<urllib2.URLError instance>>
__init__ = <bound method URLError.__init__ of <urllib2.URLError
instance>>
__module__ = 'urllib2'
__str__ = <bound method URLError.__str__ of <urllib2.URLError
instance>>
reason = <socket.gaierror instance>

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Christopher T King]

*However*, when I use it to try to fetch a non-existent page
(expecting a 404 error) - it falls at the first raise statement.
Evidently the exception *is* an IOError, but doesn't have the code
attribute.

For the "non-existent page", are you using a url of the form
'http://fake.server/some/file' or 'http://real.server/fake/file'?
Because if the host is nonexistent (and therefore a connection can't be
opened), then I'd expect urllib2 to raise an IOError. I'd expect an
HTTPError only if the server exists and explicitly returns a 404.

 

[Fuzzyman]

For the "non-existent page", are you using a url of the form
'http://fake.server/some/file' or 'http://real.server/fake/file'?
Because if the host is nonexistent (and therefore a connection can't be
opened), then I'd expect urllib2 to raise an IOError. I'd expect an
HTTPError only if the server exists and explicitly returns a 404.

Right - that would explain it.
But it makes it difficult to trap the errors - an IOError *might* mean
the page is non-existent... *or* it might mean another problem !!

Oh well.

Thanks

Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html