Internationalized domain names not working with URLopen

[John Nagle]

I'm trying to open

http://пример.иÑпытание

with

urllib2.urlopen(s1)

in Python 2.7 on Windows 7. This produces a Unicode exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\python27\lib\urllib2.py", line 394, in open
response = self._open(req, data)
File "C:\python27\lib\urllib2.py", line 412, in _open
'_open', req)
File "C:\python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\python27\lib\urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\python27\lib\urllib2.py", line 1168, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "C:\python27\lib\httplib.py", line 955, in request
self._send_request(method, url, body, headers)
File "C:\python27\lib\httplib.py", line 988, in _send_request
self.putheader(hdr, value)
File "C:\python27\lib\httplib.py", line 935, in putheader
hdr = '%s: %s' % (header, '\r\n\t'.join([str(v) for v in values]))
UnicodeEncodeError: 'ascii' codec can't encode characters in position
0-5: ordinal not in range(128)
The HTTP library is trying to put the URL in the header as ASCII. Why
isn't "urllib2" handling that?

What does "urllib2" want? Percent escapes? Punycode?

John Nagle

 

[Andrew Berg]

What does "urllib2" want? Percent escapes? Punycode?

Looks like Punycode is the correct answer:
https://en.wikipedia.org/wiki/Internationalized_domain_name#ToASCII_and_ToUnicode

I haven't tried it, though.

 

[John Nagle]

Looks like Punycode is the correct answer:
https://en.wikipedia.org/wiki/Internationalized_domain_name#ToASCII_and_ToUnicode

I haven't tried it, though.

This is Python bug #9679:

http://bugs.python.org/issue9679

It's been open for years, and the maintainers offer elaborate
excuses for not fixing the problem.

The socket module accepts Unicode domains, as does httplib.
But urllib2, which is a front end to both, is still broken.
It's failing when it constructs the HTTP headers. Domains
in HTTP headers have to be in punycode.

The code in stackoverflow doesn't really work right. Only
the domain part of a URL should be converted to punycode.
Path, port, and query parameters need to be converted to
percent-encoding. (Unclear if urllib2 or httplib does this
already. The documentation doesn't say.)

While HTTP content can be in various character sets, the
headers are currently required to be ASCII only, since the
header has to be processed to determine the character code.
(http://lists.w3.org/Archives/Public/ietf-http-wg/2011OctDec/0155.html)

Here's a workaround, for the domain part only.


#
# idnaurlworkaround -- workaround for Python defect 9679
#
PYTHONDEFECT9679FIXED = False # Python defect #9679 - change when fixed

def idnaurlworkaround(url) :
"""
Convert a URL to a form the currently broken urllib2 will accept.
Converts the domain to "punycode" if necessary.
This is a workaround for Python defect #9679.
"""
if PYTHONDEFECT9679FIXED : # if defect fixed
return(url) # use unmodified URL
url = unicode(url) # force to Unicode
(scheme, accesshost, path, params,
query, fragment) = urlparse.urlparse(url) # parse URL
if scheme == '' and accesshost == '' and path != '' : # bare domain
accesshost = path # use path as access host
path = '' # no path
labels = accesshost.split('.') # split domain into sections ("labels")
labels = [encodings.idna.ToASCII(w) for w in labels]# convert each
label to punycode if necessary
accesshost = '.'.join(labels) # reassemble domain
url = urlparse.urlunparse((scheme, accesshost, path, params, query,
fragment)) # reassemble url
return(url) # return complete URL with punycode domain

John Nagle