Urllib2 / add_password method

M

mrstephengross

I'm working on learning how to use urllib2 to use a proxy server. I've
looked through the postings on this group, and it's been helpful. I
have not, however, found complete documentation on the add_password()
functions. Here's what I've got so far:

#========================================
import urllib2
url = 'http://some.server/form.php'

proxy_url = 'http://202.62.252.3:8080' # A publicly available anonymous
proxy server

proxy_handler = urllib2.ProxyHandler( {'http': proxy_url } )

proxy_auth_handler = urllib2.HTTPBasicAuthHandler()
proxy_auth_handler.add_password('realm', 'host', 'username',
'password')

opener = urllib2.build_opener(proxy_handler, proxy_auth_handler)

urllib2.install_opener(opener)

try:
handler = urllib2.urlopen(url)
except urllib2.URLError:
print "whoops!"
else:
print handler.read()
#=======================================

It works, but I don't really know what I'm doing with the
proxy_auth_handler part. Specifying username and password make sense,
but I haven't found documentation on what 'realm' and 'host' are for.
Shouldn't username & password be sufficient?

Thanks,
--Steve ([email protected])
 
J

John J. Lee

mrstephengross said:
I'm working on learning how to use urllib2 to use a proxy server. I've
looked through the postings on this group, and it's been helpful. I
have not, however, found complete documentation on the add_password()
functions. Here's what I've got so far:

Don't use a password manager with proxy auth in 2.4, it doesn't work:

http://python.org/sf/1470846

[...]
It works, but I don't really know what I'm doing with the
proxy_auth_handler part. Specifying username and password make sense,
but I haven't found documentation on what 'realm' and 'host' are for.
Shouldn't username & password be sufficient?

Both proxy and auth handling in urllib2 are buggy in 2.4, but passing
the username and password in the "userinfo" component of the URL works
(didn't read your code, hope nothing important in there):

import urllib2

proxy_handler = urllib2.ProxyHandler({
"http": "http://john:[email protected]:3128"})
opener = urllib2.build_opener(proxy_handler)

response = opener.open(url)


However, ISTR specifying a port number as I do above (the ":3128" bit)
is not supported in 2.4 (it does in current SVN, so will in 2.5).


Hmm, I notice just now that urllib2's digest auth support breaks
urllib2's handler scheme. Must file another bug...


John
 
J

John J Lee

mrstephengross said:
I'm working on learning how to use urllib2 to use a proxy server. I've
looked through the postings on this group, and it's been helpful. I
have not, however, found complete documentation on the add_password()
functions. Here's what I've got so far:

Don't use a password manager with proxy auth in 2.4, it doesn't work:

http://python.org/sf/1470846

[...]
It works, but I don't really know what I'm doing with the
proxy_auth_handler part. Specifying username and password make sense,
but I haven't found documentation on what 'realm' and 'host' are for.
Shouldn't username & password be sufficient?

Oops, didn't read your question.

I'm surprised if it worked at all given that bug (maybe you're using an
older Python, and I forget all the details of the bug).

Anyway, to answer your question: yes, username & password *should* be
sufficient, but the password manager classes in urllib2 at present aren't
very friendly like that. You can use HTTPPasswordMgrWithDefaultRealm so
that at least you can forget about the realm part. (The "realm" is what
you see in the title bar of the pop-up that web browsers display. See RFC
2617 for more details.)

I'm adding friendlier proxy/auth support to package mechanize ATM (and
finding/fixing bugs as I go), so maybe some of that will find its way into
Python 2.6.


John
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,906
Latest member
SkinfixSkintag

Latest Threads

Top