urllib2 for HTTPS/SSL

[Kylotan]

The documentation on this module doesn't seem very clear to me...
there's an 'HTTPSHandler' object documented, but it just lists the
"https_open" function without giving an example of its use. And the
urllib2 examples page shows a totally different way to create an HTTPS
connection (just by making a Request object). There is a lot of talk
about deriving new classes, but is that necessary if I just want to
make HTTPS GET requests?

I have no idea how (or if) to use the 'HTTPSHandler' object, or what
the correct way for me to request an SSL connection is. Does anybody
have any hints? And additionally, is there any chance of the official
documentation on this useful-looking module being improved?

K.

 

[Jeremy Hylton]

The documentation on this module doesn't seem very clear to me...
there's an 'HTTPSHandler' object documented, but it just lists the
"https_open" function without giving an example of its use. And the
urllib2 examples page shows a totally different way to create an HTTPS
connection (just by making a Request object). There is a lot of talk
about deriving new classes, but is that necessary if I just want to
make HTTPS GET requests?

I have no idea how (or if) to use the 'HTTPSHandler' object, or what
the correct way for me to request an SSL connection is. Does anybody
have any hints? And additionally, is there any chance of the official
documentation on this useful-looking module being improved?

Unless I misunderstand your intent, you don't need to use the Handler
object directly. HTTPS support is provided automatically, so long as
your local Python has SSL support. (It should.)

Here's an example:

f = urllib2.urlopen("https://sf.net/")
buf = f.read()
f.close()

The module is included in the standard library docs. They are brief,
but do provide some examples:

http://www.python.org/dev/doc/devel/lib/urllib2-examples.html

Jeremy

 

[Will Stuyvesant]

[Kylotan]

........................Is there any chance of the official
documentation on this useful-looking module being improved?

It is one of the biggest problems with Python that not enough people
like to write good documentation about how to use the Python Standard
Library. I think the best documentation for you is the book "Python
Standard Library" (PSL) by Fredrik Lundh with its nice examples (could
use even more!); but some bonehead at O'Reilly made the decision to
publish "Python in a Nutshell" and "Python Cookbook" instead of the
2nd edition of PSL. But these have bad example code and the English
feels bad, not as bad as my written English perhaps but not a pleasure
to read either.

Many Python modules have a class based design and class based
frameworks need a detailed description and especially examples how to
use those classes. That will also force the designers to give a
*motivation* for the class-based design. Answer questions like "Why
do I want users of this module to subclass my Application class?
Would it not be easier for them to supply just a list of functions
with a clear description of input and output?". To my opinion there
are just too much class based designs, like for example that Twisted
thing: too much use of OO and inheritance makes it hard to understand
and often leaves the user with the question "But why...". It would
take a *LOT* of documentation to make up for the design decisions
there.

I am not against OO, it has its virtues; I am against overusing OO.
And especially against class-based designs with poor documentation
lacking motivation.

 

[Kylotan]

Unless I misunderstand your intent, you don't need to use the Handler
object directly. HTTPS support is provided automatically, so long as
your local Python has SSL support. (It should.)

Ok, so what you're saying is that the stuff documented in:
"Python Library Reference - 11.5.14 HTTPSHandler Objects"
is purely an implementational detail? In other words, they're
specifying examples of Handler classes in case you wanted to see how
to write one yourself, but to use the urllib2 library normally you'd
never need to really know this?

I'm guessing the OpenerDirector object tries the URL with each
BaseHandler-derived object to see which one should handle it.

Thanks,

K.

 

[Kylotan]

[Kylotan]
........................Is there any chance of the official
documentation on this useful-looking module being improved?

It is one of the biggest problems with Python that not enough people
like to write good documentation about how to use the Python Standard
Library.

It's a shame about the documentation, but then this stuff is generally
provided for free and so I think we're all grateful that it exists at
all, documented or not. Maybe it would be beneficial for the
official online documentation to allow users to add annotations in
much the same way that the PHP and SDL docs do? Then not only people
can get more use out of the docs, but the official maintainers of each
module can easily see about integrating any useful errata or examples
into their source documentation.

K.

 

[Jeremy Hylton]

Ok, so what you're saying is that the stuff documented in:
"Python Library Reference - 11.5.14 HTTPSHandler Objects"
is purely an implementational detail? In other words, they're
specifying examples of Handler classes in case you wanted to see how
to write one yourself, but to use the urllib2 library normally you'd
never need to really know this?

urllib2 provides a framework for writing custom handlers. If you want
to create your own handlers or an opener with a custom set of handlers,
this documentation may be helpful.

I'm guessing the OpenerDirector object tries the URL with each
BaseHandler-derived object to see which one should handle it.

That's about right. Most of the handlers are registered for particular
protocols. So the opener knows to use the HTTPSHandler for https://
requests.

Jeremy