urllib proxy support confusion

[Grant Edwards]

Reading through the doc at http://docs.python.org/lib/module-urllib.html,
there are several paragraphs (including code examples) showing
how you specify what proxies to use when calling urlopen():

Alternatively, the optional proxies argument may be used to
explicitly specify proxies. [...]

The explanation of how to specify proxies is followed by this
paragraph denying that the ability to do so exists:

The urlopen() function does not support explicit proxy
specification. If you need to override environmental proxy
settings, use URLopener, or a subclass such as
FancyURLopener.

That seems a bit baffling. If it urlopen doesn't support
specifying proxies, why are there examples showing how to do
it? If it does support specifying proxies, what is the pragraph
quoted above supposed to mean?

 

[Mark Dickinson]

That seems a bit baffling.  If it urlopen doesn't support
specifying proxies, why are there examples showing how to do
it? If it does support specifying proxies, what is the pragraph
quoted above supposed to mean?

Looks like a definite documentation bug to me, and it's still
there in the development version of the documentation. I think
a bug report should be filed.

A little bit of investigation shows that the documentation changes
were made in svn revisions 26181 and 26182 (

http://svn.python.org/view?view=rev&rev=26181

) as a fix for

http://bugs.python.org/issue523415

but this doesn't seem to help much with figuring out what the
docs intend to say.

Mark

 

[Grant Edwards]

Looks like a definite documentation bug to me, and it's still
there in the development version of the documentation. I
think a bug report should be filed.

A little bit of investigation shows that the documentation
changes were made in svn revisions 26181 and 26182 (

http://svn.python.org/view?view=rev&rev=26181

) as a fix for

http://bugs.python.org/issue523415

but this doesn't seem to help much with figuring out what the
docs intend to say.

Before I submit a bug, I'll give it a try to find out if it
does support explicit specification of proxys or not.

 

[Mark Dickinson]

Before I submit a bug, I'll give it a try to find out if it
does support explicit specification of proxys or not.

Sounds good. If it does, then I think the whole paragraph
(from "The urlopeen() function does not support explicit proxy
..." to "...subclass such as FancyURLopener.") should just be
removed.

Looking at the source, there are some slight oddities: a plain
urlopen() caches the URLopener instance it creates (in a module-level
global) and uses this cached instance in future calls to urlopen
or urlretrieve. If proxies are specified then it doesn't do the
caching. I don't *think* this really affects usage, except that
there are presumably some inefficiencies involved in multiple uses
of urlopen with explicitly specified proxies, and those inefficiencies
can be overcome by using URLopener or FancyURLopener directly instead.

And it doesn't make a lot of sense that urlopen accepts proxies, but
urlretrieve does not. But that's a different issue...

Mark

 

[Mark Dickinson]

Reading through the doc athttp://docs.python.org/lib/module-urllib.html,
there are several paragraphs (including code examples) showing
how you specify what proxies to use when calling urlopen():

See http://bugs.python.org/issue2288