FTP with urllib2 behind a proxy

[O. Koch]

Until now, i know that ftplib doesn't support proxies and that i have
to use urllib2. But i don't know how to use the urllib2 correct. I
found some examples, but i don't understand them.

Is there anyone who can help me?

 

[Anand Pillai]

You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ...

The following does the trick.

proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
'https' : 'myhttpsproxy:443',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
urllib2.HTTPHandler, urllib2.HTTPSHandler,
urllib2.FTPHandler)

# install this opener
urllib2.install_opener(opener)

# Go ahead, knock knock!

req=urlli2.Request('ftp://ftp.gnu.org')
data=urllib2.urlopen(req).read()

Of course, replace the arbit proxy values I wrote
with your proxy values. If your proxy need authentication
, you will need to do a bit more here.

proxyauth='http://' + username + '@' + password + 'myproxy:myproxport'

Then your proxy handler becomes ( I am assuming a generic proxy
for all protocols here!)

proxy_handler = urllib2.ProxyHandler ( {'http' : proxyauth,
'https' : proxyauth,
'ftp' : proxyauth } )

HTH.

-Anand

 

[John J. Lee]

You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ...

The following does the trick.

proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
'https' : 'myhttpsproxy:443',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
urllib2.HTTPHandler, urllib2.HTTPSHandler,
urllib2.FTPHandler)

# install this opener
urllib2.install_opener(opener)

# Go ahead, knock knock!

req=urlli2.Request('ftp://ftp.gnu.org')
data=urllib2.urlopen(req).read()

A couple of things to add: you don't need to add handlers that already
get added by default by build_opener (FTPHandler and HTTPHandler, for
example). ProxyHandler is one of these default handlers, so if your
environment is set up for it (http_proxy, etc. environment variables),
you don't need to supply a ProxyHandler (of course, if your
environ. *isn't*, then you do need to supply one, to give it the proxy
details). You don't need Request objects (unless you want to add
headers to a Request, or pass Requests around). You don't need to
install a global opener, unless your code expects it -- it's just a
convenience (or an inconvenience, sometimes).

Actually, did the OP say proxy basic auth. was involved? Don't
recall. I've never needed it for proxies, but there seems to be a
ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
use, rather than HTTPBasicAuthHandler (which is for website auth., not
proxy auth).

So, after all that, you end up with:

opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler)
data = opener.open('ftp://ftp.gnu.org').read()

(I like to close the response explicitly, though)


John

 

[Anand Pillai]

(e-mail address removed) (Anand Pillai) writes:

The code obviously caters to the bottom-line, in case
the programmer does not want to worry about environment
variables, would like to install the handlers himself
and his proxy needs authentication.

In the last case, the HTTP_PROXY env variable does not
help and you need to install the proxy handler yourself.

I copied this stuff from my program which needs all these
and more ( a USER-AGENT header for example ).

-Anand

 

[John J. Lee]

The code obviously caters to the bottom-line, in case
the programmer does not want to worry about environment
variables, would like to install the handlers himself
and his proxy needs authentication.

Sure, I understand that. I was just elaborating, no criticism was
intended!

It seems people are frightened to use urllib2 (perhaps partly because
the documentation has lots of sections -- though they're all very
short), so I like to point out that most of the time it's just a
matter of:

import urllib2
urllib2.urlopen("http://example.com/")


And most of the rest of the time, the only complications above that
are passing a Request instead of a URL (for adding headers and passing
around requests as first-class objects), and building your own opener
with build_opener (for choosing what features you want and don't want,
like authentication -- and adding your own features, of course).


John