FTP with urllib2 behind a proxy

Discussion in 'Python' started by O. Koch, Aug 7, 2003.

  1. O. Koch

    O. Koch Guest

    Until now, i know that ftplib doesn't support proxies and that i have
    to use urllib2. But i don't know how to use the urllib2 correct. I
    found some examples, but i don't understand them.

    Is there anyone who can help me?
    O. Koch, Aug 7, 2003
    #1
    1. Advertising

  2. O. Koch

    Anand Pillai Guest

    You need to install a proxyhandler, authhandler,
    ftphandler and httphandler. Then build yourself
    an opener, that opens the doors for you ... :)

    The following does the trick.

    proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
    'https' : 'myhttpsproxy:443',
    'ftp' : 'myftpproxy:21' } )

    opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
    urllib2.HTTPHandler, urllib2.HTTPSHandler,
    urllib2.FTPHandler)

    # install this opener
    urllib2.install_opener(opener)

    # Go ahead, knock knock!

    req=urlli2.Request('ftp://ftp.gnu.org')
    data=urllib2.urlopen(req).read()

    Of course, replace the arbit proxy values I wrote
    with your proxy values. If your proxy need authentication
    , you will need to do a bit more here.

    proxyauth='http://' + username + '@' + password + 'myproxy:myproxport'

    Then your proxy handler becomes ( I am assuming a generic proxy
    for all protocols here!)

    proxy_handler = urllib2.ProxyHandler ( {'http' : proxyauth,
    'https' : proxyauth,
    'ftp' : proxyauth } )

    HTH.

    -Anand


    (John J. Lee) wrote in message news:<>...
    > (O. Koch) writes:
    >
    > > Until now, i know that ftplib doesn't support proxies and that i have
    > > to use urllib2. But i don't know how to use the urllib2 correct. I
    > > found some examples, but i don't understand them.
    > >
    > > Is there anyone who can help me?

    >
    > import urllib2
    > response = urllib2.urlopen("ftp://ftp.example.com/pub/myfile")
    > data = response.read()
    > response.close()
    >
    >
    > Does that do the trick?
    >
    >
    > John
    Anand Pillai, Aug 13, 2003
    #2
    1. Advertising

  3. O. Koch

    John J. Lee Guest

    (Anand Pillai) writes:

    > You need to install a proxyhandler, authhandler,
    > ftphandler and httphandler. Then build yourself
    > an opener, that opens the doors for you ... :)
    >
    > The following does the trick.
    >
    > proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
    > 'https' : 'myhttpsproxy:443',
    > 'ftp' : 'myftpproxy:21' } )
    >
    > opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
    > urllib2.HTTPHandler, urllib2.HTTPSHandler,
    > urllib2.FTPHandler)
    >
    > # install this opener
    > urllib2.install_opener(opener)
    >
    > # Go ahead, knock knock!
    >
    > req=urlli2.Request('ftp://ftp.gnu.org')
    > data=urllib2.urlopen(req).read()


    A couple of things to add: you don't need to add handlers that already
    get added by default by build_opener (FTPHandler and HTTPHandler, for
    example). ProxyHandler is one of these default handlers, so if your
    environment is set up for it (http_proxy, etc. environment variables),
    you don't need to supply a ProxyHandler (of course, if your
    environ. *isn't*, then you do need to supply one, to give it the proxy
    details). You don't need Request objects (unless you want to add
    headers to a Request, or pass Requests around). You don't need to
    install a global opener, unless your code expects it -- it's just a
    convenience (or an inconvenience, sometimes).

    Actually, did the OP say proxy basic auth. was involved? Don't
    recall. I've never needed it for proxies, but there seems to be a
    ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
    use, rather than HTTPBasicAuthHandler (which is for website auth., not
    proxy auth).

    So, after all that, you end up with:

    opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler)
    data = opener.open('ftp://ftp.gnu.org').read()

    (I like to close the response explicitly, though)


    John
    John J. Lee, Aug 13, 2003
    #3
  4. O. Koch

    Anand Pillai Guest

    (John J. Lee) wrote in message news:<>...
    > (Anand Pillai) writes:

    The code obviously caters to the bottom-line, in case
    the programmer does not want to worry about environment
    variables, would like to install the handlers himself
    and his proxy needs authentication.

    In the last case, the HTTP_PROXY env variable does not
    help and you need to install the proxy handler yourself.

    I copied this stuff from my program which needs all these
    and more ( a USER-AGENT header for example ).

    -Anand

    > > You need to install a proxyhandler, authhandler,
    > > ftphandler and httphandler. Then build yourself
    > > an opener, that opens the doors for you ... :)
    > >
    > > The following does the trick.
    > >
    > > proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
    > > 'https' : 'myhttpsproxy:443',
    > > 'ftp' : 'myftpproxy:21' } )
    > >
    > > opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
    > > urllib2.HTTPHandler, urllib2.HTTPSHandler,
    > > urllib2.FTPHandler)
    > >
    > > # install this opener
    > > urllib2.install_opener(opener)
    > >
    > > # Go ahead, knock knock!
    > >
    > > req=urlli2.Request('ftp://ftp.gnu.org')
    > > data=urllib2.urlopen(req).read()

    >
    > A couple of things to add: you don't need to add handlers that already
    > get added by default by build_opener (FTPHandler and HTTPHandler, for
    > example). ProxyHandler is one of these default handlers, so if your
    > environment is set up for it (http_proxy, etc. environment variables),
    > you don't need to supply a ProxyHandler (of course, if your
    > environ. *isn't*, then you do need to supply one, to give it the proxy
    > details). You don't need Request objects (unless you want to add
    > headers to a Request, or pass Requests around). You don't need to
    > install a global opener, unless your code expects it -- it's just a
    > convenience (or an inconvenience, sometimes).
    >
    > Actually, did the OP say proxy basic auth. was involved? Don't
    > recall. I've never needed it for proxies, but there seems to be a
    > ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
    > use, rather than HTTPBasicAuthHandler (which is for website auth., not
    > proxy auth).
    >
    > So, after all that, you end up with:
    >
    > opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler)
    > data = opener.open('ftp://ftp.gnu.org').read()
    >
    > (I like to close the response explicitly, though)
    >
    >
    > John
    Anand Pillai, Aug 14, 2003
    #4
  5. O. Koch

    John J. Lee Guest

    (Anand Pillai) writes:

    > (John J. Lee) wrote in message news:<>...
    > > (Anand Pillai) writes:

    > The code obviously caters to the bottom-line, in case
    > the programmer does not want to worry about environment
    > variables, would like to install the handlers himself
    > and his proxy needs authentication.


    Sure, I understand that. I was just elaborating, no criticism was
    intended!

    It seems people are frightened to use urllib2 (perhaps partly because
    the documentation has lots of sections -- though they're all very
    short), so I like to point out that most of the time it's just a
    matter of:

    import urllib2
    urllib2.urlopen("http://example.com/")


    And most of the rest of the time, the only complications above that
    are passing a Request instead of a URL (for adding headers and passing
    around requests as first-class objects), and building your own opener
    with build_opener (for choosing what features you want and don't want,
    like authentication -- and adding your own features, of course).


    John
    John J. Lee, Aug 14, 2003
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Bob McCormick

    Re: FTP with urllib2 behind a proxy

    Bob McCormick, Feb 25, 2004, in forum: Python
    Replies:
    1
    Views:
    400
    Josiah Carlson
    Feb 25, 2004
  2. Bob McCormick

    Re: FTP with urllib2 behind a proxy

    Bob McCormick, Feb 25, 2004, in forum: Python
    Replies:
    0
    Views:
    382
    Bob McCormick
    Feb 25, 2004
  3. Martin Brodbeck

    FTP, Proxy and urllib2

    Martin Brodbeck, Jun 7, 2004, in forum: Python
    Replies:
    4
    Views:
    1,680
    Martin Brodbeck
    Jun 8, 2004
  4. jakecjacobson

    FTP example going through a FTP Proxy

    jakecjacobson, Jan 7, 2009, in forum: Python
    Replies:
    3
    Views:
    2,048
    jakecjacobson
    Jan 8, 2009
  5. D. Buck
    Replies:
    2
    Views:
    462
    D. Buck
    Jun 29, 2004
Loading...

Share This Page