urllib2 : https and proxy

Discussion in 'Python' started by cp.finances.gouv@gmail.com, Jun 28, 2007.

  1. Guest

    Hello all,

    I'm facing a strange behavior of urllib2 trying to access gmail
    account behind a proxy (Squid).

    The following works perfectly :

    wget --save-cookies cookies --keep-session-cookies --post-
    data="Email=&Passwd=cpfinances" --no-
    check-certificate -O - https://www.google.com/accounts/ServiceLoginBoxAuth


    While the following urllib2 code fail with an error 501 :

    In [18]: import urllib2

    In [19]: r=urllib2.urlopen("https://www.google.com/accounts/
    ServiceLoginBoxAuth",

    "Email=&Passwd=cpfinances")
    ---------------------------------------------------------------------------
    urllib2.HTTPError Traceback (most
    recent call last)

    /home/maric/<ipython console>

    /home/maric/urllib2.py in urlopen(url, data)
    128 if _opener is None:
    129 _opener = build_opener()
    --> 130 return _opener.open(url, data)

    ....

    /home/maric/urllib2.py in http_error_default(self, req, fp, code, msg,
    hdrs)
    478 class HTTPDefaultErrorHandler(BaseHandler):
    479 def http_error_default(self, req, fp, code, msg, hdrs):
    --> 480 raise HTTPError(req.get_full_url(), code, msg, hdrs,
    fp)
    481
    482 class HTTPRedirectHandler(BaseHandler):

    HTTPError: HTTP Error 501: Not Implemented


    I understood it's a response from the proxy about incompatible
    protocols and queries but this seem rather strange in this case. Could
    there be a bug in the ProxyHandler class ? Any idea about what could
    cause this error on the proxy ?
     
    , Jun 28, 2007
    #1
    1. Advertising

  2. > Hello all,
    >
    > I'm facing a strange behavior of urllib2 trying to access gmail
    > account behind a proxy (Squid).
    >


    First off, your password wont work. See if you can get a new one. :)
    Then to setup proxy suppor for urllib2, do the steps:



    proxy_url = 'http://' + PROXY_USER + ':' + PROXY_PASSWORD + \
    '@' + PROXY_IP + ':' + PROXY_PORT
    proxy_support = urllib2.ProxyHandler({'http': proxy_url})
    opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
    urllib2.install_opener(opener)

    Substitute PROXY_* with actual values and then do urllib2.urlopen(site) which will go through the proxy.

    --
    O.R.Senthil Kumaran
    http://uthcode.sarovar.org
     
    O.R.Senthil Kumaran, Jun 28, 2007
    #2
    1. Advertising

  3. * Maric Michaud <> [2007-06-28 23:38:39]:

    > Hmmm, i've tried something similar but I get the same result.
    >
    > For the code example I gave, I've put correctly the http_proxy and
    > https_proxy environment variables, the default ProxyHandler get them right
    > as this is the proxy itself which returns the error. I checked this with
    > wireshark, the proxy said that there is incompatibilities in some request
    > methods, like POST, with some protocols, Gopher is the example it gave.
    > That doesn't make much sense to me as the wget command I tried is intended
    > to do exactly the same thing (POST over https and retrieve some cookies from
    > the response), and go through with no problems.
    >


    Maric, in the ActiveState urllib2 HOWTO which seems to apply for Python 2.5, I
    came across a point which mentions urllib2 does not support PROXY for HTTPS.
    Your case seems the same, I would suggest you to look up the Python Tracker
    for any already open issues, or if not kindly log one yourself.
    I shall if I can help further, working on urllib2 is a project am doing as
    part of Summer of Code 2007.

    Thanks,

    --
    O.R.Senthil Kumaran
    http://uthcode.sarovar.org
     
    O.R.Senthil Kumaran, Jul 2, 2007
    #3
  4. Guest

    On 2 juil, 05:04, "O.R.Senthil Kumaran"
    <> wrote:
    > * Maric Michaud <> [2007-06-28 23:38:39]:
    >
    > > Hmmm, i've tried something similar but I get the same result.

    >
    > > For the code example I gave, I've put correctly the http_proxy and
    > > https_proxy environment variables, the default ProxyHandler get them right
    > > as this is the proxy itself which returns the error. I checked this with
    > > wireshark, the proxy said that there is incompatibilities in some request
    > > methods, like POST, with some protocols, Gopher is the example it gave.
    > > That doesn't make much sense to me as the wget command I tried is intended
    > > to do exactly the same thing (POST over https and retrieve some cookies from
    > > the response), and go through with no problems.

    >
    > Maric, in the ActiveState urllib2 HOWTO which seems to apply for Python 2.5, I
    > came across a point which mentions urllib2 does not support PROXY for HTTPS.


    Yes, I realize this too, I think this should be considered as a bug in
    the documentation.

    > Your case seems the same, I would suggest you to look up the Python Tracker
    > for any already open issues, or if not kindly log one yourself.



    Didn't find one when i made a quick search, but i would suggest to
    submit a documentation bug.

    BTW, I find this recipe on python cookbook,
    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195,
    works fine for me (with the two additionals comments in my case). We
    could add a pointer to this in the doc before fixing urllib2.

    What would you suggest ?



    > --
    > O.R.Senthil Kumaranhttp://uthcode.sarovar.org
     
    , Jul 2, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. news.easynet.be

    UrlLib2 Proxy and Https

    news.easynet.be, Dec 10, 2004, in forum: Python
    Replies:
    4
    Views:
    9,684
    Jacobo belbo
    Dec 13, 2004
  2. Fuzzyman

    urllib2, https, and proxies

    Fuzzyman, Jun 29, 2005, in forum: Python
    Replies:
    0
    Views:
    424
    Fuzzyman
    Jun 29, 2005
  3. Josef Cihal
    Replies:
    0
    Views:
    881
    Josef Cihal
    Sep 5, 2005
  4. Astan Chee

    urllib2, proxies and https

    Astan Chee, Aug 18, 2006, in forum: Python
    Replies:
    1
    Views:
    425
    John J. Lee
    Aug 20, 2006
  5. Barry
    Replies:
    0
    Views:
    323
    Barry
    Sep 20, 2009
Loading...

Share This Page