urllib(2) and https blues? try pytunnel for python tunnelling

Discussion in 'Python' started by john, Jul 30, 2003.

  1. john

    john Guest

    The python libraries like urllib and httplib do not support ssl
    through a proxy.
    Urllib2 supports http through a proxy or https alone, but not https
    through a proxy.

    A while ago my wife complained to me that perl had trouble w/https. I
    was surprised since perl has a lot of networking libraries. Preferring
    python, I took a look a python and found the same problem. Eventually
    I figured out what python was missing: tunnelling. Though she may
    _coax_ me to port the code to perl, I'm writing it first in my
    favorite language.

    Pytunnel offers a tunnel that can be used for (among other things)
    tunnelling the ssl through a proxy.

    I'd be willing to offer suggestions or myself add the necessary code
    to urllib2. Anyone know how one goes about doing either?

    Your python code does not need to be changed except that the port and
    ip are supplied by the tunnel. Pytunnel uses a custom recvall which is
    like sendall but for recv. Recvall uses non-blocking sockets,timeouts
    and sleeps to attempt to get all of the data. If you have a bad
    network connection you'd want to set the timeout to be high. Recvall
    will probably be slower than a standard recvall.

    The code is rough and needs better error handling, comments and such.
    But since I got it working I decided to post it.

    You can find it at:

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/213238
    Example of usage:

    import pytunnel,httplib

    def tunnel_this(ip,port):
    conn = httplib.HTTPSConnection(ip,port=port)
    conn.putrequest('GET', '/')
    conn.endheaders()
    response = conn.getresponse()
    print response.read()

    tunnel=pytunnel.build(host='login.yahoo.com',proxy_host='h1',proxy_user='u',proxy_pass='p')
    tunnel.run(tunnel_this)
    john, Jul 30, 2003
    #1
    1. Advertising

  2. According to john <>:
    > The python libraries like urllib and httplib do not support ssl
    > through a proxy.
    > Urllib2 supports http through a proxy or https alone, but not https
    > through a proxy.


    M2Crypto's contrib/ contains an isaac.httpslib.py, with the following
    blurb:

    This is Isaac Salzberg's application of Mihai Ibanescu's patch
    (available on SF) that allows HTTPS tunneling through an
    authenticating proxy.

    This one's a double whammy: it works with IIS through the
    authenticating proxy, whereas the one on SF, which uses Python's
    built-in SSL, doesn't.

    This code is not folded into the main distribution because:

    1. Apparently Mihai is still working on it.
    2. Mihai uses Python's built-in SSL. Isaac patched it to use
    M2Crypto.SSL. The stuff is essentially #ifdef'ed code.
    3. I don't have an authenticating proxy nor an IIS server to test
    against, so I can't clean up the code. Volunteers welcome. ;-)

    Thanks Isaac.


    --
    Ng Pheng Siong <>

    http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
    http://www.post1.com/home/ngps -+- Open Source Python Crypto & SSL
    Ng Pheng Siong, Aug 1, 2003
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. john

    pytunnel 0.21 and urllib3.py

    john, Aug 8, 2003, in forum: Python
    Replies:
    0
    Views:
    450
  2. Replies:
    3
    Views:
    668
  3. Jonathan Gardner

    Asynchronous urllib (urllib+asyncore)?

    Jonathan Gardner, Feb 26, 2008, in forum: Python
    Replies:
    1
    Views:
    461
    Terry Jones
    Feb 27, 2008
  4. Chris McDonald
    Replies:
    0
    Views:
    272
    Chris McDonald
    Nov 1, 2010
  5. Marcus
    Replies:
    8
    Views:
    49
    Roy Smith
    Mar 3, 2014
Loading...

Share This Page