httplib and Proxy

Discussion in 'Python' started by Rolf Wester, Nov 20, 2003.

  1. Rolf Wester

    Rolf Wester Guest

    Hi,

    I want to fetch some web-pages via http. I know how to do this without a
    proxy server in between but unfortunately we can only access the
    internet via a proxy. I would be very appriciative if anybody could tell
    me how to do this.

    Thanks in advance

    Rolf Wester

    P.S.: I would not mind to use sockets directly
     
    Rolf Wester, Nov 20, 2003
    #1
    1. Advertisements

  2. Rolf Wester

    Chris Gray Guest

    You can use httplib. The the host and port for the HTTPConnection should
    be the proxy server and the URL for the request should include the host
    name. For instance:

    h1 = httplib.HTTPConnection("my.proxy.com", 80)
    h1.request("GET", "http://www.python.org/doc/current/lib/lib.html")

    Chris Gray

    "Okey dokey.. free my mind. Right, no problem, free my mind, free my mind,
    no problem, right... " -The Matrix
     
    Chris Gray, Nov 20, 2003
    #2
    1. Advertisements

  3. use urllib, not httplib.
    you can of course set the http_proxy environment variable before you start
    Python. on Windows, you can also use the standard "Internet Options"
    dialogue to configure the proxy.

    also see:

    http://www.python.org/doc/current/lib/module-urllib.html

    </F>
     
    Fredrik Lundh, Nov 20, 2003
    #3
  4. Rolf Wester

    Eric Baker Guest

    From the Library Reference:
    The urlopen() function works transparently with proxies which do not require
    authentication. In a Unix or Windows environment, set the http_proxy,
    ftp_proxy or gopher_proxy environment variables to a URL that identifies the
    proxy server before starting the Python interpreter. For example (the "%" is
    the command prompt):


    % http_proxy="http://www.someproxy.com:3128"
    % export http_proxy
    % python
    ....
     
    Eric Baker, Nov 20, 2003
    #4
  5. Rolf Wester

    Rolf Wester Guest

    Hi,

    thank you all for your help. I tried urllib, httplib and sockets.


    With urllib:

    f = urllib.urlopen("http://www.python.org/index.html")

    I get:

    ....
    invalid proxy for http: 'cache.ilt.fhg.de:81'


    httplib works:

    conn = httplib.HTTPConnection("cache.ilt.fhg.de", 81)
    conn.request("GET", "http://www.python.org/index.html")
    r = conn.getresponse()
    print r.status, r.reason
    print r.msg
    while 1:
    data = r.read(1024)
    if len(data) < 1024: break
    print data

    and so do sockets:

    HOST = 'cache.ilt.fhg.de'
    PORT = 81
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((HOST, PORT))
    s.send('GET http://www.python.org/index.html HTTP/1.1\r\nAccept:
    text/plain\r\n\r\n')
    while 1:
    data = s.recv(1024)
    print data
    if len(data) < 1024: break
    s.close()

    Thanks again

    Rolf Wester
     
    Rolf Wester, Nov 21, 2003
    #5
  6. Rolf Wester

    Eric Baker Guest

    I dont have a proxy handy, but perhaps you need to include the "http://...."
    in the proxy environement setting.
     
    Eric Baker, Nov 21, 2003
    #6
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.