httplib and Proxy

[Rolf Wester]

Hi,

I want to fetch some web-pages via http. I know how to do this without a
proxy server in between but unfortunately we can only access the
internet via a proxy. I would be very appriciative if anybody could tell
me how to do this.

Thanks in advance

Rolf Wester

P.S.: I would not mind to use sockets directly

 

[Chris Gray]

I want to fetch some web-pages via http. I know how to do this without a
proxy server in between but unfortunately we can only access the
internet via a proxy. I would be very appriciative if anybody could tell
me how to do this.

Thanks in advance

Rolf Wester

P.S.: I would not mind to use sockets directly

You can use httplib. The the host and port for the HTTPConnection should
be the proxy server and the URL for the request should include the host
name. For instance:

h1 = httplib.HTTPConnection("my.proxy.com", 80)
h1.request("GET", "http://www.python.org/doc/current/lib/lib.html")

Chris Gray

"Okey dokey.. free my mind. Right, no problem, free my mind, free my mind,
no problem, right... " -The Matrix

 

[Fredrik Lundh]

I want to fetch some web-pages via http. I know how to do this without a
proxy server in between but unfortunately we can only access the
internet via a proxy. I would be very appriciative if anybody could tell
me how to do this.

use urllib, not httplib.

import os, urllib
os.environ["http_proxy"] = "http://proxyserver:3128"
data = urllib.urlopen("http://server/page").read()

you can of course set the http_proxy environment variable before you start
Python. on Windows, you can also use the standard "Internet Options"
dialogue to configure the proxy.

also see:

http://www.python.org/doc/current/lib/module-urllib.html

</F>

 

[Eric Baker]

Hi,

I want to fetch some web-pages via http. I know how to do this without a
proxy server in between but unfortunately we can only access the
internet via a proxy. I would be very appriciative if anybody could tell
me how to do this.

Thanks in advance

Rolf Wester

P.S.: I would not mind to use sockets directly

From the Library Reference:
The urlopen() function works transparently with proxies which do not require
authentication. In a Unix or Windows environment, set the http_proxy,
ftp_proxy or gopher_proxy environment variables to a URL that identifies the
proxy server before starting the Python interpreter. For example (the "%" is
the command prompt):


% http_proxy="http://www.someproxy.com:3128"
% export http_proxy
% python
....

 

[Rolf Wester]

Hi,

I want to fetch some web-pages via http. I know how to do this without a
proxy server in between but unfortunately we can only access the
internet via a proxy. I would be very appriciative if anybody could tell
me how to do this.

Thanks in advance

Rolf Wester

P.S.: I would not mind to use sockets directly

Hi,

thank you all for your help. I tried urllib, httplib and sockets.


With urllib:

f = urllib.urlopen("http://www.python.org/index.html")

I get:

....
invalid proxy for http: 'cache.ilt.fhg.de:81'


httplib works:

conn = httplib.HTTPConnection("cache.ilt.fhg.de", 81)
conn.request("GET", "http://www.python.org/index.html")
r = conn.getresponse()
print r.status, r.reason
print r.msg
while 1:
data = r.read(1024)
if len(data) < 1024: break
print data

and so do sockets:

HOST = 'cache.ilt.fhg.de'
PORT = 81
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.send('GET http://www.python.org/index.html HTTP/1.1\r\nAccept:
text/plain\r\n\r\n')
while 1:
data = s.recv(1024)
print data
if len(data) < 1024: break
s.close()

Thanks again

Rolf Wester

 

[Eric Baker]

With urllib:

f = urllib.urlopen("http://www.python.org/index.html")

I get:

...
invalid proxy for http: 'cache.ilt.fhg.de:81'

I dont have a proxy handy, but perhaps you need to include the "http://...."
in the proxy environement setting.