Python equivalent of LWP and HTTP in Perl

[Dekaritae]

I have a script that I've written in Perl that retrieves files generated
from a template. It works decently enough, but I'd like to rewrite it in
Python (Perl was just a detour; it was originally Sed).

Was wondering what the closest thing to what I'm using now in Perl
(LWP::UserAgent and HTTP::Request:Common) is in Python. The main bit of
my code is fairly simple.

my($ua) = LWP::UserAgent->new;
my $req = GET "$uri";
$req->header(Referer => "$referer");
$ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
$response = $ua->request($req);
$respcode = $response->code;

 

[Irmen de Jong]

I have a script that I've written in Perl that retrieves files generated
from a template. It works decently enough, but I'd like to rewrite it in
Python (Perl was just a detour; it was originally Sed).

my($ua) = LWP::UserAgent->new;
my $req = GET "$uri";
$req->header(Referer => "$referer");
$ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
$response = $ua->request($req);
$respcode = $response->code;

import urllib
proxies = {'http': 'http://localhost:8080/'}
response = urllib.urlopen("some_url", proxies=proxies).read()

--Irmen

 

[Erik Johnson]

import urllib
proxies = {'http': 'http://localhost:8080/'}
response = urllib.urlopen("some_url", proxies=proxies).read()

.... which is short and clean, but ignores the Referer header.
I have worked with httplib some, but not urllib. Maybe urllib can do this,
but may I suggest checking out httplib:
http://docs.python.org/lib/httplib-examples.html

-ej

 

[David M. Cooke]

... which is short and clean, but ignores the Referer header.
I have worked with httplib some, but not urllib. Maybe urllib can do this,
but may I suggest checking out httplib:
http://docs.python.org/lib/httplib-examples.html

Try urllib2:

import urllib2

proxy_support = urllib2.ProxyHandler({"http" : "http://localhost:8080/"})
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)

req = urllib2.Request("some_url")
req.add_header("Referer", referer)
response = urllib2.urlopen(req)


There's fancier handlers too, if you need digest authentication, for
example. If you have the environment variable 'http_proxy' set to your
proxy, you don't need to do the install_opener jazz.

 

[Michael Foord]

I have a script that I've written in Perl that retrieves files generated
from a template. It works decently enough, but I'd like to rewrite it in
Python (Perl was just a detour; it was originally Sed).

Was wondering what the closest thing to what I'm using now in Perl
(LWP::UserAgent and HTTP::Request:Common) is in Python. The main bit of
my code is fairly simple.

my($ua) = LWP::UserAgent->new;
my $req = GET "$uri";
$req->header(Referer => "$referer");
$ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
$response = $ua->request($req);
$respcode = $response->code;

urllib2 is definitely the way forward for adding proxy support and
request headers. Very easy. If you want cookie handling you'll need to
look at ClientCookie which does similar things to the perl library you
mention.

Regards,

Fuzzy
http://www.voidspace.org.uk/atlantibots/pythonutils.html