Python equivalent of LWP and HTTP in Perl

Discussion in 'Python' started by Dekaritae, Oct 28, 2004.

  1. Dekaritae

    Dekaritae Guest

    I have a script that I've written in Perl that retrieves files generated
    from a template. It works decently enough, but I'd like to rewrite it in
    Python (Perl was just a detour; it was originally Sed).

    Was wondering what the closest thing to what I'm using now in Perl
    (LWP::UserAgent and HTTP::Request:Common) is in Python. The main bit of
    my code is fairly simple.

    my($ua) = LWP::UserAgent->new;
    my $req = GET "$uri";
    $req->header(Referer => "$referer");
    $ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
    $response = $ua->request($req);
    $respcode = $response->code;
    Dekaritae, Oct 28, 2004
    #1
    1. Advertising

  2. Dekaritae wrote:
    > I have a script that I've written in Perl that retrieves files generated
    > from a template. It works decently enough, but I'd like to rewrite it in
    > Python (Perl was just a detour; it was originally Sed).


    > my($ua) = LWP::UserAgent->new;
    > my $req = GET "$uri";
    > $req->header(Referer => "$referer");
    > $ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
    > $response = $ua->request($req);
    > $respcode = $response->code;



    import urllib
    proxies = {'http': 'http://localhost:8080/'}
    response = urllib.urlopen("some_url", proxies=proxies).read()

    --Irmen
    Irmen de Jong, Oct 28, 2004
    #2
    1. Advertising

  3. Dekaritae

    Erik Johnson Guest

    "Irmen de Jong" wrote

    > > my($ua) = LWP::UserAgent->new;
    > > my $req = GET "$uri";
    > > $req->header(Referer => "$referer");
    > > $ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
    > > $response = $ua->request($req);
    > > $respcode = $response->code;

    >
    >
    > import urllib
    > proxies = {'http': 'http://localhost:8080/'}
    > response = urllib.urlopen("some_url", proxies=proxies).read()


    .... which is short and clean, but ignores the Referer header.
    I have worked with httplib some, but not urllib. Maybe urllib can do this,
    but may I suggest checking out httplib:
    http://docs.python.org/lib/httplib-examples.html

    -ej
    Erik Johnson, Oct 29, 2004
    #3
  4. "Erik Johnson" <> writes:

    > "Irmen de Jong" wrote
    >
    >> > my($ua) = LWP::UserAgent->new;
    >> > my $req = GET "$uri";
    >> > $req->header(Referer => "$referer");
    >> > $ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
    >> > $response = $ua->request($req);
    >> > $respcode = $response->code;

    >>
    >>
    >> import urllib
    >> proxies = {'http': 'http://localhost:8080/'}
    >> response = urllib.urlopen("some_url", proxies=proxies).read()

    >
    > ... which is short and clean, but ignores the Referer header.
    > I have worked with httplib some, but not urllib. Maybe urllib can do this,
    > but may I suggest checking out httplib:
    > http://docs.python.org/lib/httplib-examples.html


    Try urllib2:

    import urllib2

    proxy_support = urllib2.ProxyHandler({"http" : "http://localhost:8080/"})
    opener = urllib2.build_opener(proxy_support)
    urllib2.install_opener(opener)

    req = urllib2.Request("some_url")
    req.add_header("Referer", referer)
    response = urllib2.urlopen(req)


    There's fancier handlers too, if you need digest authentication, for
    example. If you have the environment variable 'http_proxy' set to your
    proxy, you don't need to do the install_opener jazz.

    --
    |>|\/|<
    /--------------------------------------------------------------------------\
    |David M. Cooke
    |cookedm(at)physics(dot)mcmaster(dot)ca
    David M. Cooke, Oct 29, 2004
    #4
  5. Dekaritae <> wrote in message news:<vAbgd.24$>...
    > I have a script that I've written in Perl that retrieves files generated
    > from a template. It works decently enough, but I'd like to rewrite it in
    > Python (Perl was just a detour; it was originally Sed).
    >
    > Was wondering what the closest thing to what I'm using now in Perl
    > (LWP::UserAgent and HTTP::Request:Common) is in Python. The main bit of
    > my code is fairly simple.
    >
    > my($ua) = LWP::UserAgent->new;
    > my $req = GET "$uri";
    > $req->header(Referer => "$referer");
    > $ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
    > $response = $ua->request($req);
    > $respcode = $response->code;


    urllib2 is definitely the way forward for adding proxy support and
    request headers. Very easy. If you want cookie handling you'll need to
    look at ClientCookie which does similar things to the perl library you
    mention.

    Regards,

    Fuzzy
    http://www.voidspace.org.uk/atlantibots/pythonutils.html
    Michael Foord, Oct 29, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Donald Gordon

    UTF-8, LWP and http-equiv meta tags

    Donald Gordon, Feb 25, 2004, in forum: Perl
    Replies:
    0
    Views:
    575
    Donald Gordon
    Feb 25, 2004
  2. Vinay Gupta
    Replies:
    2
    Views:
    1,104
    Michael Schlenker
    Jul 27, 2004
  3. Replies:
    13
    Views:
    2,736
    Arne Vajhøj
    Mar 18, 2008
  4. Martin Kissner

    LWP and HTTP HEAD

    Martin Kissner, Mar 23, 2006, in forum: Perl Misc
    Replies:
    9
    Views:
    154
    Martin Kißner
    Mar 25, 2006
  5. Replies:
    2
    Views:
    413
    Martijn Lievaart
    Aug 2, 2008
Loading...

Share This Page