LWP with proxy problem

Discussion in 'Perl' started by Woogie, Dec 20, 2003.

  1. Woogie

    Woogie Guest

    When running the sample code below without a proxy the GET returns the
    expected data. When run with the $proxy uncommented the GET returns
    the content of the login page for the site being accessed. The site
    in the code is valid for ease of testing. I also am including the LWP
    debug info for each attempt.

    Can anyone explain this behavior and what can I do to correct it?

    Thanks in advance

    Trace without proxy:

    LWP::UserAgent::new: ()
    LWP::UserAgent::request: ()
    LWP::UserAgent::send_request: GET
    https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    LWP::UserAgent::_need_proxy: Not proxied
    LWP::protocol::http::request: ()
    LWP::protocol::collect: read 28 bytes
    LWP::UserAgent::request: Simple response: OK

    Home
    PE


    Trace with proxy:

    LWP::UserAgent::new: ()
    LWP::UserAgent::proxy: https http://148.245.207.85:8080
    LWP::UserAgent::request: ()
    LWP::UserAgent::send_request: GET
    https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    LWP::UserAgent::_need_proxy: Proxied to http://148.245.207.85:8080
    LWP::protocol::http::request: ()
    LWP::protocol::collect: read 236 bytes
    LWP::protocol::collect: read 594 bytes
    LWP::protocol::collect: read 416 bytes
    LWP::protocol::collect: read 450 bytes
    LWP::protocol::collect: read 1017 bytes
    LWP::protocol::collect: read 443 bytes
    LWP::protocol::collect: read 643 bytes
    LWP::UserAgent::request: Simple response: OK

    <html lang="en">

    <!-- Start Head -->
    <head>
    <title>

    Error

    </title>
    <script language="JavaScript">
    ....



    Here is the sample code:


    #!/usr/bin/perl -w

    use LWP::UserAgent;
    use HTTP::Request;
    use HTTP::Response;
    use Crypt::SSLeay;

    LWP::Debug::level('+');

    $url = "https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv";
    #$proxy="http://xxx.xxx.xxx.xxx:8080";

    $ua = LWP::UserAgent->new();

    if (defined $proxy)
    {
    $ENV{HTTPS_PROXY} = $proxy;

    # initialize from environment variables
    $ua->env_proxy;
    }

    $req = HTTP::Request->new(GET => $url);
    $response = $ua->request($req);
    if ($response->is_error())
    {
    printf " %s\n", $response->status_line;
    }

    else
    {
    $content = $response->content();
    print $content;
    }

    exit;
    Woogie, Dec 20, 2003
    #1
    1. Advertising

  2. Woogie

    Roy Johnson Guest

    Note: you'll reach more people at comp.lang.perl.misc. This newsgroup
    does not officially exist, and so is not universally propogated.

    LWP::UserAgent sends the wrong request for HTTPS connections by proxy.
    In the docs for Crypt::SSLeay, you'll find this:
    http://search.cpan.org/~chamas/Crypt-SSLeay-0.51/SSLeay.pm#PROXY_SUPPORT
    LWP::UserAgent has its own methods of proxying which may work for you
    and is likely incompatible with Crypt::SSLeay proxy support. To use
    LWP::UserAgent proxy support, try something like:

    my $ua = new LWP::UserAgent;
    $ua->proxy([qw( https http )], "$proxy_ip:$proxy_port");

    At the time of this writing, libwww v5.6 seems to proxy https requests
    fine with an Apache mod_proxy server. It sends a line like:

    GET https://www.nodeworks.com HTTP/1.1

    to the proxy server, which is not the CONNECT request that some
    proxies would expect, so this may not work with other proxy servers
    than mod_proxy. The CONNECT method is used by Crypt::SSLeay's internal
    proxy support.
    Crypt::SSLeay Proxy Support

    For native Crypt::SSLeay proxy support of https requests, you need to
    set an environment variable HTTPS_PROXY to your proxy server & port,
    as in:

    # PROXY SUPPORT
    $ENV{HTTPS_PROXY} = 'http://proxy_hostname_or_ip:port';
    $ENV{HTTPS_PROXY} = '127.0.0.1:8080';

    Use of the HTTPS_PROXY environment variable in this way is similar to
    LWP::UserAgent->env_proxy() usage, but calling that method will likely
    override or break the Crypt::SSLeay support, so do not mix the two.
    </excerpt>

    This is the code I use, *after* the UserAgent has been created:

    # LWP proxying does not work for HTTPS
    # This must cannot be set at the time $a is created
    $ENV{HTTPS_PROXY} = $proxy;
    Roy Johnson, Dec 22, 2003
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. demonhunter
    Replies:
    3
    Views:
    2,115
  2. Great Deals
    Replies:
    1
    Views:
    108
  3. Woogie

    LWP with proxy problem

    Woogie, Dec 23, 2003, in forum: Perl Misc
    Replies:
    4
    Views:
    186
    Woogie
    Dec 27, 2003
  4. buildmorelines

    sanity checking proxy URL before passing to LWP

    buildmorelines, Nov 6, 2004, in forum: Perl Misc
    Replies:
    2
    Views:
    91
    buildmorelines
    Nov 6, 2004
  5. Mike

    Proxy and LWP::UserAgent

    Mike, Jun 27, 2005, in forum: Perl Misc
    Replies:
    1
    Views:
    126
    RedGrittyBrick
    Jun 30, 2005
Loading...

Share This Page