LWP with proxy problem

Discussion in 'Perl Misc' started by Woogie, Dec 23, 2003.

  1. Woogie

    Woogie Guest

    When running the sample code below without a proxy the GET returns the
    expected data. When run with the $proxy uncommented the GET returns
    the content of the login page for the site being accessed. The site
    in the code is valid for ease of testing. I also am including the LWP
    debug info for each attempt.

    Can anyone explain this behavior and what can I do to correct it?

    Thanks in advance

    Trace without proxy:

    LWP::UserAgent::new: ()
    LWP::UserAgent::request: ()
    LWP::UserAgent::send_request: GET
    https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    LWP::UserAgent::_need_proxy: Not proxied
    LWP::protocol::http::request: ()
    LWP::protocol::collect: read 28 bytes
    LWP::UserAgent::request: Simple response: OK

    Home
    PE


    Trace with proxy:

    LWP::UserAgent::new: ()
    LWP::UserAgent::proxy: https http://148.245.207.85:8080
    LWP::UserAgent::request: ()
    LWP::UserAgent::send_request: GET
    https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    LWP::UserAgent::_need_proxy: Proxied to http://148.245.207.85:8080
    LWP::protocol::http::request: ()
    LWP::protocol::collect: read 236 bytes
    LWP::protocol::collect: read 594 bytes
    LWP::protocol::collect: read 416 bytes
    LWP::protocol::collect: read 450 bytes
    LWP::protocol::collect: read 1017 bytes
    LWP::protocol::collect: read 443 bytes
    LWP::protocol::collect: read 643 bytes
    LWP::UserAgent::request: Simple response: OK

    <html lang="en">

    <!-- Start Head -->
    <head>
    <title>

    Error

    </title>
    <script language="JavaScript">
    ....



    Here is the sample code:


    #!/usr/bin/perl -w

    use LWP::UserAgent;
    use HTTP::Request;
    use HTTP::Response;
    use Crypt::SSLeay;

    LWP::Debug::level('+');

    $url = "https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv";
    #$proxy="http://xxx.xxx.xxx.xxx:8080";

    $ua = LWP::UserAgent->new();

    if (defined $proxy)
    {
    $ENV{HTTPS_PROXY} = $proxy;

    # initialize from environment variables
    $ua->env_proxy;
    }

    $req = HTTP::Request->new(GET => $url);
    $response = $ua->request($req);
    if ($response->is_error())
    {
    printf " %s\n", $response->status_line;
    }

    else
    {
    $content = $response->content();
    print $content;
    }

    exit;
     
    Woogie, Dec 23, 2003
    #1
    1. Advertising

  2. Woogie

    Sara Guest

    (Woogie) wrote in message news:<>...
    > When running the sample code below without a proxy the GET returns the
    > expected data. When run with the $proxy uncommented the GET returns
    > the content of the login page for the site being accessed. The site
    > in the code is valid for ease of testing. I also am including the LWP
    > debug info for each attempt.
    >
    > Can anyone explain this behavior and what can I do to correct it?
    >
    > Thanks in advance
    >
    > Trace without proxy:
    >
    > LWP::UserAgent::new: ()
    > LWP::UserAgent::request: ()
    > LWP::UserAgent::send_request: GET
    > https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    > LWP::UserAgent::_need_proxy: Not proxied
    > LWP::protocol::http::request: ()
    > LWP::protocol::collect: read 28 bytes
    > LWP::UserAgent::request: Simple response: OK
    >
    > Home
    > PE
    >
    >
    > Trace with proxy:
    >
    > LWP::UserAgent::new: ()
    > LWP::UserAgent::proxy: https http://148.245.207.85:8080
    > LWP::UserAgent::request: ()
    > LWP::UserAgent::send_request: GET
    > https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    > LWP::UserAgent::_need_proxy: Proxied to http://148.245.207.85:8080
    > LWP::protocol::http::request: ()
    > LWP::protocol::collect: read 236 bytes
    > LWP::protocol::collect: read 594 bytes
    > LWP::protocol::collect: read 416 bytes
    > LWP::protocol::collect: read 450 bytes
    > LWP::protocol::collect: read 1017 bytes
    > LWP::protocol::collect: read 443 bytes
    > LWP::protocol::collect: read 643 bytes
    > LWP::UserAgent::request: Simple response: OK
    >
    > <html lang="en">
    >
    > <!-- Start Head -->
    > <head>
    > <title>
    >
    > Error
    >
    > </title>
    > <script language="JavaScript">
    > ...
    >
    >
    >
    > Here is the sample code:
    >
    >
    > #!/usr/bin/perl -w
    >
    > use LWP::UserAgent;
    > use HTTP::Request;
    > use HTTP::Response;
    > use Crypt::SSLeay;
    >
    > LWP::Debug::level('+');
    >
    > $url = "https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv";
    > #$proxy="http://xxx.xxx.xxx.xxx:8080";
    >
    > $ua = LWP::UserAgent->new();
    >
    > if (defined $proxy)
    > {
    > $ENV{HTTPS_PROXY} = $proxy;
    >
    > # initialize from environment variables
    > $ua->env_proxy;
    > }
    >
    > $req = HTTP::Request->new(GET => $url);
    > $response = $ua->request($req);
    > if ($response->is_error())
    > {
    > printf " %s\n", $response->status_line;
    > }
    >
    > else
    > {
    > $content = $response->content();
    > print $content;
    > }
    >
    > exit;


    might want to try on one of the cgi groups or even comp.perl.modules ?
     
    Sara, Dec 24, 2003
    #2
    1. Advertising

  3. (Sara) wrote in message news:<>...
    > (Woogie) wrote in message news:<>...
    > > When running the sample code below without a proxy the GET returns the
    > > expected data. When run with the $proxy uncommented the GET returns
    > > the content of the login page for the site being accessed. The site
    > > in the code is valid for ease of testing. I also am including the LWP
    > > debug info for each attempt.
    > >
    > > Can anyone explain this behavior and what can I do to correct it?
    > >
    > > Thanks in advance
    > >


    -snip-
    >
    > might want to try on one of the cgi groups or even comp.perl.modules ?


    Why? This has nothing (necessarily) to do with CGI. CGI is for web
    clients and web servers talking to each other. And although LWP is
    actually a web client, the example code does not appear related to any
    CGI...

    Kevin
     
    Kevin Collins, Dec 24, 2003
    #3
  4. (Woogie) wrote in message news:<>...
    > When running the sample code below without a proxy the GET returns the
    > expected data. When run with the $proxy uncommented the GET returns
    > the content of the login page for the site being accessed. The site
    > in the code is valid for ease of testing. I also am including the LWP
    > debug info for each attempt.
    >
    > Can anyone explain this behavior and what can I do to correct it?
    >
    > Thanks in advance
    >
    > Trace without proxy:
    >
    > LWP::UserAgent::new: ()
    > LWP::UserAgent::request: ()
    > LWP::UserAgent::send_request: GET
    > https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    > LWP::UserAgent::_need_proxy: Not proxied
    > LWP::protocol::http::request: ()
    > LWP::protocol::collect: read 28 bytes
    > LWP::UserAgent::request: Simple response: OK
    >
    > Home
    > PE
    >
    >
    > Trace with proxy:
    >
    > LWP::UserAgent::new: ()
    > LWP::UserAgent::proxy: https http://148.245.207.85:8080
    > LWP::UserAgent::request: ()
    > LWP::UserAgent::send_request: GET
    > https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    > LWP::UserAgent::_need_proxy: Proxied to http://148.245.207.85:8080
    > LWP::protocol::http::request: ()
    > LWP::protocol::collect: read 236 bytes
    > LWP::protocol::collect: read 594 bytes
    > LWP::protocol::collect: read 416 bytes
    > LWP::protocol::collect: read 450 bytes
    > LWP::protocol::collect: read 1017 bytes
    > LWP::protocol::collect: read 443 bytes
    > LWP::protocol::collect: read 643 bytes
    > LWP::UserAgent::request: Simple response: OK
    >
    > <html lang="en">
    >
    > <!-- Start Head -->
    > <head>
    > <title>
    >
    > Error
    >
    > </title>
    > <script language="JavaScript">
    > ...
    >
    >
    >
    > Here is the sample code:
    >
    >
    > #!/usr/bin/perl -w
    >
    > use LWP::UserAgent;
    > use HTTP::Request;
    > use HTTP::Response;
    > use Crypt::SSLeay;
    >
    > LWP::Debug::level('+');
    >
    > $url = "https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv";
    > #$proxy="http://xxx.xxx.xxx.xxx:8080";
    >
    > $ua = LWP::UserAgent->new();
    >
    > if (defined $proxy)
    > {
    > $ENV{HTTPS_PROXY} = $proxy;
    >
    > # initialize from environment variables
    > $ua->env_proxy;
    > }
    >
    > $req = HTTP::Request->new(GET => $url);
    > $response = $ua->request($req);
    > if ($response->is_error())
    > {
    > printf " %s\n", $response->status_line;
    > }
    >
    > else
    > {
    > $content = $response->content();
    > print $content;
    > }
    >
    > exit;


    Your code has the definition of $proxy commented out, so I am not sure
    your example is relevant to your output.

    BUT, if it were uncommented, you are not including the ID and password
    in the proxy URL (as you are in the "real" URL), so it makes some
    sense that are being prompted for it.

    Additionally, why do you want to redirect your request through a proxy
    when you can get to it directly?

    Kevin
     
    Kevin Collins, Dec 24, 2003
    #4
  5. Woogie

    Woogie Guest

    (Kevin Collins) wrote in message news:<>...
    > (Woogie) wrote in message news:<>...
    > > When running the sample code below without a proxy the GET returns the
    > > expected data. When run with the $proxy uncommented the GET returns
    > > the content of the login page for the site being accessed. The site
    > > in the code is valid for ease of testing. I also am including the LWP
    > > debug info for each attempt.
    > >
    > > Can anyone explain this behavior and what can I do to correct it?
    > >
    > > Thanks in advance
    > >
    > > Trace without proxy:
    > >
    > > LWP::UserAgent::new: ()
    > > LWP::UserAgent::request: ()
    > > LWP::UserAgent::send_request: GET
    > > https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    > > LWP::UserAgent::_need_proxy: Not proxied
    > > LWP::protocol::http::request: ()
    > > LWP::protocol::collect: read 28 bytes
    > > LWP::UserAgent::request: Simple response: OK
    > >
    > > Home
    > > PE
    > >
    > >
    > > Trace with proxy:
    > >
    > > LWP::UserAgent::new: ()
    > > LWP::UserAgent::proxy: https http://148.245.207.85:8080
    > > LWP::UserAgent::request: ()
    > > LWP::UserAgent::send_request: GET
    > > https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
    > > LWP::UserAgent::_need_proxy: Proxied to http://148.245.207.85:8080
    > > LWP::protocol::http::request: ()
    > > LWP::protocol::collect: read 236 bytes
    > > LWP::protocol::collect: read 594 bytes
    > > LWP::protocol::collect: read 416 bytes
    > > LWP::protocol::collect: read 450 bytes
    > > LWP::protocol::collect: read 1017 bytes
    > > LWP::protocol::collect: read 443 bytes
    > > LWP::protocol::collect: read 643 bytes
    > > LWP::UserAgent::request: Simple response: OK
    > >
    > > <html lang="en">
    > >
    > > <!-- Start Head -->
    > > <head>
    > > <title>
    > >
    > > Error
    > >
    > > </title>
    > > <script language="JavaScript">
    > > ...
    > >
    > >
    > >
    > > Here is the sample code:
    > >
    > >
    > > #!/usr/bin/perl -w
    > >
    > > use LWP::UserAgent;
    > > use HTTP::Request;
    > > use HTTP::Response;
    > > use Crypt::SSLeay;
    > >
    > > LWP::Debug::level('+');
    > >
    > > $url = "https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv";
    > > #$proxy="http://xxx.xxx.xxx.xxx:8080";
    > >
    > > $ua = LWP::UserAgent->new();
    > >
    > > if (defined $proxy)
    > > {
    > > $ENV{HTTPS_PROXY} = $proxy;
    > >
    > > # initialize from environment variables
    > > $ua->env_proxy;
    > > }
    > >
    > > $req = HTTP::Request->new(GET => $url);
    > > $response = $ua->request($req);
    > > if ($response->is_error())
    > > {
    > > printf " %s\n", $response->status_line;
    > > }
    > >
    > > else
    > > {
    > > $content = $response->content();
    > > print $content;
    > > }
    > >
    > > exit;

    >
    > Your code has the definition of $proxy commented out, so I am not sure
    > your example is relevant to your output.
    >
    > BUT, if it were uncommented, you are not including the ID and password
    > in the proxy URL (as you are in the "real" URL), so it makes some
    > sense that are being prompted for it.
    >
    > Additionally, why do you want to redirect your request through a proxy
    > when you can get to it directly?
    >
    > Kevin



    As stated in the comments at the beginning I provided output with and
    without the $proxy set. The example trace outputs were derived from
    two different environments. The first where a proxy was not required
    and the second where one was.

    As far as the ID and password are concerned, the code was intended to
    support multiple environments, the proxy I was behind for the example
    did not require ID or password.

    While it is true that the request can be accessed in an open
    environment directly, there are times when the code will be run in an
    environment that has a proxy to the web. I've tried to write a
    relatively generic solution.

    Kent
     
    Woogie, Dec 27, 2003
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Woogie

    LWP with proxy problem

    Woogie, Dec 20, 2003, in forum: Perl
    Replies:
    1
    Views:
    1,041
    Roy Johnson
    Dec 22, 2003
  2. demonhunter
    Replies:
    3
    Views:
    2,164
  3. Great Deals
    Replies:
    1
    Views:
    132
  4. buildmorelines

    sanity checking proxy URL before passing to LWP

    buildmorelines, Nov 6, 2004, in forum: Perl Misc
    Replies:
    2
    Views:
    102
    buildmorelines
    Nov 6, 2004
  5. Mike

    Proxy and LWP::UserAgent

    Mike, Jun 27, 2005, in forum: Perl Misc
    Replies:
    1
    Views:
    145
    RedGrittyBrick
    Jun 30, 2005
Loading...

Share This Page