LWP with proxy problem

Woogie · Dec 20, 2003

When running the sample code below without a proxy the GET returns the
expected data. When run with the $proxy uncommented the GET returns
the content of the login page for the site being accessed. The site
in the code is valid for ease of testing. I also am including the LWP
debug info for each attempt.

Can anyone explain this behavior and what can I do to correct it?

Thanks in advance

Trace without proxy:

LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
LWP::UserAgent::_need_proxy: Not proxied
LWP:

rotocol::http::request: ()
LWP:

rotocol::collect: read 28 bytes
LWP::UserAgent::request: Simple response: OK

Home
PE

Trace with proxy:

LWP::UserAgent::new: ()
LWP::UserAgent:

roxy: https http://148.245.207.85:8080
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET
https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv
LWP::UserAgent::_need_proxy: Proxied to http://148.245.207.85:8080
LWP:

rotocol::http::request: ()
LWP:

rotocol::collect: read 236 bytes
LWP:

rotocol::collect: read 594 bytes
LWP:

rotocol::collect: read 416 bytes
LWP:

rotocol::collect: read 450 bytes
LWP:

rotocol::collect: read 1017 bytes
LWP:

rotocol::collect: read 443 bytes
LWP:

rotocol::collect: read 643 bytes
LWP::UserAgent::request: Simple response: OK

<html lang="en">


<head>
<title>

Error

</title>
<script language="JavaScript">
....

Here is the sample code:

#!/usr/bin/perl -w

use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use Crypt::SSLeay;

LWP:

ebug::level('+');

$url = "https://squid.servebeer.com/getservices.do?user=Guest&password=JustLooking&format=csv";
#$proxy="http://xxx.xxx.xxx.xxx:8080";

$ua = LWP::UserAgent->new();

if (defined $proxy)
{
$ENV{HTTPS_PROXY} = $proxy;

# initialize from environment variables
$ua->env_proxy;
}

$req = HTTP::Request->new(GET => $url);
$response = $ua->request($req);
if ($response->is_error())
{
printf " %s\n", $response->status_line;
}

else
{
$content = $response->content();
print $content;
}

exit;

Roy Johnson · Dec 22, 2003

Note: you'll reach more people at comp.lang.perl.misc. This newsgroup
does not officially exist, and so is not universally propogated.

LWP::UserAgent sends the wrong request for HTTPS connections by proxy.
In the docs for Crypt::SSLeay, you'll find this:
http://search.cpan.org/~chamas/Crypt-SSLeay-0.51/SSLeay.pm#PROXY_SUPPORT
LWP::UserAgent has its own methods of proxying which may work for you
and is likely incompatible with Crypt::SSLeay proxy support. To use
LWP::UserAgent proxy support, try something like:

my $ua = new LWP::UserAgent;
$ua->proxy([qw( https http )], "$proxy_ip:$proxy_port");

At the time of this writing, libwww v5.6 seems to proxy https requests
fine with an Apache mod_proxy server. It sends a line like:

GET https://www.nodeworks.com HTTP/1.1

to the proxy server, which is not the CONNECT request that some
proxies would expect, so this may not work with other proxy servers
than mod_proxy. The CONNECT method is used by Crypt::SSLeay's internal
proxy support.
Crypt::SSLeay Proxy Support

For native Crypt::SSLeay proxy support of https requests, you need to
set an environment variable HTTPS_PROXY to your proxy server & port,
as in:

# PROXY SUPPORT
$ENV{HTTPS_PROXY} = 'http://proxy_hostname_or_ip:port';
$ENV{HTTPS_PROXY} = '127.0.0.1:8080';

Use of the HTTPS_PROXY environment variable in this way is similar to
LWP::UserAgent->env_proxy() usage, but calling that method will likely
override or break the Crypt::SSLeay support, so do not mix the two.
</excerpt>

This is the code I use, *after* the UserAgent has been created:

# LWP proxying does not work for HTTPS
# This must cannot be set at the time $a is created
$ENV{HTTPS_PROXY} = $proxy;

https request failing	2	Sep 18, 2012
NTLM and LWP::UserAgent	4	Sep 12, 2006
LWP gives 302 Found after update?	5	Aug 16, 2005
reading LWP in chunks	6	Oct 18, 2010
trying to use a proxy	1	Aug 9, 2007
LWP::Simple get() problem	0	Sep 29, 2003
LWP::Simple get() refined problem	6	Sep 29, 2003
LWP and getting secured documents	2	May 3, 2005

LWP with proxy problem

Woogie

Roy Johnson

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads