A
Alex
I am trying to harness the power of LWP:
arallel::UserAgent module
(on RedHat Linux 9, module version 2.56) but for some reason cannot
get it to follow redirects. To illustrate the problem I wrote two HTML
files: test01.html and redirect.html (which redirects to test01.html)
as well as a script (straight from the example at LWP:arallel). The
script and its output (including a dump of the entire $entries
variable) are attached below.
The only potential issue that I see is that while
$entries->{'redirect_ok'} is '1',
$entries->{'_permitted'}->{'redirect_ok'} is "undef". I am not sure
what the significance of that is as I could not find any reference to
the _permitted variable in the documentation for the module.
Your help will be greatly appreciated!
Feel free to cc me on the email.
Alex
************************************************
SCRIPT:
#!/usr/bin/perl
use CGI;
use URI;
use LWP:arallel::UserAgent;
use HTTP::Cookies;
use HTTP::Request;
use Data:umper;
# shortcut for demo URLs
my $url1 = "http://localhost/~aturchin/test/parallel/test01.html";
my $url2 = "http://localhost/~aturchin/test/parallel/redirect.html";
my $reqs =
[
HTTP::Request->new('GET', $url1),
HTTP::Request->new('GET', $url1),
HTTP::Request->new('GET', $url2),
HTTP::Request->new('GET', $url2),
];
my $pua = LWP:arallel::UserAgent->new();
$pua->in_order (1); # handle requests in order of registration
$pua->duplicates(0); # ignore duplicates
$pua->timeout (2); # in seconds
$pua->redirect (1); # follow redirects
foreach my $req (@$reqs)
{
print "Registering '".$req->url."'\n";
if ( my $res = $pua->register ($req) )
{
print $res->error_as_HTML;
}
}
my $entries = $pua->wait();
foreach (keys %$entries)
{
my $res = $entries->{$_}->response;
print "Asked for '",$res->request->url, "'.\n";
print "Answer was ", $res->code,": ", $res->message,"\n";
print "Result was:\n", $res->content();
}
print "Dumping all of the entries:\n\n", Dumper($entries);
*******************************************************
OUTPUT (LONG!):
Registering 'http://localhost/~aturchin/test/parallel/test01.html'
Registering 'http://localhost/~aturchin/test/parallel/test01.html'
Registering 'http://localhost/~aturchin/test/parallel/redirect.html'
Registering 'http://localhost/~aturchin/test/parallel/redirect.html'
Asked for 'http://localhost/~aturchin/test/parallel/test01.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>
test page for lwp:arallel
</title>
</head>
<body>
test page for lwp:arallel::useragent
</body>
</html>
Asked for 'http://localhost/~aturchin/test/parallel/redirect.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>Redirect to new page</title>
<meta http-equiv="Refresh" content="10; URL=test01.html">
</head>
<body>
This is the redirect test page.
</body>
</html>
Asked for 'http://localhost/~aturchin/test/parallel/redirect.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>Redirect to new page</title>
<meta http-equiv="Refresh" content="10; URL=test01.html">
</head>
<body>
This is the redirect test page.
</body>
</html>
Asked for 'http://localhost/~aturchin/test/parallel/test01.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>
test page for lwp:arallel
</title>
</head>
<body>
test page for lwp:arallel::useragent
</body>
</html>
Dumping all of the entries:
$VAR1 = {
'HTTP::Request=HASH(0x84dfc98)' => bless( {
'protocol' =>
bless( {
'ua' => {},
'parse_head' => 1,
'scheme' => 'http',
'max_size' => undef
}, 'LWP:arallel:rotocol::http' ),
'redirect_ok' =>
1,
'request' =>
bless( {
'_content' => '',
'_uri' => bless( do{\(my $o =
'http://localhost/~aturchin/test/parallel/test01.html')}, 'URI::http'
),
'_headers' => bless( {
'user-agent' => 'libwww-perl/5.65'
}, 'HTTP::Headers' ),
'_method' => 'GET'
}, 'HTTP::Request' ),
'arg' => undef,
'response' =>
bless( {
'_protocol' => 'HTTP/1.1',
'_content' => '<html>
<head>
<title>
test page for lwp:arallel
</title>
</head>
<body>
test page for lwp:arallel::useragent
</body>
</html>
',
'_rc' => '200',
'_headers' => bless( {
'etag' => '"29c257-83-6447c680"',
'content-type' => 'text/html;
charset=ISO-8859-1',
'connection' => 'close',
'last-modified' => 'Tue, 02 Dec 2003
17:07:22 GMT',
'accept-ranges' => 'bytes',
'date' => 'Tue, 02 Dec 2003 17:21:01
GMT',
'title' => 'test page for lwp:arallel',
'client-peer' => '127.0.0.1:80',
'content-length' => '131',
'server' => 'Apache/2.0.40 (Red Hat
Linux)'
}, 'HTTP::Headers' ),
'_msg' => 'OK',
'_request' => $VAR1->{'HTTP::Request=HASH(0x84dfc98)'}{'request'}
}, 'HTTP::Response' ),
'cmd_socket' =>
undef,
'content_size'
=> 131,
'_permitted' =>
{
'request' => undef,
'redirect_ok' => undef,
'protocol' => undef,
'response' => undef,
'arg' => undef,
'content_size' => undef,
'cmd_socket' => undef,
'size' => undef,
'listen_socket' => undef,
'fullpath' => undef,
'proxy' => undef
},
'size' => 8192,
'listen_socket'
=> undef,
'fullpath' =>
'/~aturchin/test/parallel/test01.html',
'proxy' => undef
},
'LWP:arallel::UserAgent::Entry' ),
'HTTP::Request=HASH(0x84dfcf8)' => bless( {
'protocol' =>
bless( {
'ua' => {},
'parse_head' => 1,
'scheme' => 'http',
'max_size' => undef
}, 'LWP:arallel:rotocol::http' ),
'redirect_ok' =>
1,
'request' =>
bless( {
'_content' => '',
'_uri' => bless( do{\(my $o =
'http://localhost/~aturchin/test/parallel/redirect.html')},
'URI::http' ),
'_headers' => bless( {
'user-agent' => 'libwww-perl/5.65'
}, 'HTTP::Headers' ),
'_method' => 'GET'
}, 'HTTP::Request' ),
'arg' => undef,
'response' =>
bless( {
'_protocol' => 'HTTP/1.1',
'_content' => '<html>
<head>
<title>Redirect to new page</title>
<meta http-equiv="Refresh" content="10; URL=test01.html">
</head>
<body>
This is the redirect test page.
</body>
</html>
',
'_rc' => '200',
'_headers' => bless( {
'etag' => '"29c256-b2-5afb6780"',
'content-type' => 'text/html;
charset=ISO-8859-1',
'connection' => 'close',
'refresh' => '10; URL=test01.html',
'last-modified' => 'Tue, 02 Dec 2003
17:04:46 GMT',
'accept-ranges' => 'bytes',
'date' => 'Tue, 02 Dec 2003 17:21:01
GMT',
'title' => 'Redirect to new page',
'client-peer' => '127.0.0.1:80',
'content-length' => '178',
'server' => 'Apache/2.0.40 (Red Hat
Linux)'
}, 'HTTP::Headers' ),
'_msg' => 'OK',
'_request' => $VAR1->{'HTTP::Request=HASH(0x84dfcf8)'}{'request'}
}, 'HTTP::Response' ),
'cmd_socket' =>
undef,
'content_size'
=> 178,
'_permitted' =>
$VAR1->{'HTTP::Request=HASH(0x84dfc98)'}{'_permitted'},
'size' => 8192,
'listen_socket'
=> undef,
'fullpath' =>
'/~aturchin/test/parallel/redirect.html',
'proxy' => undef
},
'LWP:arallel::UserAgent::Entry' ),
AND SO ON... CAN SEND THE WHOLE THING IF YOU THINK IT'D BE MORE
HELPFUL
arallel::UserAgent module(on RedHat Linux 9, module version 2.56) but for some reason cannot
get it to follow redirects. To illustrate the problem I wrote two HTML
files: test01.html and redirect.html (which redirects to test01.html)
as well as a script (straight from the example at LWP:
arallel). Thescript and its output (including a dump of the entire $entries
variable) are attached below.
The only potential issue that I see is that while
$entries->{'redirect_ok'} is '1',
$entries->{'_permitted'}->{'redirect_ok'} is "undef". I am not sure
what the significance of that is as I could not find any reference to
the _permitted variable in the documentation for the module.
Your help will be greatly appreciated!
Feel free to cc me on the email.
Alex
************************************************
SCRIPT:
#!/usr/bin/perl
use CGI;
use URI;
use LWP:
arallel::UserAgent;use HTTP::Cookies;
use HTTP::Request;
use Data:
umper;# shortcut for demo URLs
my $url1 = "http://localhost/~aturchin/test/parallel/test01.html";
my $url2 = "http://localhost/~aturchin/test/parallel/redirect.html";
my $reqs =
[
HTTP::Request->new('GET', $url1),
HTTP::Request->new('GET', $url1),
HTTP::Request->new('GET', $url2),
HTTP::Request->new('GET', $url2),
];
my $pua = LWP:
arallel::UserAgent->new();$pua->in_order (1); # handle requests in order of registration
$pua->duplicates(0); # ignore duplicates
$pua->timeout (2); # in seconds
$pua->redirect (1); # follow redirects
foreach my $req (@$reqs)
{
print "Registering '".$req->url."'\n";
if ( my $res = $pua->register ($req) )
{
print $res->error_as_HTML;
}
}
my $entries = $pua->wait();
foreach (keys %$entries)
{
my $res = $entries->{$_}->response;
print "Asked for '",$res->request->url, "'.\n";
print "Answer was ", $res->code,": ", $res->message,"\n";
print "Result was:\n", $res->content();
}
print "Dumping all of the entries:\n\n", Dumper($entries);
*******************************************************
OUTPUT (LONG!):
Registering 'http://localhost/~aturchin/test/parallel/test01.html'
Registering 'http://localhost/~aturchin/test/parallel/test01.html'
Registering 'http://localhost/~aturchin/test/parallel/redirect.html'
Registering 'http://localhost/~aturchin/test/parallel/redirect.html'
Asked for 'http://localhost/~aturchin/test/parallel/test01.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>
test page for lwp:
arallel</title>
</head>
<body>
test page for lwp:
arallel::useragent</body>
</html>
Asked for 'http://localhost/~aturchin/test/parallel/redirect.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>Redirect to new page</title>
<meta http-equiv="Refresh" content="10; URL=test01.html">
</head>
<body>
This is the redirect test page.
</body>
</html>
Asked for 'http://localhost/~aturchin/test/parallel/redirect.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>Redirect to new page</title>
<meta http-equiv="Refresh" content="10; URL=test01.html">
</head>
<body>
This is the redirect test page.
</body>
</html>
Asked for 'http://localhost/~aturchin/test/parallel/test01.html'.
Answer was 200: OK
Result was:
<html>
<head>
<title>
test page for lwp:
arallel</title>
</head>
<body>
test page for lwp:
arallel::useragent</body>
</html>
Dumping all of the entries:
$VAR1 = {
'HTTP::Request=HASH(0x84dfc98)' => bless( {
'protocol' =>
bless( {
'ua' => {},
'parse_head' => 1,
'scheme' => 'http',
'max_size' => undef
}, 'LWP:
arallel:rotocol::http' ),'redirect_ok' =>
1,
'request' =>
bless( {
'_content' => '',
'_uri' => bless( do{\(my $o =
'http://localhost/~aturchin/test/parallel/test01.html')}, 'URI::http'
),
'_headers' => bless( {
'user-agent' => 'libwww-perl/5.65'
}, 'HTTP::Headers' ),
'_method' => 'GET'
}, 'HTTP::Request' ),
'arg' => undef,
'response' =>
bless( {
'_protocol' => 'HTTP/1.1',
'_content' => '<html>
<head>
<title>
test page for lwp:
arallel</title>
</head>
<body>
test page for lwp:
arallel::useragent</body>
</html>
',
'_rc' => '200',
'_headers' => bless( {
'etag' => '"29c257-83-6447c680"',
'content-type' => 'text/html;
charset=ISO-8859-1',
'connection' => 'close',
'last-modified' => 'Tue, 02 Dec 2003
17:07:22 GMT',
'accept-ranges' => 'bytes',
'date' => 'Tue, 02 Dec 2003 17:21:01
GMT',
'title' => 'test page for lwp:
arallel','client-peer' => '127.0.0.1:80',
'content-length' => '131',
'server' => 'Apache/2.0.40 (Red Hat
Linux)'
}, 'HTTP::Headers' ),
'_msg' => 'OK',
'_request' => $VAR1->{'HTTP::Request=HASH(0x84dfc98)'}{'request'}
}, 'HTTP::Response' ),
'cmd_socket' =>
undef,
'content_size'
=> 131,
'_permitted' =>
{
'request' => undef,
'redirect_ok' => undef,
'protocol' => undef,
'response' => undef,
'arg' => undef,
'content_size' => undef,
'cmd_socket' => undef,
'size' => undef,
'listen_socket' => undef,
'fullpath' => undef,
'proxy' => undef
},
'size' => 8192,
'listen_socket'
=> undef,
'fullpath' =>
'/~aturchin/test/parallel/test01.html',
'proxy' => undef
},
'LWP:
arallel::UserAgent::Entry' ),'HTTP::Request=HASH(0x84dfcf8)' => bless( {
'protocol' =>
bless( {
'ua' => {},
'parse_head' => 1,
'scheme' => 'http',
'max_size' => undef
}, 'LWP:
arallel:rotocol::http' ),'redirect_ok' =>
1,
'request' =>
bless( {
'_content' => '',
'_uri' => bless( do{\(my $o =
'http://localhost/~aturchin/test/parallel/redirect.html')},
'URI::http' ),
'_headers' => bless( {
'user-agent' => 'libwww-perl/5.65'
}, 'HTTP::Headers' ),
'_method' => 'GET'
}, 'HTTP::Request' ),
'arg' => undef,
'response' =>
bless( {
'_protocol' => 'HTTP/1.1',
'_content' => '<html>
<head>
<title>Redirect to new page</title>
<meta http-equiv="Refresh" content="10; URL=test01.html">
</head>
<body>
This is the redirect test page.
</body>
</html>
',
'_rc' => '200',
'_headers' => bless( {
'etag' => '"29c256-b2-5afb6780"',
'content-type' => 'text/html;
charset=ISO-8859-1',
'connection' => 'close',
'refresh' => '10; URL=test01.html',
'last-modified' => 'Tue, 02 Dec 2003
17:04:46 GMT',
'accept-ranges' => 'bytes',
'date' => 'Tue, 02 Dec 2003 17:21:01
GMT',
'title' => 'Redirect to new page',
'client-peer' => '127.0.0.1:80',
'content-length' => '178',
'server' => 'Apache/2.0.40 (Red Hat
Linux)'
}, 'HTTP::Headers' ),
'_msg' => 'OK',
'_request' => $VAR1->{'HTTP::Request=HASH(0x84dfcf8)'}{'request'}
}, 'HTTP::Response' ),
'cmd_socket' =>
undef,
'content_size'
=> 178,
'_permitted' =>
$VAR1->{'HTTP::Request=HASH(0x84dfc98)'}{'_permitted'},
'size' => 8192,
'listen_socket'
=> undef,
'fullpath' =>
'/~aturchin/test/parallel/redirect.html',
'proxy' => undef
},
'LWP:
arallel::UserAgent::Entry' ),AND SO ON... CAN SEND THE WHOLE THING IF YOU THINK IT'D BE MORE
HELPFUL