Re: LWP::Simple header information problems

[cp]

I have apache running on 2 linux boxes, one running Redhat 7.3 and the other
Mandrake 9.1. I have a simple Perl script that uses "head $url" from
LWP::Simple to access the header of a web page. It runs fine on the
Mandrake machine but returns 'false' on the Redhat machine with no other
errors. They both have the LWP::Simple module installed but are using
different versions of perl (if that matters).

I might. But you haven't provided any code that illustrates the
problem, so it's hard to tell.

If I run another perl script that uses "get $url", they both return the
webpage. So why can't the Mandrake machine access the header information?

Dunno. Show some code?

Thanks for any insight you could give me.

Help us to help you. Please be more specific. Instead of "it runs
fines..." and "it returns false" it is more helpful to provides some
specifics. What did you expect it to return? What did it return
instead? Were there messages in the error log? Can you run it from the
command line? Is the output different when you do?

Shooting in the dark, perhpas this bit from perldoc LWP::Simple
provides insight:

CAVEAT
Note that if you are using both LWP::Simple and the very
popular CGI.pm module, you may be importing a `head' func-
tion from each module, producing a warning like "Prototype
mismatch: sub main::head ($) vs none". Get around this
problem by just not importing LWP::Simple's `head' func-
tion, like so:

use LWP::Simple qw(!head);
use CGI qwstandard); # then only CGI.pm defines a
head()

Then if you do need LWP::Simple's `head' function, you can
just call it as `LWP::Simple::head($url)'.

SEE ALSO
the LWP manpage, the lwpcook manpage, the LWP::UserAgent
manpage, the HTTP::Status manpage, the lwp-request man-
page, the lwp-mirror manpage

 

[Mitchua]

I might. But you haven't provided any code that illustrates the
problem, so it's hard to tell.

I wrote a quick test script for you:

 

[Mitchua]

I might. But you haven't provided any code that illustrates the
problem, so it's hard to tell.

I wrote a test program to demonstrate the problem. Here's the code:
#!/usr/bin/perl -w

use LWP::Simple;

print "Content-type: text/html\n\n";

my @headinfo = head($ARGV[0]);

if (@headinfo) {

print @headinfo . "Got header information\n";

print "Modified date: " . localtime($headinfo[2]) . "\n";

} else {

print @headinfo . "Problem with head\n";

}

my $content = get($ARGV[0]);

if ($content) {

print "Got content too!\n";

}


When I run it on the Mandrake machine with
http://mandrakemachine_address/cgi-bin/getheadtest.cgi?http://www.netkee.com
I get a webpage: "5Got header information Modified date: Sun May 18 11:01:19
2003 Got content too! "

When I run it on the Redhat machine with
http://redhatmachine_address/cgi-bin/getheadtest.cgi?http://www.netkee.com I
get a webpage: "0Problem with head Got content too! "

So like I said, both machine are getting the page, but the redhat machine is
getting nothing from it's 'head' call for some reason.

Thanks for your help,
Mitchua

 

[cp]

When I run it on the Mandrake machine with
http://mandrakemachine_address/cgi-bin/getheadtest.cgi?http://www.netkee.com
I get a webpage: "5Got header information Modified date: Sun May 18 11:01:19
2003 Got content too! "

When I run it on the Redhat machine with
http://redhatmachine_address/cgi-bin/getheadtest.cgi?http://www.netkee.com I
get a webpage: "0Problem with head Got content too! "

So like I said, both machine are getting the page, but the redhat machine is
getting nothing from it's 'head' call for some reason.

okay, my last thought on the subject, before I move onto something that
pays the rent. perldoc LWP::Simple gives the head method as a way to
see that the file exists, and mentions that, if you want more control
of the headers, you need to use LWP::UserAgent. Example are provided in
the docs, and in perldoc lwpcook. One pertains, which I've modified to
be:

#!/usr/bin/perl
use strict;
use warnings;

use CGI qw(header param);

my $url = param('url') || 'http://www.yahoo.com/';

use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
$ua->agent("Mozilla/7.0");
$ua->proxy('http', 'http://10.1.0.1:8080/' );

my $req = HTTP::Request->new( HEAD => $url );
$req->header('Accept' => '*/*');

# send request
my $res = $ua->request($req);

# check the outcome
if ( $res->is_success ) {
print header('text/plain'),
$res->headers_as_string, "\n";
}
else {
print header('text/plain'), "Error: " . $res->status_line . "\n";
}

Outputs:

Cache-Control: private
Date: Wed, 25 Jun 2003 18:23:07 GMT
Age: 0
Content-Length: 31427
Content-Type: text/html
Client-Date: Wed, 25 Jun 2003 18:23:08 GMT
Client-Peer: 10.1.0.1:8080
Client-Response-Num: 1
P3p: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR
ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi
PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV"
Proxy-Connection: close

Of course, then I realized that what I have a chaching Firewall, and
what I have here is the page that was retrieved and cahced. So that is
something else to investigate.