LWP and HTTP HEAD

[Martin Kissner]

hello together,

I want to capture all HTTP headers a webserver sends for a website using
LWP. I have checked perldoc LWP and perldoc LWP::Simple but did't find
anything helpful.
Gogle also didn't bring me any further.

I tried this script giving th URL as an argument (i.e.
http://www.google.com)
---
i#!/usr/bin/perl

use warnings;
use strict;
use LWP;

my $url = shift;
my $agent = LWP::UserAgent->new;
my $request = HTTP::Request->new(HEAD => $url);

my $response = $agent->request($request);

print $response->content;
print $response->header("Content-type"),"\n";
---

I thought the content of a HEAD request would be all header but
obviously that is wrong. If I know the specific header (here
Content-type) I get the value of it. But I want to get *all* headers.

The function head($url) of LWP::Simple only returns the following 5
values: "$content_type, $document_length, $modified_time, $expires,
$server".

How can I capture all headers of a HTTP Request using LWP?

Thanks in advance.
Best regards
Martin

 

[J. Gleixner]

How can I capture all headers of a HTTP Request using LWP?

use LWP;

my $url = 'http://blah.blah.com/';
my $agent = LWP::UserAgent->new;
my $request = HTTP::Request->new( GET => $url );
my $response = $agent->request($request);
print $response->headers->as_string;

perldoc HTTP::Headers

 

[Brian Wakem]

hello together,

I want to capture all HTTP headers a webserver sends for a website using
LWP. I have checked perldoc LWP and perldoc LWP::Simple but did't find
anything helpful.
Gogle also didn't bring me any further.

I tried this script giving th URL as an argument (i.e.
http://www.google.com)
---
i#!/usr/bin/perl

use warnings;
use strict;
use LWP;

my $url = shift;
my $agent = LWP::UserAgent->new;
my $request = HTTP::Request->new(HEAD => $url);

my $response = $agent->request($request);

print $response->content;
print $response->header("Content-type"),"\n";
---

I thought the content of a HEAD request would be all header but
obviously that is wrong. If I know the specific header (here
Content-type) I get the value of it. But I want to get *all* headers.

The function head($url) of LWP::Simple only returns the following 5
values: "$content_type, $document_length, $modified_time, $expires,
$server".

How can I capture all headers of a HTTP Request using LWP?

Thanks in advance.
Best regards
Martin

my $ua = LWP::UserAgent->new();
my $res = $ua->head($url);
print $res->as_string;

 

[Charles DeRykus]

hello together,

I want to capture all HTTP headers a webserver sends for a website using
LWP. I have checked perldoc LWP and perldoc LWP::Simple but did't find
anything helpful.
Gogle also didn't bring me any further.

I tried this script giving th URL as an argument (i.e.
http://www.google.com)
---
i#!/usr/bin/perl

use warnings;
use strict;
use LWP;

my $url = shift;
my $agent = LWP::UserAgent->new;
my $request = HTTP::Request->new(HEAD => $url);

my $response = $agent->request($request);

print $response->content;
print $response->header("Content-type"),"\n";
---

I thought the content of a HEAD request would be all header but
obviously that is wrong. If I know the specific header (here
Content-type) I get the value of it. But I want to get *all* headers.

The function head($url) of LWP::Simple only returns the following 5
values: "$content_type, $document_length, $modified_time, $expires,
$server".

How can I capture all headers of a HTTP Request using LWP?

Maybe, there are betters ways but you could easily pry them out with the
response object's as_string method:

print $response->as_string;

#This generated:
# HTTP/1.1 200 OK
# Cache-Control: private
# Connection: close
# Date: Thu, 23 Mar 2006 22:05:32 GMT
# Server: GWS/2.1
# Content-Length: 0
# Content-Type: text/html
# Client-Date: Thu, 23 Mar 2006 22:05:32 GMT
# Client-Peer: 192.48.21.150:31060
#Client-Response-Num: 1
# Set-Cookie: #
#PREF=ID=c342c5866caf9eb2:TM=1143151532:LM=1143151532:S=0eWtpUkdWEoWH
#cXY; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com

 

[Martin Kissner]

J. Gleixner wrote :

use LWP;

my $url = 'http://blah.blah.com/';
my $agent = LWP::UserAgent->new;
my $request = HTTP::Request->new( GET => $url );
my $response = $agent->request($request);
print $response->headers->as_string;

perldoc HTTP::Headers

Thank you. This works (almost) perfectly.

When I compare the results to a telnet session I always get three
additional fileds (the values are from de.yahoo.com):

Client-Date: Thu, 23 Mar 2006 21:55:37 GMT
Client-Peer: 217.12.3.11:80
Client-Response-Num: 1

These are produced by LWP an not part of the server's response.
AFAIK they are not even part of the HTTP spec.
Why are thes returned as "header fields"?

Best regards
Martin

 

[J. Gleixner]

When I compare the results to a telnet session I always get three
additional fileds (the values are from de.yahoo.com):

Client-Date: Thu, 23 Mar 2006 21:55:37 GMT
Client-Peer: 217.12.3.11:80
Client-Response-Num: 1

These are produced by LWP an not part of the server's response.
AFAIK they are not even part of the HTTP spec.
Why are thes returned as "header fields"?

They're set by LPW:rotocol::http.pm, no idea why. Try asking the author.

 

[simoivanov]

perldoc HTTP::Headers

 

[Martin Kissner]

perldoc HTTP::Headers

quite helpful

 

[Martin Kißner]

J. Gleixner wrote :

They're set by LPW:rotocol::http.pm, no idea why. Try asking the author.

I did so and I will let you know the answer.

Best regards
Martin

 

[Martin Kißner]

Martin Kißner wrote :

J. Gleixner wrote :

I did so and I will let you know the answer.

This is the (shortened) answer by Gisle Aas, which I recieved about 20
minutes after I had asked:

- LWP adds them [the Client-... headers] by itself to be helpful. There
- can be other Client-* headers depending on what protocol module
- processed the request, see
- http://search.cpan.org/dist/libwww-perl/lib/LWP.pm#NETWORK_SUPPORT

Thank you, Gisle.

Best regards
Martin