LWP and HTTP HEAD

Discussion in 'Perl Misc' started by Martin Kissner, Mar 23, 2006.

  1. hello together,

    I want to capture all HTTP headers a webserver sends for a website using
    LWP. I have checked perldoc LWP and perldoc LWP::Simple but did't find
    anything helpful.
    Gogle also didn't bring me any further.

    I tried this script giving th URL as an argument (i.e.
    http://www.google.com)
    ---
    i#!/usr/bin/perl

    use warnings;
    use strict;
    use LWP;

    my $url = shift;
    my $agent = LWP::UserAgent->new;
    my $request = HTTP::Request->new(HEAD => $url);

    my $response = $agent->request($request);

    print $response->content;
    print $response->header("Content-type"),"\n";
    ---

    I thought the content of a HEAD request would be all header but
    obviously that is wrong. If I know the specific header (here
    Content-type) I get the value of it. But I want to get *all* headers.

    The function head($url) of LWP::Simple only returns the following 5
    values: "$content_type, $document_length, $modified_time, $expires,
    $server".

    How can I capture all headers of a HTTP Request using LWP?

    Thanks in advance.
    Best regards
    Martin

    --
    perl -e '$S=[[73,116,114,115,31,96],[108,109,114,102,99,112],
    [29,77,98,111,105,29],[100,93,95,103,97,110]];
    for(0..3){for$s(0..5){print(chr($S->[$_]->[$s]+$_+1))}}'
    Martin Kissner, Mar 23, 2006
    #1
    1. Advertising

  2. Martin Kissner

    J. Gleixner Guest

    Martin Kissner wrote:

    > How can I capture all headers of a HTTP Request using LWP?


    use LWP;

    my $url = 'http://blah.blah.com/';
    my $agent = LWP::UserAgent->new;
    my $request = HTTP::Request->new( GET => $url );
    my $response = $agent->request($request);
    print $response->headers->as_string;

    perldoc HTTP::Headers
    J. Gleixner, Mar 23, 2006
    #2
    1. Advertising

  3. Martin Kissner

    Brian Wakem Guest

    Martin Kissner wrote:

    > hello together,
    >
    > I want to capture all HTTP headers a webserver sends for a website using
    > LWP. I have checked perldoc LWP and perldoc LWP::Simple but did't find
    > anything helpful.
    > Gogle also didn't bring me any further.
    >
    > I tried this script giving th URL as an argument (i.e.
    > http://www.google.com)
    > ---
    > i#!/usr/bin/perl
    >
    > use warnings;
    > use strict;
    > use LWP;
    >
    > my $url = shift;
    > my $agent = LWP::UserAgent->new;
    > my $request = HTTP::Request->new(HEAD => $url);
    >
    > my $response = $agent->request($request);
    >
    > print $response->content;
    > print $response->header("Content-type"),"\n";
    > ---
    >
    > I thought the content of a HEAD request would be all header but
    > obviously that is wrong. If I know the specific header (here
    > Content-type) I get the value of it. But I want to get *all* headers.
    >
    > The function head($url) of LWP::Simple only returns the following 5
    > values: "$content_type, $document_length, $modified_time, $expires,
    > $server".
    >
    > How can I capture all headers of a HTTP Request using LWP?
    >
    > Thanks in advance.
    > Best regards
    > Martin
    >


    my $ua = LWP::UserAgent->new();
    my $res = $ua->head($url);
    print $res->as_string;


    --
    Brian Wakem
    Email: http://homepage.ntlworld.com/b.wakem/myemail.png
    Brian Wakem, Mar 23, 2006
    #3
  4. Martin Kissner wrote:
    > hello together,
    >
    > I want to capture all HTTP headers a webserver sends for a website using
    > LWP. I have checked perldoc LWP and perldoc LWP::Simple but did't find
    > anything helpful.
    > Gogle also didn't bring me any further.
    >
    > I tried this script giving th URL as an argument (i.e.
    > http://www.google.com)
    > ---
    > i#!/usr/bin/perl
    >
    > use warnings;
    > use strict;
    > use LWP;
    >
    > my $url = shift;
    > my $agent = LWP::UserAgent->new;
    > my $request = HTTP::Request->new(HEAD => $url);
    >
    > my $response = $agent->request($request);
    >
    > print $response->content;
    > print $response->header("Content-type"),"\n";
    > ---
    >
    > I thought the content of a HEAD request would be all header but
    > obviously that is wrong. If I know the specific header (here
    > Content-type) I get the value of it. But I want to get *all* headers.
    >
    > The function head($url) of LWP::Simple only returns the following 5
    > values: "$content_type, $document_length, $modified_time, $expires,
    > $server".
    >
    > How can I capture all headers of a HTTP Request using LWP?
    >


    Maybe, there are betters ways but you could easily pry them out with the
    response object's as_string method:

    print $response->as_string;

    #This generated:
    # HTTP/1.1 200 OK
    # Cache-Control: private
    # Connection: close
    # Date: Thu, 23 Mar 2006 22:05:32 GMT
    # Server: GWS/2.1
    # Content-Length: 0
    # Content-Type: text/html
    # Client-Date: Thu, 23 Mar 2006 22:05:32 GMT
    # Client-Peer: 192.48.21.150:31060
    #Client-Response-Num: 1
    # Set-Cookie: #
    #PREF=ID=c342c5866caf9eb2:TM=1143151532:LM=1143151532:S=0eWtpUkdWEoWH
    #cXY; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com

    --
    Charles DeRykus
    Charles DeRykus, Mar 23, 2006
    #4
  5. J. Gleixner wrote :
    > Martin Kissner wrote:
    >
    >> How can I capture all headers of a HTTP Request using LWP?

    >
    > use LWP;
    >
    > my $url = 'http://blah.blah.com/';
    > my $agent = LWP::UserAgent->new;
    > my $request = HTTP::Request->new( GET => $url );
    > my $response = $agent->request($request);
    > print $response->headers->as_string;
    >
    > perldoc HTTP::Headers


    Thank you. This works (almost) perfectly.

    When I compare the results to a telnet session I always get three
    additional fileds (the values are from de.yahoo.com):

    Client-Date: Thu, 23 Mar 2006 21:55:37 GMT
    Client-Peer: 217.12.3.11:80
    Client-Response-Num: 1

    These are produced by LWP an not part of the server's response.
    AFAIK they are not even part of the HTTP spec.
    Why are thes returned as "header fields"?

    Best regards
    Martin

    --
    perl -e '$S=[[73,116,114,115,31,96],[108,109,114,102,99,112],
    [29,77,98,111,105,29],[100,93,95,103,97,110]];
    for(0..3){for$s(0..5){print(chr($S->[$_]->[$s]+$_+1))}}'
    Martin Kissner, Mar 23, 2006
    #5
  6. Martin Kissner

    J. Gleixner Guest

    Martin Kissner wrote:

    > When I compare the results to a telnet session I always get three
    > additional fileds (the values are from de.yahoo.com):
    >
    > Client-Date: Thu, 23 Mar 2006 21:55:37 GMT
    > Client-Peer: 217.12.3.11:80
    > Client-Response-Num: 1
    >
    > These are produced by LWP an not part of the server's response.
    > AFAIK they are not even part of the HTTP spec.
    > Why are thes returned as "header fields"?


    They're set by LPW::protocol::http.pm, no idea why. Try asking the author.
    J. Gleixner, Mar 24, 2006
    #6
  7. Martin Kissner

    Guest

    perldoc HTTP::Headers
    , Mar 25, 2006
    #7
  8. wrote :
    > perldoc HTTP::Headers


    quite helpful


    --
    perl -e '$S=[[73,116,114,115,31,96],[108,109,114,102,99,112],
    [29,77,98,111,105,29],[100,93,95,103,97,110]];
    for(0..3){for$s(0..5){print(chr($S->[$_]->[$s]+$_+1))}}'
    Martin Kissner, Mar 25, 2006
    #8
  9. J. Gleixner wrote :
    > Martin Kißner wrote:
    >
    >> When I compare the results to a telnet session I always get three
    >> additional fileds (the values are from de.yahoo.com):
    >>
    >> Client-Date: Thu, 23 Mar 2006 21:55:37 GMT
    >> Client-Peer: 217.12.3.11:80
    >> Client-Response-Num: 1
    >>
    >> These are produced by LWP an not part of the server's response.
    >> AFAIK they are not even part of the HTTP spec.
    >> Why are thes returned as "header fields"?

    >
    > They're set by LPW::protocol::http.pm, no idea why. Try asking the author.


    I did so and I will let you know the answer.

    Best regards
    Martin


    --
    perl -e '$S=[[73,116,114,115,31,96],[108,109,114,102,99,112],
    [29,77,98,111,105,29],[100,93,95,103,97,110]];
    for(0..3){for$s(0..5){print(chr($S->[$_]->[$s]+$_+1))}}'
    Martin Kißner, Mar 25, 2006
    #9
  10. Martin Kißner wrote :
    > J. Gleixner wrote :
    >> Martin Kißner wrote:
    >>
    >>> When I compare the results to a telnet session I always get three
    >>> additional fileds (the values are from de.yahoo.com):
    >>>
    >>> Client-Date: Thu, 23 Mar 2006 21:55:37 GMT
    >>> Client-Peer: 217.12.3.11:80
    >>> Client-Response-Num: 1
    >>>
    >>> These are produced by LWP an not part of the server's response.
    >>> AFAIK they are not even part of the HTTP spec.
    >>> Why are thes returned as "header fields"?

    >>
    >> They're set by LPW::protocol::http.pm, no idea why. Try asking the author.

    >
    > I did so and I will let you know the answer.


    This is the (shortened) answer by Gisle Aas, which I recieved about 20
    minutes after I had asked:

    - LWP adds them [the Client-... headers] by itself to be helpful. There
    - can be other Client-* headers depending on what protocol module
    - processed the request, see
    - http://search.cpan.org/dist/libwww-perl/lib/LWP.pm#NETWORK_SUPPORT

    Thank you, Gisle.

    Best regards
    Martin

    --
    perl -e '$S=[[73,116,114,115,31,96],[108,109,114,102,99,112],
    [29,77,98,111,105,29],[100,93,95,103,97,110]];
    for(0..3){for$s(0..5){print(chr($S->[$_]->[$s]+$_+1))}}'
    Martin Kißner, Mar 25, 2006
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Brian W
    Replies:
    10
    Views:
    754
    Brian W
    Jul 2, 2003
  2. showme

    PHP and ASP.NET go HEAD to HEAD

    showme, Jul 8, 2004, in forum: ASP .Net
    Replies:
    13
    Views:
    710
    Sherif ElMetainy
    Jul 10, 2004
  3. Karl
    Replies:
    0
    Views:
    385
  4. Jiho Han

    Write into <HEAD></HEAD> section?

    Jiho Han, Jan 14, 2004, in forum: ASP .Net Building Controls
    Replies:
    6
    Views:
    221
    Sam Fields
    Jan 16, 2004
  5. CronJob
    Replies:
    5
    Views:
    154
    Eric Pozharski
    Mar 20, 2009
Loading...

Share This Page