Obtaining verbose info for http transfers.

Discussion in 'Perl Misc' started by Sisyphus, Aug 17, 2005.

  1. Sisyphus

    Sisyphus Guest

    Hi,

    One of the nice things about Net::FTP is that if you run it with debugging
    switched on, you get a report of the actual communication that's taking
    place between the local box and the remote ftp server.

    Is there a module that provides the same sort of report in relation to http
    downloads ?

    I have a satellite broadband connection, and when surfing the web I find
    it's about as slow as my old 28kbps dial-up connection was. (There's no
    problem when it comes to http or ftp downloads of large files - it's just
    that in general surfing it's fairly slow.) I suspect this has something to
    do with the latency involved in the passing of communications via satellite
    between my PC and the remote web server - but I don't know how many times
    messages are passed back and forth, and it would be informative if there was
    a perl module that would dispay this info (if only in terms of a report
    similar to that offered by Net::FTP).

    It's not uncommon for my browser to take 15-20 seconds to download a web
    page that doesn't contain a lot of data (a few small pictures, an
    advertisement or 2, maybe more than one frame, but not much in terms of
    overall data to be transferred) - which implies that if this is explained in
    terms of the aforementioned latency, then there's quite a few messages going
    back and forth. (A ping of a remote site takes about 1.3 seconds.)

    Cheers,
    Rob

    --
    To reply by email send to optusnet.com.au instead of nomail.afraid.org
     
    Sisyphus, Aug 17, 2005
    #1
    1. Advertising

  2. Sisyphus

    Simon Taylor Guest

    Hello Rob,

    > One of the nice things about Net::FTP is that if you run it with debugging
    > switched on, you get a report of the actual communication that's taking
    > place between the local box and the remote ftp server.


    Yes, it's rather slick.

    > Is there a module that provides the same sort of report in relation to http
    > downloads ?


    I've sometimes used the LWP GET command as follows:

    GET -Sdxu http://www.yahoo.com.au

    Regards,

    Simon Taylor
     
    Simon Taylor, Aug 17, 2005
    #2
    1. Advertising

  3. Sisyphus

    Sisyphus Guest

    "Simon Taylor" <> wrote in message
    news:ddug4i$1k90$...
    > Hello Rob,
    >
    > > One of the nice things about Net::FTP is that if you run it with

    debugging
    > > switched on, you get a report of the actual communication that's taking
    > > place between the local box and the remote ftp server.

    >
    > Yes, it's rather slick.
    >
    > > Is there a module that provides the same sort of report in relation to

    http
    > > downloads ?

    >
    > I've sometimes used the LWP GET command as follows:
    >
    > GET -Sdxu http://www.yahoo.com.au
    >


    That's a nix command, right ? It's probably the sort of thing I'm looking
    for ... but I'm on Win32 :)

    I half expected that LWP::UserAgent or HTTP::Request/Response might
    implement the verbosity I'm after since they obviously know all about the
    http protocol, but I can't find anything in their docs that helps in that
    regard.

    I vaguely recall having used some sniffer type (non-perl) program a few
    years back .... I might have to google that up again if there's no
    ready-made perl solution.

    Cheers,
    Rob
     
    Sisyphus, Aug 17, 2005
    #3
  4. Sisyphus

    Brian Wakem Guest

    Brian Wakem, Aug 17, 2005
    #4
  5. Sisyphus

    ko Guest

    Sisyphus wrote:
    > Hi,
    >
    > One of the nice things about Net::FTP is that if you run it with debugging
    > switched on, you get a report of the actual communication that's taking
    > place between the local box and the remote ftp server.
    >
    > Is there a module that provides the same sort of report in relation to http
    > downloads ?
    >
    > I have a satellite broadband connection, and when surfing the web I find
    > it's about as slow as my old 28kbps dial-up connection was. (There's no
    > problem when it comes to http or ftp downloads of large files - it's just
    > that in general surfing it's fairly slow.) I suspect this has something to
    > do with the latency involved in the passing of communications via satellite
    > between my PC and the remote web server - but I don't know how many times
    > messages are passed back and forth, and it would be informative if there was
    > a perl module that would dispay this info (if only in terms of a report
    > similar to that offered by Net::FTP).
    >
    > It's not uncommon for my browser to take 15-20 seconds to download a web
    > page that doesn't contain a lot of data (a few small pictures, an
    > advertisement or 2, maybe more than one frame, but not much in terms of
    > overall data to be transferred) - which implies that if this is explained in
    > terms of the aforementioned latency, then there's quite a few messages going
    > back and forth. (A ping of a remote site takes about 1.3 seconds.)
    >
    > Cheers,
    > Rob
    >


    For a start, how about something like this:

    use strict;
    use warnings;
    use LWP;

    my $ua = LWP::UserAgent->new(
    requests_redirectable => [],
    max_redirect => 100,
    );

    verbose_http('http://hotmail.com/');

    sub verbose_http {
    push my @urls, shift;
    while (my $url = shift @urls) {
    my $r = $ua->get($url);
    if ($r->is_redirect) {
    print $r->headers->as_string . "\n";
    my $redirect = $r->header('Location');
    push @urls, $redirect;
    } elsif ($r->is_success) {
    print $r->content . "\n";
    }
    }
    }
    __END__

    HTH - keith
     
    ko, Aug 17, 2005
    #5
  6. Sisyphus <> wrote:


    > One of the nice things about Net::FTP is that if you run it with debugging
    > switched on, you get a report of the actual communication that's taking
    > place between the local box and the remote ftp server.
    >
    > Is there a module that provides the same sort of report in relation to http
    > downloads ?



    Web Scraping Proxy

    http://www.research.att.com/~hpk/wsp/


    > I have a satellite broadband connection, and when surfing the web I find
    > it's about as slow as my old 28kbps dial-up connection was. (There's no
    > problem when it comes to http or ftp downloads of large files - it's just
    > that in general surfing it's fairly slow.)



    Spyware would provide those symptoms too...


    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
     
    Tad McClellan, Aug 17, 2005
    #6
  7. Sisyphus

    ko Guest

    Eric Schwartz wrote:
    > ko <> writes:
    >
    >>sub verbose_http {
    >> push my @urls, shift;
    >> while (my $url = shift @urls) {

    >
    >
    > What's the point of this? Why not just
    > sub verbose_http {
    > my $url = shift;
    >
    > <snip rest of code>
    >
    > -=Eric


    Yes, could've use recursion:

    sub verbose_http {
    my $url = shift;
    my $r = $ua->get($url);
    if ($r->is_redirect) {
    print $r->headers->as_string . "\n\n";
    my $redirect = $r->header('Location');
    verbose_http($redirect);
    } elsif ($r->is_success) {
    print $r->content . "\n";
    }
    }

    Just a personal preference not to :)

    <offtopic>
    If the OP uses Firefox, have a look here:
    http://livehttpheaders.mozdev.org/

    If I remember correctly, it was a ~60KB or so download. Very nice.
    </offtopic>

    keith
     
    ko, Aug 23, 2005
    #7
  8. Sisyphus

    Anno Siegel Guest

    ko <> wrote in comp.lang.perl.misc:
    > Eric Schwartz wrote:
    > > ko <> writes:
    > >
    > >>sub verbose_http {
    > >> push my @urls, shift;
    > >> while (my $url = shift @urls) {

    > >
    > >
    > > What's the point of this? Why not just
    > > sub verbose_http {
    > > my $url = shift;
    > >
    > > <snip rest of code>
    > >
    > > -=Eric

    >
    > Yes, could've use recursion:
    >
    > sub verbose_http {
    > my $url = shift;
    > my $r = $ua->get($url);
    > if ($r->is_redirect) {
    > print $r->headers->as_string . "\n\n";
    > my $redirect = $r->header('Location');
    > verbose_http($redirect);
    > } elsif ($r->is_success) {
    > print $r->content . "\n";
    > }
    > }
    >
    > Just a personal preference not to :)


    Huh? I don't get it. How does one force you to use recursion and
    the other doesn't?

    sub foo {
    my @urls = shift;
    while ( my $url = shift @urls ) {
    # do something
    }
    }

    sub bar {
    my $url = shift;
    # do something
    }


    These do the same thing, except that foo() is unnecessarily roundabout.
    Minor differences may result from "# do something" being enclosed in a
    block in foo(), but not in bar(), but how foo() would avoid recursion
    needed in bar() is beyond me.

    Anno
    --
    If you want to post a followup via groups.google.com, don't use
    the broken "Reply" link at the bottom of the article. Click on
    "show options" at the top of the article, then click on the
    "Reply" at the bottom of the article headers.
     
    Anno Siegel, Aug 25, 2005
    #8
  9. Sisyphus

    ko Guest

    Anno Siegel wrote:
    > ko <> wrote in comp.lang.perl.misc:
    >
    >>Eric Schwartz wrote:


    [snip]

    > Huh? I don't get it. How does one force you to use recursion and
    > the other doesn't?
    >
    > sub foo {
    > my @urls = shift;
    > while ( my $url = shift @urls ) {
    > # do something
    > }
    > }
    >
    > sub bar {
    > my $url = shift;
    > # do something
    > }
    >
    >
    > These do the same thing, except that foo() is unnecessarily roundabout.


    I understood that they accomplish the same thing. The roundabout part, I
    obviously did not :)

    > Minor differences may result from "# do something" being enclosed in a
    > block in foo(), but not in bar(), but how foo() would avoid recursion
    > needed in bar() is beyond me.


    Sorry, mistaken/sloppy terminology. I've always associated the word
    recursion with a subroutine *explicitly* calling itself:

    sub bar {
    # do stuff and sometime later..
    bar()
    }

    Rather than the sub's *depth*. For whatever reason when I started
    programming it just didn't look right to have a sub call itself, and the
    habit has stuck with me...

    Thanks - keith
     
    ko, Aug 25, 2005
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Morris

    slow network image transfers

    Morris, Oct 25, 2004, in forum: Java
    Replies:
    2
    Views:
    933
    Morris
    Nov 5, 2004
  2. Matt
    Replies:
    5
    Views:
    469
    John Harrison
    Jul 12, 2004
  3. Limey Drink

    Automating FTP file transfers

    Limey Drink, Nov 12, 2003, in forum: Python
    Replies:
    7
    Views:
    636
    Jamey Cribbs
    Nov 15, 2003
  4. Andrew Robert
    Replies:
    1
    Views:
    1,450
    Dariusz Suchojad
    May 7, 2006
  5. John Nagle
    Replies:
    11
    Views:
    649
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Dec 14, 2006
Loading...

Share This Page