Very Interesting max_size in UserAgent

Discussion in 'Perl Misc' started by Public Interest, Oct 7, 2003.

  1. #!/usr/bin/perl
    use Net::HTTP;
    use LWP::UserAgent;

    $ua = LWP::UserAgent->new ( );
    $ua->max_size(2000);
    $url = 'http://www.yahoo.com';

    my $req = HTTP::Request->new(GET => "$url");

    $htmlcode = $ua->request($req)->content;
    print $htmlcode;

    # the problem is if i change 2000 to 1950 in max_size, the $htmlcode is
    still the same. Looks to me that the server is flushing and client is using
    buffer to hold it and the server decide how much each flushing is giving,
    and if each flushing is 1024 bytes, and max_size is checked after each
    flushing. So, max size of 1025 to max size of 2047 will give the same
    result: both are 2048. Is that right? So max_size in UserAgent is really an
    estimated value.
    Public Interest, Oct 7, 2003
    #1
    1. Advertising

  2. Public Interest

    ko Guest

    Public Interest wrote:
    > #!/usr/bin/perl
    > use Net::HTTP;
    > use LWP::UserAgent;
    >
    > $ua = LWP::UserAgent->new ( );
    > $ua->max_size(2000);
    > $url = 'http://www.yahoo.com';
    >
    > my $req = HTTP::Request->new(GET => "$url");
    >
    > $htmlcode = $ua->request($req)->content;
    > print $htmlcode;
    >
    > # the problem is if i change 2000 to 1950 in max_size, the $htmlcode is
    > still the same. Looks to me that the server is flushing and client is using
    > buffer to hold it and the server decide how much each flushing is giving,
    > and if each flushing is 1024 bytes, and max_size is checked after each
    > flushing. So, max size of 1025 to max size of 2047 will give the same
    > result: both are 2048. Is that right? So max_size in UserAgent is really an
    > estimated value.


    No, it is a specific limit. From the LWP::UserAgent docs:

    $ua->max_size([$bytes])

    Get/set the size limit for response content. The default is undef, which
    means that there is no limit. If the returned response content is only
    partial, because the size limit was exceeded, then a ``Client-Aborted''
    header will be added to the response.

    So if you want to check if max_size has been exceeded, you have to test
    for if there is a Client-Aborted header:

    #!/usr/intel/bin/perl -w
    use strict;

    use LWP::UserAgent;

    my $ua = LWP::UserAgent->new( max_size => 0);
    my $url = 'http://www.yahoo.com';
    print "Exceeded max_size\n" if $ua->get($url)->header('Client-Aborted');

    HTH - keith
    ko, Oct 7, 2003
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin Goodsell

    Exceeding container::max_size()?

    Kevin Goodsell, Apr 3, 2004, in forum: C++
    Replies:
    1
    Views:
    362
    Victor Bazarov
    Apr 4, 2004
  2. alexey_m
    Replies:
    2
    Views:
    327
    =?ISO-8859-1?Q?Stefan_N=E4we?=
    Dec 22, 2005
  3. t

    max_size()

    t, Oct 2, 2007, in forum: C++
    Replies:
    4
    Views:
    403
    James Kanze
    Oct 3, 2007
  4. Sarath
    Replies:
    2
    Views:
    591
    Knockr
    Aug 19, 2009
  5. Great Deals
    Replies:
    1
    Views:
    230
    laura fairhead
    Oct 1, 2003
Loading...

Share This Page