Very Interesting max_size in UserAgent

[Public Interest]

#!/usr/bin/perl
use Net::HTTP;
use LWP::UserAgent;

$ua = LWP::UserAgent->new ( );
$ua->max_size(2000);
$url = 'http://www.yahoo.com';

my $req = HTTP::Request->new(GET => "$url");

$htmlcode = $ua->request($req)->content;
print $htmlcode;

# the problem is if i change 2000 to 1950 in max_size, the $htmlcode is
still the same. Looks to me that the server is flushing and client is using
buffer to hold it and the server decide how much each flushing is giving,
and if each flushing is 1024 bytes, and max_size is checked after each
flushing. So, max size of 1025 to max size of 2047 will give the same
result: both are 2048. Is that right? So max_size in UserAgent is really an
estimated value.

 

[ko]

#!/usr/bin/perl
use Net::HTTP;
use LWP::UserAgent;

$ua = LWP::UserAgent->new ( );
$ua->max_size(2000);
$url = 'http://www.yahoo.com';

my $req = HTTP::Request->new(GET => "$url");

$htmlcode = $ua->request($req)->content;
print $htmlcode;

# the problem is if i change 2000 to 1950 in max_size, the $htmlcode is
still the same. Looks to me that the server is flushing and client is using
buffer to hold it and the server decide how much each flushing is giving,
and if each flushing is 1024 bytes, and max_size is checked after each
flushing. So, max size of 1025 to max size of 2047 will give the same
result: both are 2048. Is that right? So max_size in UserAgent is really an
estimated value.

No, it is a specific limit. From the LWP::UserAgent docs:

$ua->max_size([$bytes])

Get/set the size limit for response content. The default is undef, which
means that there is no limit. If the returned response content is only
partial, because the size limit was exceeded, then a ``Client-Aborted''
header will be added to the response.

So if you want to check if max_size has been exceeded, you have to test
for if there is a Client-Aborted header:

#!/usr/intel/bin/perl -w
use strict;

use LWP::UserAgent;

my $ua = LWP::UserAgent->new( max_size => 0);
my $url = 'http://www.yahoo.com';
print "Exceeded max_size\n" if $ua->get($url)->header('Client-Aborted');

HTH - keith