Newbie LWP Question

Bumble · Jan 7, 2004

Hi all, trying to develop a program for my Uni project, it is given part of
the address from a form and then it should go to that address and retrieve
the page, now I only want a section of that page, so I'm wondering where I
could do that? Would I need to put it into a text file and then find the
required text from there, or can I manipulate a variable already used to
extract just the text I need? Sorry, just having a bit of trouble getting my
head around the code...

#!/usr/bin/perl -w
use CGI ':standard';
require LWP::UserAgent;

$ua = LWP::UserAgent->new;

$thisisit=param('thisisitinput');

$request = HTTP::Request->new('GET', 'http://www.whatever.com/$thisisit');

$response = $ua->request($request);

print ("<html><body bgcolor\"white\">");
print ("<p>$locate</p>");

print ("</body> </html>");

--
Bumble
The 18th 'Biggest Gay on Usenet'
http://bumble.rumble.at

"Your heart felt good it was drippin' pitch and made of wood, and your hands
and knees felt cold and wet on the grass to me. Outside naked, shiverin'
looking blue, from the cold sunlight that's reflected off the moon. Baby cum
angels fly around you reminding you we used to be three and not just two.
And that's how the world began. And that's how the world will end."

Bumble · Jan 8, 2004

Sherm said:
Either approach could be used to give you the end result you're
looking for. So the real question isn't "which do I have to do," but
"which is best for my situation?"

In the simplest case, where you're retrieving the full text every
time, storing it to a file and reading the file back in is a waste of
disk i/o and space.

However, if the page you're retrieving rarely changes, then fetching
it every time you need it is a waste of network bandwidth. You could
get better results in this case by first doing a minimal HTTP "HEAD"
request to get the modification time, and only doing the full "GET"
request if the locally-cached copy of the file needs to be refreshed.

sherm--

Okay, thanks. And now for a really dumbass question, where is the retrieved
html? I'm guessing it's in the $response? But when I've tried to get it to
print $response to the screen it's not doing anything?

--
Bumble
The 18th 'Biggest Gay on Usenet'
http://bumble.rumble.at

"Your heart felt good it was drippin' pitch and made of wood, and your
hands and knees felt cold and wet on the grass to me. Outside naked,
shiverin' looking blue, from the cold sunlight that's reflected off the
moon. Baby cum angels fly around you reminding you we used to be three
and not just two. And that's how the world began. And that's how the
world will end."

gnari · Jan 8, 2004

Bumble said:
Okay, thanks. And now for a really dumbass question, where is the retrieved
html? I'm guessing it's in the $response? But when I've tried to get it to
print $response to the screen it's not doing anything?

if $response is an instance of HTTP::Response then try
perldoc HTTP::Response

or http://perldoc.com/perl5.8.0/lib/HTTP/Response.html

probably there is a suitable method documented.

gnari

Sherm Pendley · Jan 8, 2004

Okay, thanks. And now for a really dumbass question, where is the retrieved
html? I'm guessing it's in the $response? But when I've tried to get it to
print $response to the screen it's not doing anything?

$response is an object - along with the retrieved html, it has the HTTP
status, among other things. So, for example, you could do this:

if ($response->is_success) {
print $response->content;
} else {
print "Error: " . $response->status_line . "\n";
}

You might want to also have a look at the 'lwpcook' perldoc page - lots of
interesting examples there.

sherm--

Bumble · Jan 9, 2004

Sherm said:
$response is an object - along with the retrieved html, it has the
HTTP status, among other things. So, for example, you could do this:

if ($response->is_success) {
print $response->content;
} else {
print "Error: " . $response->status_line . "\n";
}

You might want to also have a look at the 'lwpcook' perldoc page -
lots of interesting examples there.

sherm--

Thanks all!

--
Bumble
The 18th 'Biggest Gay on Usenet'
http://bumble.rumble.at

"Your heart felt good it was drippin' pitch and made of wood, and your
hands and knees felt cold and wet on the grass to me. Outside naked,
shiverin' looking blue, from the cold sunlight that's reflected off the
moon. Baby cum angels fly around you reminding you we used to be three
and not just two. And that's how the world began. And that's how the
world will end."

Bo Johanson · Jan 11, 2004

Bumble said:
Thanks all!

To extract the portion of the page you're getting, take a look at the
HTML:: modules. Check perldoc HTML::TokeParser.

Bumble · Jan 11, 2004

Having further problems with the below, I'm getting a 'Error: 500 Can't
connect...' when running it from the server, however when I run it from my
own PC it seems to be working? I'm unsure why it would work on my PC from
the command prompt but why it won't retrieve a HTML page when it's on the
server?

TIA

Hi all, trying to develop a program for my Uni project, it is given
part of the address from a form and then it should go to that address
and retrieve the page, now I only want a section of that page, so I'm
wondering where I could do that? Would I need to put it into a text
file and then find the required text from there, or can I manipulate
a variable already used to extract just the text I need? Sorry, just
having a bit of trouble getting my head around the code...

#!/usr/bin/perl -w
use CGI ':standard';
require LWP::UserAgent;

$ua = LWP::UserAgent->new;

$thisisit=param('thisisitinput');

$request = HTTP::Request->new('GET',
'http://www.whatever.com/$thisisit');

$response = $ua->request($request);

print ("<html><body bgcolor\"white\">");
print ("<p>$locate</p>");

print ("</body> </html>");

--
Bumble
The 18th 'Biggest Gay on Usenet'
http://bumble.rumble.at

"Your heart felt good it was drippin' pitch and made of wood, and your
hands and knees felt cold and wet on the grass to me. Outside naked,
shiverin' looking blue, from the cold sunlight that's reflected off the
moon. Baby cum angels fly around you reminding you we used to be three
and not just two. And that's how the world began. And that's how the
world will end."

Sherm Pendley · Jan 11, 2004

Having further problems with the below, I'm getting a 'Error: 500 Can't
connect...' when running it from the server, however when I run it from my
own PC it seems to be working? I'm unsure why it would work on my PC from
the command prompt but why it won't retrieve a HTML page when it's on the
server?

Sounds like outbound connections from the web server may be blocked at the
firewall. This is often done to help prevent the spread of viruses - if
the web server is infected somehow, the firewall will prevent the virus
from connecting to other servers to propagate itself.

Ask your system admin about it.

sherm--

reading LWP in chunks	6	Oct 18, 2010
Strange behaviour of LWP::UserAgent	0	May 21, 2010
can LWP handle this?	7	Oct 21, 2008
LWP and 302 redirects	2	Oct 4, 2006
Script using LWP::UserAgent is sometimes failing with 500 error,although server reports 200	9	Aug 23, 2011
NTLM and LWP::UserAgent	4	Sep 12, 2006
How can I keep LWP::UserAgent from adding the http-equiv strings fromthe Head section of the page?	5	Mar 18, 2009
case study: want to print redirects when LWP::UserAgent makes arequest	2	Mar 22, 2010

Newbie LWP Question

Bumble

Bumble

gnari

Sherm Pendley

Bumble

Bo Johanson

Bumble

Sherm Pendley

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads