LWP Post

D

Dale Gerdemann

I'm trying to figure out how use LWP::UserAgent to get dynamically
generated Web pages using the POST method. I've followed the examples
in the O'Reilly book: Perl & LWP by Sean M. Burke. There's an example
there of posting to the AltaVista Babelfish translator, and this
example works fine for me. But when I try to do someting similar on
another page, it fails.

For example, I tried to get morphological analysis (word structure
analysis) from http://www.uni-plovdiv.bg/dcs/morphe.htm, and even
though it's a very simple minded form, I had no success. Sean Burke
has a program in his book called formpairs, which allows you to test
what posted info a form is expecting. I used this and checked very
carefully, but still, no success.

Here's what the form looks like:

function FrontPage_Form1_Validator(theForm)
{

if (theForm.D1.selectedIndex < 0)
{
alert("Please select one of the \"D1\" options.");
theForm.D1.focus();
return (false);
}
return (true);
}
//--></script><!--webbot BOT="GeneratedScript" endspan --><form
action="http://rdesc.pu.acad.bg/cgi-bin/cgi_morf.exe" method="POST"
onsubmit="return FrontPage_Form1_Validator(this)"
name="FrontPage_Form1">
<p><textarea name="S1" rows="5" cols="82"></textarea></p>
<p><!--webbot bot="Validation" b-value-required="TRUE" --><select
name="D1" size="3">
<option>1.Windows ( àáâãäåæçèéêëìíîïðñòóôõö÷øùúüþÿ )</option>
<option>2.Macintosh ( abvgde'zijklmnoprstufxchw]=;_q )</option>
<option>3.QWERTY ( abwgdevzijklmnoprstufhc`[]yx\q )</option>
</select></p>

<p><input type="submit" name="B1" value="Analysis"></p>
</form>


You see that there's a bit of Javascript getting in the way, but I
don't think thats a problem here. [You also see that the author of the
page mixed up is terminology and referred to ASCII as QWERTY.
But that's beside the point.] I tried to access this page with the
following:

use strict;
use LWP;
my $browser;
sub do_POST {
# Parameters:
# the URL,
# an arrayref or hashref for the key/value pairs,
# and then, optionally, any header lines: (key,value, key,value)
$browser = LWP::UserAgent->new( ) unless $browser;
my $resp = $browser->post(@_);
return ($resp->content, $resp->status_line, $resp->is_success, $resp)
if wantarray;
return unless $resp->is_success;
return $resp->content;
}



my ($content, $message, $is_success) = do_POST(
'http://www.uni-plovdiv.bg/dcs/morphe.htm',
[ 'S1' => 'ne', 'B1' => "Analysis",
'D1'=>'3.QWERTY ( abwgdevzijklmnoprstufhc`[]yx\q )' ],
);

print "$content\n";



Anybody have any ideas???


Dale Gerdemann
 
T

Tim Heaney

my ($content, $message, $is_success) = do_POST(
'http://www.uni-plovdiv.bg/dcs/morphe.htm',
[ 'S1' => 'ne', 'B1' => "Analysis",
'D1'=>'3.QWERTY ( abwgdevzijklmnoprstufhc`[]yx\q )' ],
);

print "$content\n";



Anybody have any ideas???

You don't want to post to the HTML page with the form on it, you want
to post to their program. Try something like

my ($content, $message, $is_success) = do_POST(
'http://rdesc.pu.acad.bg/cgi-bin/cgi_morf.exe',
[ 'S1' => 'ne',
'D1' => '3.QWERTY ( abwgdevzijklmnoprstufhc`[]yx\q )',
'B1' => "Analysis",
],
);

instead (note the different URL). When I do that, I get

<HTML><HEAD><TITLE>DELPHI CGI TEST</TITLE></HEAD>
<BODY bgcolor="#FFFFFF">
<STRONG>Result:<BR><BR>
<P>ne :particle, :189 1 base form:ne</P>
</B></P>
</BODY>

I hope this helps,

Tim
 
T

Tad McClellan

Dale Gerdemann said:
I'm trying to figure out how use LWP::UserAgent to get dynamically
generated Web pages using the POST method.
For example, I tried to get morphological analysis (word structure
analysis) from http://www.uni-plovdiv.bg/dcs/morphe.htm, and even
though it's a very simple minded form, I had no success.

Here's what the form looks like:
[snip]

<form
action="http://rdesc.pu.acad.bg/cgi-bin/cgi_morf.exe" method="POST"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


There is where the message should be posted to...

my ($content, $message, $is_success) = do_POST(
'http://www.uni-plovdiv.bg/dcs/morphe.htm',
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


.... but that isn't where you are actually posting it!

Anybody have any ideas???


Post to the correct URL. :)



BTW, the "web scraping proxy" is a wildly valuable tool for
reverse-engineering such things:

http://www.research.att.com/~hpk/wsp/

It wrote this Perl code for me when I tried it:

$request = POST "http://rdesc.pu.acad.bg/cgi-bin/cgi_morf.exe" , [
'S1' => "a Bulgarian sentence",
'D1' => "3.QWERTY ( abwgdevzijklmnoprstufhc`[]yx\q )",
'B1' => "Analysis",
] ;
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,755
Messages
2,569,535
Members
45,007
Latest member
obedient dusk

Latest Threads

Top