Understanding Mechanize

B

Bryan Balfour

hi, I'm new to perl and have been trying to get into using
WWW::Mechanize on WindowsXP. I'm stuck on trying to use it to submit a
username and password. The following snippet of code shows what I'm
trying to do:

my($mech) = WWW::Mechanize->new(autocheck => 1,
cookie_jar => $cookieJar);
$mech->get($url);
$mech->follow_link(text => "Begin");
if($mech->content() =~ m/Please log in./)
{
print("***** Please log in. *****\n");
outputContent($mech->content(), "Login.html");
$mech->set_visible($userName, $password);
my($response) = $mech->click();
if($response->content() =~ m/Please try again/)
{
print("Logon details incorrect\n");
}
outputContent($response->content(), "Loggedin.html");
}

sub outputContent
{
my($content, $fileName) = @_;
my($tree) = HTML::TreeBuilder->new;
$tree->ignore_ignorable_whitespace(0);
$tree->no_space_compacting(1);
$tree->parse($content) || die $!;
open(OUT, ">$fileName") || die "Can't write: $!";
print OUT $tree->as_HTML;
close(OUT);
$tree->delete;
}

I'm deliberately setting an incorrect password so expect to see the
messages:

***** Please log in. *****
Logon details incorrect

I'm getting the first one but not the second. Conclusion: I've got a
problem with the 'set_visible' or 'click' methods.

The 2 files output by outputContent are identical and are the html for
entering username and password. Looking at these two input fields, they
are NOT defined on a form but in a table. (Is the html treated as one
form by default?)

I've tried using other Mechanize methods such as:

$mech->form_number(1);
$mech->field('user', $userName);
$mech->field('password', $password);
my($response) = $mech->click('submit');

but get the same result. (Curiously, the page contains only one form at
the beginning but that is hidden. The above code generates no error
messages so what form is form_number(1)?)

Am I missing a step as the 'click' method is not returning a new
response but the old one?

I'd appreciate your comments, hints etc.

Bryan
 
B

Brian Wakem

Bryan said:
Am I missing a step as the 'click' method is not returning a new
response but the old one?

I'd appreciate your comments, hints etc.

Bryan


Most WWW::Mechanize problems are caused by javascript on the page you are
playing with. Does it use any javascript in the form or upon submission?
 
B

Bryan Balfour

Brian said:
Most WWW::Mechanize problems are caused by javascript on the page you are
playing with. Does it use any javascript in the form or upon submission?
Yes, it's full of it. For example:

<script language="javascript">
<!--
if(window!=top)
top.location.href=location.href;
// -->
</script>
<script language="javascript">
<!--
function doSubmit() {if(document.forms[0].btn.value == "" ) {return
false;} else { return true; } }// -->
</script>
<script language="JavaScript">
function userMessage(message)
{
document.write("<table border='0' cellpadding='0' cellspacing='1'
bgcolor='#535353' width='695' height='40' align='center'>\n");
document.write("<tr><td style='border-left: 1px solid black;
border-right: 1px solid black; border-bottom: 1px solid black'
valign='center' align='center' colspan='3' bgcolor='#ffffff'>\n");
document.write("<font class='cn'>"+message+"</font>\n");
document.write("</td></tr>\n");
document.write("</table>\n");
}
document.cookie = "Enabled=true";
var cookiesEnabled = false;
var cookieValid = document.cookie;
if (cookieValid.indexOf("Enabled=true") != -1) cookiesEnabled = true;
else cookiesEnabled = false;
//cookiesEnabled = false;
</script>

What sort of things should I be looking for?

I appreciate your help....

Bryan
 
B

Brian Wakem

Bryan said:
Yes, it's full of it. For example:

<script language="javascript">
</script>

What sort of things should I be looking for?

I appreciate your help....

Bryan


WWW:Mechanize does not 'do' javascript, so you are screwed. Well not quite,
you have to work out what the javascript does to the form params and
contruct your own POST request accordingly.
 
B

Bryan Balfour

WWW:Mechanize does not 'do' javascript, so you are screwed. Well not quite,
you have to work out what the javascript does to the form params and
contruct your own POST request accordingly.
I was afraid that would be the case. Looks like I'll need to learn a
bit of javascript.

As a thought, do you know if there are any tools around that would
enable me to run the html I've captured (or the actual html) and show
me what's posted? (Ideally single stepping with watches.)

Thanks for taking the time and trouble of replying. No doubt I'll be
posting again soon as I delve into POST requests etc.

Bryan
 
B

Brian Wakem

Bryan said:
I was afraid that would be the case. Looks like I'll need to learn a
bit of javascript.

As a thought, do you know if there are any tools around that would
enable me to run the html I've captured (or the actual html) and show
me what's posted? (Ideally single stepping with watches.)

Thanks for taking the time and trouble of replying. No doubt I'll be
posting again soon as I delve into POST requests etc.

Bryan


I use firefox with the Live HTTP Headers plugin -
http://livehttpheaders.mozdev.org/ - which will show all the communication
going back and forth. Just submit the form and watch what happens. Then
emulate this with $mech->post

push @{ $mech->requests_redirectable }, 'POST'; # in case of 302 redirect

$mech->post( $url,
Content => [
"username" => "user",
"password" => "pass",
"etc" => "etc",
],
);

WWW::Mechanize uses LWP::UserAgent, read more at
http://cpan.uwinnipeg.ca/htdocs/libwww-perl/LWP/UserAgent.html
 
C

ced

Bryan said:
I was afraid that would be the case. Looks like I'll need to learn a
bit of javascript.

As a thought, do you know if there are any tools around that would
enable me to run the html I've captured (or the actual html) and show
me what's posted? (Ideally single stepping with watches.)

Thanks for taking the time and trouble of replying. No doubt I'll be
posting again soon as I delve into POST requests etc.

You can also see what LWP is doing behind the curtain by adding:


use LWP::Debug qw(+);


hth,
 
B

Bryan Balfour

Thanks for that. I'll give it a go.

P.S. Thanks too to all the others who took the trouble of replying with
their suggestions.

Bryan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,769
Messages
2,569,579
Members
45,053
Latest member
BrodieSola

Latest Threads

Top