form requires two parameters

[Guest]

I am trying to submit a form that requires hidden fields from another form
on the same page before being submitted the relevant html is: < <form
name="poxy_url_form" method="get" action="/index.php">
<input type="hidden" name="q" value="" id="url_input" />
<input type="hidden" name="hl" value="" />
</form>
<form name="poxy_settings_form" method="get" action="" onsubmit="return
submit_form();">
<table style="width: 100%">
<tr><td class="option" style="width: 20%">URL</td><td class="option"
style="width: 80%">&nbsp;<input type="text" name="url" size="70" value=""
/></td></tr>

I have tried it with both $mech->submit("poxy_url_form") and
$mech->submit("poxy_settings_form"), the first passes the value but nothing
is submitted. The full code is:

#!/usr/bin/perl;
use WWW::Mechanize;

my $mech = WWW::Mechanize->new();
$mech->get("http://witchproxy.info");
$mech->form("poxy_url_form");
$mech->set_visible('poxy_url_form','http://news.google.com');
$mech->form("poxy_settings_form");
$mech->field('url','http://news.google.com');
print $mech->content();

############################################################################
#################
As seen from the print out it doesn't show any content from news.google.com
but just the content from the first url, meaning nothing was submitted to
the form

 

[Mumia W.]

I am trying to submit a form that requires hidden fields from another form
on the same page before being submitted [...]

my $mech = WWW::Mechanize->new();
$mech->get("http://witchproxy.info");
[...]
As seen from the print out it doesn't show any content from news.google.com
but just the content from the first url, meaning nothing was submitted to
the form

That website, witchproxy.info, says that Javascript is required to use
the page. Perhaps they don't want automated access there.

 

[Bill H]

I am trying to submit a form that requires hidden fields from another form
on the same page before being submitted the relevant html is: < <form
name="poxy_url_form" method="get" action="/index.php">
<input type="hidden" name="q" value="" id="url_input" />
<input type="hidden" name="hl" value="" />
</form>
<form name="poxy_settings_form" method="get" action="" onsubmit="return
submit_form();">
<table style="width: 100%">
<tr><td class="option" style="width: 20%">URL</td><td class="option"
style="width: 80%">&nbsp;<input type="text" name="url" size="70" value=""
/></td></tr>

I have tried it with both $mech->submit("poxy_url_form") and
$mech->submit("poxy_settings_form"), the first passes the value but nothing
is submitted. The full code is:

#!/usr/bin/perl;
use WWW::Mechanize;

my $mech = WWW::Mechanize->new();
$mech->get("http://witchproxy.info");
$mech->form("poxy_url_form");
$mech->set_visible('poxy_url_form','http://news.google.com');
$mech->form("poxy_settings_form");
$mech->field('url','http://news.google.com');
print $mech->content();

###########################################################################­#
#################
As seen from the print out it doesn't show any content from news.google.com
but just the content from the first url, meaning nothing was submitted to
the form

Both of the forms are on the same web page? If so, I dont see how
using perl would give you values from another form when it is
submitted, the browser is only going to send what is in the form being
submitted, not what is in other forms on the same web page. To get
values from another form on the same webpage into the form you are
submitting you need to use JavaScript in the page so that the clients
browser gets the values for you and puts them into hidden fields on
the form being submitted, Or am I completly missing the meaning of
this problem?

Bill H

 

[kelticeye]

On 08/14/2007 08:48 PM, Nospam wrote:


That website, witchproxy.info, says that Javascript is required to use
the page. Perhaps they don't want automated access there.

Mechanize should have no problem with hidden fields, but does have a
problem with javascript (as stated in cpan there is no support). The
form ("poxy_settings_form") which is the form I think you are trying
to submit to, seems to use a Javascript function submit_form() which
itself links to two other javascript functions, the default being
base64_encode(), which seems to transform the text in the "url" field.
You need to figure out what this transformation is before submitting
your url.

Using Firefox firebug is probably the best way to look through the
Javascript of a webpage.

34/* base64 encode code below is not my own, although I did modify
it. */
35
36function base64_encode(str)
37{
38 var out = '';
39 var t, x, y ,z;
40
41 for (var i = 0; i < str.length; i += 3)
42 {
43 t = Math.min(3, str.length - i);
44 if (t == 1)
45 {
46 x = str.charCodeAt(i);
47 out += alnum.charAt((x >> 2));
48 out += alnum.charAt(((x & 0X00000003) << 4));
49 out += '--';
50 }
51 else if (t == 2)
52 {
53 x = str.charCodeAt(i);
54 y = str.charCodeAt(i+1);
55 out += alnum.charAt((x >> 2));
56 out += alnum.charAt((((x & 0X00000003) << 4) | (y >> 4)));
57 out += alnum.charAt(((y & 0X0000000f) << 2));
58 out += '-';
59 }
60 else
61 {
62 x = str.charCodeAt(i);
63 y = str.charCodeAt(i+1);
64 z = str.charCodeAt(i+2);
65 out += alnum.charAt((x >> 2));
66 out += alnum.charAt((((x & 0x00000003) << 4) | (y >> 4)));
67 out += alnum.charAt((((y & 0X0000000f) << 2) | (z >> 6)));
68 out += alnum.charAt((z & 0X0000003f));
69 }
70 }
71
72 return out;
73}

 

[Guest]

I have noticed if I were to manually deselect base 64 that the url when
submitted is

http://witchproxy.info/index.php?q=http://news.google.com&hl=1111100001

Does anyone know where on the form this value of &hl=1111100001 is obtained,
or must I assume it comes with the submit_form() javascript function?

 

[Guest]

I also notice when manually submitted compared to trying it with the perl
script that several lines of

%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&o
ps%5
B%5D=on

are appended to the url (even though it is not passed through the
submit_form() javascript function), when submitted manually via the bre is
no such addition.

 

[kelticeye]

I also notice when manually submitted compared to trying it with the perl
script that several lines of

%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&ops%5B%5D=on&o
ps%5
B%5D=on

are appended to the url (even though it is not passed through the
submit_form() javascript function), when submitted manually via the bre is
no such addition.

Sorry for taking a while to get back to you-

I've looked at the form again and have determined that you cannot
submit the form directly as the form relies on javascript in the form
to past flags - the binary code that follows "hl=" is related to
whether each sucessive checkbox in the form is 'on' or 'off'.

As mentioned ad nauseum, Mechanize cannot cope with javascript, but
because the form uses this to encode it's url GET method, the results
of submitting a form can be replicated by substituting parts of the
url.

Trying this in the browser is unsuccessful as the webpage script
obviously checks the referer header (as many others do), however,
Mechanize allows us to define our headers through it's add_header()
method.

Hence:

use WWW::Mechanize;

my $mech = WWW::Mechanize->new();
my $query = "http://news.bbc.co.uk"; # whatever url you want
# a submission where all checkboxes are unticked:
my $url = "http://www.witchproxy.info/index.php?q=
$query&hl=0000000000";

$mech->add_header( Referer => "http://www.witchproxy.info");

$mech->get($url);
print $mech->content(),"\n";

 

[kelticeye]

Sorry for taking a while to get back to you-

I've looked at the form again and have determined that you cannot
submit the form directly as the form relies on javascript in the form
to past flags - the binary code that follows "hl=" is related to
whether each sucessive checkbox in the form is 'on' or 'off'.

As mentioned ad nauseum, Mechanize cannot cope with javascript, but
because the form uses this to encode it's url GET method, the results
of submitting a form can be replicated by substituting parts of the
url.

Trying this in the browser is unsuccessful as the webpage script
obviously checks the referer header (as many others do), however,
Mechanize allows us to define our headers through it's add_header()
method.

Hence:

use WWW::Mechanize;

my $mech = WWW::Mechanize->new();
my $query = "http://news.bbc.co.uk"; # whatever url you want
# a submission where all checkboxes are unticked:
my $url = "http://www.witchproxy.info/index.php?q=
$query&hl=0000000000";

$mech->add_header( Referer => "http://www.witchproxy.info");

$mech->get($url);
print $mech->content(),"\n";