using LWP to POST data via "javascript:document.form.submit()" link

[schruthensis]

Hello.

Can anybody solve this problem?

I need to POST data to a HTTPS login script that doesn't have a
standard "submit" button. The form appears to use javascript to submit
the document via the browser.

<form action='loginScript.cgi' method="post" name="loginForm">
<input name="userid" />
<input name="password" />
<a href="javascript:document.loginForm.submit()" >
<img src="someImage.gif" >
</a>
</form>

I have verified this by
1) disabling javascript in my a browser and replicating the error of
being unable to submit ... and ...
2) creating my own simple form (roughly the same as the above code)
on my own site to submit the data (to the site of interest) without
problem!

Any ideas on how to use perl to submit via this javascript dependent
form?

the only think I can think of is to have perl create and run a
temporary javascript enabled form/webpage and capture the results...
but this would be highly inelegant..

Thanks in advance,

Dave

 

[Eric Bohlman]

I need to POST data to a HTTPS login script that doesn't have a
standard "submit" button. The form appears to use javascript to submit
the document via the browser.

<form action='loginScript.cgi' method="post" name="loginForm">
<input name="userid" />
<input name="password" />
<a href="javascript:document.loginForm.submit()" >
<img src="someImage.gif" >
</a>
</form>

The only thing Javascript is being used for here is to allow submitting the
form by clicking a link image rather than a button. All the code in
question does is a normal POST submit with items "userid" and "password,"
and that's all you need to do with LWP; just treat it as if it were an
ordinary form with a submit button.

 

[John Bokma]

The only thing Javascript is being used for here is to allow
submitting the form by clicking a link image rather than a button.

Which is odd, since this can be done without JavaScript if I am not
mistaken :-D.

 

[Tad McClellan]

I need to POST data to a HTTPS login script that doesn't have a
standard "submit" button. The form appears to use javascript to submit
the document via the browser.

Any ideas on how to use perl to submit via this javascript dependent
form?

Reverse engineer what the javascript does, or simply spy on
the output from the javascript by using the

Web Scraping Proxy

http://www.research.att.com/~hpk/wsp/

and then duplicate that output.

 

[schruthensis]

Which is odd, since this can be done without JavaScript if I am not
mistaken :-D.

right, thats kinda the point. I have no problem with a standard
"submit button" (no javascript) version of any form

 

[schruthensis]

you are right! (ill explain below)

 

[schruthensis]

I have verified this by
1) disabling javascript in my a browser and replicating the error of
being unable to submit ... and ...
2) creating my own simple form (roughly the same as the above code)
on my own site to submit the data (to the site of interest) without
problem!

Any ideas on how to use perl to submit via this javascript dependent
form?

Well, looks like i didn't try an important permutation of submission
possibilities:

submitting from my own form with a submit button instead of the
javascript. (which does work)...... ,however, this does NOT work if i
disable javascript... which implies that somehow javascript is
required AFTER the login at some point and perhaps that the javascript
login is just a way to make sure only javascript enabled browsers
enter....

now, what do i do?

I need to figure out how javascript is working after the login.... I
would scrape, but doesn't that require that the communications are
clear text? i have been sniffing the transactions but this is a
secure, encrypted connection.. and i can't read anything..

 

[Matt Garrish]

you are right! (ill explain below)

Who is right? Explain below what?

I'm really starting to get a hate on for Google...

Matt

 

[Samwyse]

submitting from my own form with a submit button instead of the
javascript. (which does work)...... ,however, this does NOT work if i
disable javascript... which implies that somehow javascript is
required AFTER the login at some point and perhaps that the javascript
login is just a way to make sure only javascript enabled browsers
enter....

I'll wager that it's setting a cookie. You should be able to look at
the javascript and mentally execute it; I'd be surprised if it's more
that a few lines long.

 

[schruthensis]

I'll wager that it's setting a cookie. You should be able to look at
the javascript and mentally execute it; I'd be surprised if it's more
that a few lines long.

i think youre right, samwyse, the only javascript on the page (after
loging in) thats any different from the previous pages is the following
code snippet:

<script>
function getCookie(name)
{
//uses document.cookie.indexOf();
}
function setCookie(name, value, expires, path, domain, secure)
{
//uses something like:
document.cookie=(name,value,expires,path,domain,secure)
}
setCookie("cookiename0","eKIQUeS7L3meAjSRyCChvstAojxX7q3GUduqdSxE9dhylKsekTtVTnOSwawkYsSQEZ7JvnqbfzmUCfeUk-HczPbJDwiXjkWwHhUYVHRnUxCEUKPQlV5BqA8XgE9OHg3A7AUhvVTHqu+Mp2gtbNKvLaIKd8Jb5Aznc5i6s-G2lv8n0nW-wpgl3nkTG59e3Uw5i4mDRhm28KGjh3r-shVXOUZgsbd+Wsrp+r6u+aUmX1TrsyA+52mC057ArY12i0JlfijUI4Fhcpnt-e0rdPAcD32xRAC1BMdAVuSOzW13RAFVLA6QFKm2WaqIX1XTHXv4wMKW7ELlw6B6T-q8VxpaGV2dlTSXXhNphVR73ZjWhXBPfg+b7jPjaIUttZhRRGKZdEeREeEX+YOdPZcXcOB2IFaNK6tKCLjP-LlLxn1ugzmDAEvxDJ63YD2YQ5UAtUBRJuI6ul-u38JRL2Qs7vZ5snVj3soyZhJv164z51mquua1szFR-t-C0IovEFgkFF2I22KGC3QE+A35NZBBusvVPttyUJksYlpqTagPkA5aLGSSLLJ017xICP8kAAeWD39AJA",
"-1", "path", ".domain.com");
setCookie("cookiename1","codenumber=i3NOHiv7DS5gS", "-1", "path",
".domain.com");
setCookie("cookiename2","name=name", "expireTime", "path",
".domain.com");
</script>

I am thinking of using HTTP::cookies

But what about these randomly generated cookie values?? (like:
eKIQUeS7L3meAjS...etc) how am i supposed to reverse engineer that??
Does HTTP::cookies allow values to be set from javascript? I am
doubting it...

Thanks again,
d

 

[Samwyse]

setCookie("cookiename0","eKIQUeS7L3meAjSRyCChvstAojxX7q3GUduqdSxE9dhylKsekTtVTnOSwawkYsSQEZ7JvnqbfzmUCfeUk-HczPbJDwiXjkWwHhUYVHRnUxCEUKPQlV5BqA8XgE9OHg3A7AUhvVTHqu+Mp2gtbNKvLaIKd8Jb5Aznc5i6s-G2lv8n0nW-wpgl3nkTG59e3Uw5i4mDRhm28KGjh3r-shVXOUZgsbd+Wsrp+r6u+aUmX1TrsyA+52mC057ArY12i0JlfijUI4Fhcpnt-e0rdPAcD32xRAC1BMdAVuSOzW13RAFVLA6QFKm2WaqIX1XTHXv4wMKW7ELlw6B6T-q8VxpaGV2dlTSXXhNphVR73ZjWhXBPfg+b7jPjaIUttZhRRGKZdEeREeEX+YOdPZcXcOB2IFaNK6tKCLjP-LlLxn1ugzmDAEvxDJ63YD2YQ5UAtUBRJuI6ul-u38JRL2Qs7vZ5snVj3soyZhJv164z51mquua1szFR-t-C0IovEFgkFF2I22KGC3QE+A35NZBBusvVPttyUJksYlpqTagPkA5aLGSSLLJ017xICP8kAAeWD39AJA",
"-1", "path", ".domain.com");
setCookie("cookiename1","codenumber=i3NOHiv7DS5gS", "-1", "path",
".domain.com");
setCookie("cookiename2","name=name", "expireTime", "path",
".domain.com");
</script>

I am thinking of using HTTP::cookies

But what about these randomly generated cookie values?? (like:
eKIQUeS7L3meAjS...etc) how am i supposed to reverse engineer that??
Does HTTP::cookies allow values to be set from javascript? I am
doubting it...

I'd GET the initial page, then just scan the text for a regex:
$regex = quotemeta('setCookie("X", "X", "X", "X", "X");');
$regex =~ s/X/([^"]*)/g;
$regex =~ s/ /\\s*/g;
while (@args = ($page =~ /$regex/og)) { ... }

Once you capture the cookies, feed the information to a Perl routine
that you hand-translated from Javascript. Possibly you can get away
with just calling HTTP::Cookies to create them.

 

[robic0]

setCookie("cookiename0","eKIQUeS7L3meAjSRyCChvstAojxX7q3GUduqdSxE9dhylKsekTtVTnOSwawkYsSQEZ7JvnqbfzmUCfeUk-HczPbJDwiXjkWwHhUYVHRnUxCEUKPQlV5BqA8XgE9OHg3A7AUhvVTHqu+Mp2gtbNKvLaIKd8Jb5Aznc5i6s-G2lv8n0nW-wpgl3nkTG59e3Uw5i4mDRhm28KGjh3r-shVXOUZgsbd+Wsrp+r6u+aUmX1TrsyA+52mC057ArY12i0JlfijUI4Fhcpnt-e0rdPAcD32xRAC1BMdAVuSOzW13RAFVLA6QFKm2WaqIX1XTHXv4wMKW7ELlw6B6T-q8VxpaGV2dlTSXXhNphVR73ZjWhXBPfg+b7jPjaIUttZhRRGKZdEeREeEX+YOdPZcXcOB2IFaNK6tKCLjP-LlLxn1ugzmDAEvxDJ63YD2YQ5UAtUBRJuI6ul-u38JRL2Qs7vZ5snVj3soyZhJv164z51mquua1szFR-t-C0IovEFgkFF2I22KGC3QE+A35NZBBusvVPttyUJksYlpqTagPkA5aLGSSLLJ017xICP8kAAeWD39AJA",
"-1", "path", ".domain.com");
setCookie("cookiename1","codenumber=i3NOHiv7DS5gS", "-1", "path",
".domain.com");
setCookie("cookiename2","name=name", "expireTime", "path",
".domain.com");
</script>

I am thinking of using HTTP::cookies

But what about these randomly generated cookie values?? (like:
eKIQUeS7L3meAjS...etc) how am i supposed to reverse engineer that??
Does HTTP::cookies allow values to be set from javascript? I am
doubting it...

I'd GET the initial page, then just scan the text for a regex:
$regex = quotemeta('setCookie("X", "X", "X", "X", "X");');
$regex =~ s/X/([^"]*)/g;
$regex =~ s/ /\\s*/g;
while (@args = ($page =~ /$regex/og)) { ... }

Once you capture the cookies, feed the information to a Perl routine
that you hand-translated from Javascript. Possibly you can get away
with just calling HTTP::Cookies to create them.

You will never ever get sucess from obscure modules. You think cause
its got a "::" in it its gonna be nice and save you from actually having
to think. When it doesen't do for you what you want, your think revolves
around workarounds and "if's" and trying to examine flawed code to
comment 1 line so it will magically work.

The energy and loss of time is unbelievable. Perl modules all just fucking
suck. There's no other way to put it. If I'm totally off base and Cpan "::"
is somehow God syntax, then by all means knock yourself out. I feel sorry
for you.
-robic0-

 

[schruthensis]

I'd GET the initial page, then just scan the text for a regex:
$regex = quotemeta('setCookie("X", "X", "X", "X", "X");');

Once you capture the cookies, feed the information to a Perl routine
that you hand-translated from Javascript. Possibly you can get away
with just calling HTTP::Cookies to create them.

that would be nice if i could get the page at all in the first place...
I am getting 302 (page moved) errors and no HTML to work with.
The only thing i can think of is that there is communication in the
pre-HTML headers that I am not properly handling.

I finally got the ATT scraping proxy to work (not on windows....(a
waste of an entire day), but via an x-window of mozilla from my linux
server to my windows machine.... long story) and it seems to indicate
that cookies are being used extensively and perhaps to help build the
URL.. ...

 

[schruthensis]

robic0 wrote:
You will never ever get sucess from obscure modules. You think cause
its got a "::" in it its gonna be nice and save you from actually having
to think.

That's funny, I've had success with just about every module I've
ever tried. and yes, it has saved some brain time. If nothing else,
I learn from looking at their code. What exactly defines an "obscure"
module anyway? If you have a list of modules that you think I should
steer clear of that might be nice, but blanket statements like these
are hardly useful (except, perhaps to make you feel better about
yourself)

robic0 wrote:
The energy and loss of time is unbelievable. Perl modules all just fucking
suck. There's no other way to put it.

Okay, I can be a (coding) army of one. I'll just write all of this
code from scratch. I was thinking of writing my own operating system
while I'm at it.. and manufacturing my own mother boards and chips. I
was thinking of doing it all blind folded and with one arm tied behind
my back too. Afterwards I thought I might try and re-invent the wheel
a couple of times.

If I'm totally off base and Cpan "::"
is somehow God syntax, then by all means knock yourself out. I feel sorry
for you.

I'm not expecting miracles here, i was merely asking for input and
suggestions on what I might try.... to see if someone has already
wrestled with the problem a bit and bounce ideas off of them.

If you are such a coding genius, perhaps you could do something useful
like help me solve my problem or submit better modules to CPAN.
Instead of verbally abusing a newbie (like me) or a particular faction
of the community (like CPAN) to boost your own jaded ego try doing
something that somebody appreciates.

 

[J. Gleixner]

that would be nice if i could get the page at all in the first place...
I am getting 302 (page moved) errors and no HTML to work with.
The only thing i can think of is that there is communication in the
pre-HTML headers that I am not properly handling.

LWP::UserAgent doesn't handle redirect for a POST, by default. See
perldoc LWP::UserAgent, specifically the requests_redirectable method.

I think I post this answer every other month. Maybe the docs should
make this more prominent or be included in lwpcook.pod, but then
someone would actually have to read them.

 

[schruthensis]

LWP::UserAgent doesn't handle redirect for a POST, by default. See
perldoc LWP::UserAgent, specifically the requests_redirectable method.

I already have that code in place, it just allows me to be redirected
back to the main page. There is more than just redirection going on.

 

[Samwyse]

I already have that code in place, it just allows me to be redirected
back to the main page. There is more than just redirection going on.

Well, if all else fails, you can set up a proxy server and send all of
your browser requests through it. Afterwards, check the log files for
what was going on.

P.S. Setting up a proxy server is outside the scope of this group.

 

[schruthensis]

Well, if all else fails, you can set up a proxy server and send all of
your browser requests through it. Afterwards, check the log files for
what was going on.

P.S. Setting up a proxy server is outside the scope of this group.

I kind of already did that (as mentioned two posts ago). I did just
as Tad McClellan (of the second response to my original posting)
suggested to do: I downloaded, installed, and ran the ATT Web Scraping
Proxy.

I guess i just have to do the hard work of looking through all the
headers, cookies and (sometimes purposefully obfuscated) JavaScript to
figure this out. I was just wondering if anybody had seen this before,
if I was missing something really obvious, or if some sites are just
plain impossible to PERL into because of certain known limitations.

Thanks for all the help anyway

 

[Matt Garrish]

If you are such a coding genius, perhaps you could do something useful
like help me solve my problem or submit better modules to CPAN.
Instead of verbally abusing a newbie (like me) or a particular faction
of the community (like CPAN) to boost your own jaded ego try doing
something that somebody appreciates.

Trust me, you *don't* want him to help you with your problem; it'll just put
you in a worse position. I think the general perception is he's a bored kid
who's acting out on usenet to get attention (classic troll). The more you
give him that attention the more it keeps him coming back.

Matt