help with LWP and log in after redirect

[eatmoreoats]

I'm trying to automate the log in to a website, which I've done before
successfully with lwp. However in this case I'm having problems and
need some help. After posting the username/password, a redirect
occurs to a new url where the username and password have been encoded.
With post redirects enabled, I end up back at the log in page, rather
than following the redirect.

Here are the request and response headers, as observed with Firefox
plugins (HTTP LIve Headers, and Tamper Data). These are observed
during a successful login with my firefox browser.

REQUEST HEADER

Host=www.mywebsite.com
User-Agent=Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:
1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
Accept=text/xml,application/xml,application/xhtml+xml,text/
html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language=en-us,en;q=0.5
Accept-Encoding=gzip,deflate
Accept-Charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive=300
Connection=keep-alive
Referer=http://www.mywebsite.com/newmywebsite/logon.aspx
Cookie=MyCookie=; mysrv=abc;
JSESSIONID=DC638726E6831E7D6EEE996823FEECA0; JOSSO_SESSIONID=-
Content-Type=application/x-www-form-urlencoded
Content-Length=171
POSTDATA=username=theusername&password=thepassword


RESPONSE HEADER

Status=Moved Temporarily - 302
Server=Apache-Coyote/1.1
X-Powered-By=Servlet 2.4; JBoss-4.0.2 (build: CVSTag=JBoss_4_0_2
date=200505022023)/Tomcat-5.5
X-Node-Name=abc
Set-Cookie=SERVER-X=abc; Domain=.website.com; Expires=Tue, 04-Mar-2008
15:03:03 GMT; Path=/
NewMywebsite=null; Expires=Thu, 01-Jan-1970 00:00:10 GMT
Location=http://www.mywebsite.com/newmywebsite/somenewurl/
somenewscript.wotever?
cmd=login&username=YWR2YW5jZWludGVybmV0&password=c2FuaXR5
Content-Type=text/html
Content-Length=0
Date=Tue, 04 Mar 2008 14:03:03 GMT

Notice the Location in the response - if I take that url and do a get
request on it, I'm logged in. However, I can't seem to get the code to
follow the redirect down this url path. If I disable redirects, and
grab the Location header value, it is Location=http://
www.mywebsite.com/newmywebsite/somenewurl/somenewscript.wotever?cmd=login&username=null.
I'd like the script to just follow the redirect to the new location
but its not clear why thats not working. Any ideas how to make this
work ?

Here is the code example :

use HTTP::Cookies;
use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$cookies = new HTTP::Cookies();
$ua->cookie_jar($cookies);
$ua->timeout(300);
$ua->requests_redirectable (['GET', 'HEAD', 'POST']);

# step 1 - hit the login page first
$response = $ua->get('http://www.mywebsite.com/newmywebsite/
logon.aspx');

# step 2 - post in the username/password
$response = $ua->post('http://www.mywebsite.com/newmywebsite/
logonvalidate.aspx',
[
username=>"theusername",
password=>"thepassword",
]
);
if ($response->is_success) { print $response->content; } else { print
$response->status_line; }

# the content at this point is the same as the logon.aspx page in step
1 - why is the redirect not working here ?

Thanks
-emo

 

[Ben Morrow]

I'm trying to automate the log in to a website, which I've done before
successfully with lwp. However in this case I'm having problems and
need some help. After posting the username/password, a redirect
occurs to a new url where the username and password have been encoded.
With post redirects enabled, I end up back at the log in page, rather
than following the redirect.

Here are the request and response headers, as observed with Firefox
plugins (HTTP LIve Headers, and Tamper Data). These are observed
during a successful login with my firefox browser.

REQUEST HEADER

Referer=http://www.mywebsite.com/newmywebsite/logon.aspx

RESPONSE HEADER

Status=Moved Temporarily - 302

Location=http://www.mywebsite.com/newmywebsite/somenewurl/
somenewscript.wotever?
cmd=login&username=YWR2YW5jZWludGVybmV0&password=c2FuaXR5

Notice the Location in the response - if I take that url and do a get
request on it, I'm logged in. However, I can't seem to get the code to
follow the redirect down this url path. If I disable redirects, and
grab the Location header value, it is Location=http://
www.mywebsite.com/newmywebsite/somenewurl/somenewscript.wotever?
cmd=login&username=null.

So whatever LWP is sending, the server doesn't like it and doesn't
accept it as a valid logon.

I'd like the script to just follow the redirect to the new location
but its not clear why thats not working. Any ideas how to make this
work ?

Here is the code example :

use HTTP::Cookies;
use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$cookies = new HTTP::Cookies();
$ua->cookie_jar($cookies);
$ua->timeout(300);
$ua->requests_redirectable (['GET', 'HEAD', 'POST']);

You shouldn't need this. requests_redirectable refers to the method used
to follow the redirect, not the method the redirect is a reply to; and
LWP treats 302 as 'GET this URL' rather than 'repeat the previous
request with a different URL', in line with the behaviour of most (all?)
actual browsers (this is strictly against the letter of the standard,
but RFC2616 recognises that this is the actual implemented behaviour).

# step 1 - hit the login page first
$response = $ua->get('http://www.mywebsite.com/newmywebsite/
logon.aspx');

# step 2 - post in the username/password
$response = $ua->post('http://www.mywebsite.com/newmywebsite/
logonvalidate.aspx',
[
username=>"theusername",
password=>"thepassword",
]
);

This will not set the Referer header. I would recommend using
WWW::Mechanize instead, which tries as hard as possible to emulate an
actual browser.

Ben

 

[eatmoreoats]

Quotheatmoreoats<[email protected]>:

I'm trying to automate the log in to a website, which I've done before
successfully with lwp. However in this case I'm having problems and
need some help. After posting the username/password, a redirect
occurs to a new url where the username and password have been encoded.
With post redirects enabled, I end up back at the log in page, rather
than following the redirect.

Here are the request and response headers, as observed with Firefox
plugins (HTTP LIve Headers, and Tamper Data). These are observed
during a successful login with my firefox browser.

REQUEST HEADER

Referer=http://www.mywebsite.com/newmywebsite/logon.aspx

RESPONSE HEADER

Status=Moved Temporarily - 302

Location=http://www.mywebsite.com/newmywebsite/somenewurl/
somenewscript.wotever?
cmd=login&username=YWR2YW5jZWludGVybmV0&password=c2FuaXR5

Notice the Location in the response - if I take that url and do a get
request on it, I'm logged in. However, I can't seem to get the code to
follow the redirect down this url path. If I disable redirects, and
grab the Location header value, it is Location=http://
www.mywebsite.com/newmywebsite/somenewurl/somenewscript.wotever?
cmd=login&username=null.

So whatever LWP is sending, the server doesn't like it and doesn't
accept it as a valid logon.

I'd like the script to just follow the redirect to the new location
but its not clear why thats not working. Any ideas how to make this
work ?

Here is the code example :

use HTTP::Cookies;
use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$cookies = new HTTP::Cookies();
$ua->cookie_jar($cookies);
$ua->timeout(300);
$ua->requests_redirectable (['GET', 'HEAD', 'POST']);

You shouldn't need this. requests_redirectable refers to the method used
to follow the redirect, not the method the redirect is a reply to; and
LWP treats 302 as 'GET this URL' rather than 'repeat the previous
request with a different URL', in line with the behaviour of most (all?)
actual browsers (this is strictly against the letter of the standard,
but RFC2616 recognises that this is the actual implemented behaviour).

# step 1 - hit the login page first
$response = $ua->get('http://www.mywebsite.com/newmywebsite/
logon.aspx');

# step 2 - post in the username/password
$response = $ua->post('http://www.mywebsite.com/newmywebsite/
logonvalidate.aspx',
[
username=>"theusername",
password=>"thepassword",
]
);

This will not set the Referer header. I would recommend using
WWW::Mechanize instead, which tries as hard as possible to emulate an
actual browser.

Ben

Hi Ben,
Thanks for the suggestion and I will give it a try. However, I did
actually try setting all of the other header key-value pairs in the
post request but without success.
Dom