help with LWP and log in after redirect

Discussion in 'Perl Misc' started by eatmoreoats, Mar 4, 2008.

  1. eatmoreoats

    eatmoreoats Guest

    I'm trying to automate the log in to a website, which I've done before
    successfully with lwp. However in this case I'm having problems and
    need some help. After posting the username/password, a redirect
    occurs to a new url where the username and password have been encoded.
    With post redirects enabled, I end up back at the log in page, rather
    than following the redirect.

    Here are the request and response headers, as observed with Firefox
    plugins (HTTP LIve Headers, and Tamper Data). These are observed
    during a successful login with my firefox browser.

    REQUEST HEADER

    Host=www.mywebsite.com
    User-Agent=Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:
    1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
    Accept=text/xml,application/xml,application/xhtml+xml,text/
    html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    Accept-Language=en-us,en;q=0.5
    Accept-Encoding=gzip,deflate
    Accept-Charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Keep-Alive=300
    Connection=keep-alive
    Referer=http://www.mywebsite.com/newmywebsite/logon.aspx
    Cookie=MyCookie=; mysrv=abc;
    JSESSIONID=DC638726E6831E7D6EEE996823FEECA0; JOSSO_SESSIONID=-
    Content-Type=application/x-www-form-urlencoded
    Content-Length=171
    POSTDATA=username=theusername&password=thepassword


    RESPONSE HEADER

    Status=Moved Temporarily - 302
    Server=Apache-Coyote/1.1
    X-Powered-By=Servlet 2.4; JBoss-4.0.2 (build: CVSTag=JBoss_4_0_2
    date=200505022023)/Tomcat-5.5
    X-Node-Name=abc
    Set-Cookie=SERVER-X=abc; Domain=.website.com; Expires=Tue, 04-Mar-2008
    15:03:03 GMT; Path=/
    NewMywebsite=null; Expires=Thu, 01-Jan-1970 00:00:10 GMT
    Location=http://www.mywebsite.com/newmywebsite/somenewurl/
    somenewscript.wotever?
    cmd=login&username=YWR2YW5jZWludGVybmV0&password=c2FuaXR5
    Content-Type=text/html
    Content-Length=0
    Date=Tue, 04 Mar 2008 14:03:03 GMT

    Notice the Location in the response - if I take that url and do a get
    request on it, I'm logged in. However, I can't seem to get the code to
    follow the redirect down this url path. If I disable redirects, and
    grab the Location header value, it is Location=http://
    www.mywebsite.com/newmywebsite/somenewurl/somenewscript.wotever?cmd=login&username=null.
    I'd like the script to just follow the redirect to the new location
    but its not clear why thats not working. Any ideas how to make this
    work ?

    Here is the code example :

    use HTTP::Cookies;
    use LWP::UserAgent;

    $ua = LWP::UserAgent->new;
    $cookies = new HTTP::Cookies();
    $ua->cookie_jar($cookies);
    $ua->timeout(300);
    $ua->requests_redirectable (['GET', 'HEAD', 'POST']);

    # step 1 - hit the login page first
    $response = $ua->get('http://www.mywebsite.com/newmywebsite/
    logon.aspx');

    # step 2 - post in the username/password
    $response = $ua->post('http://www.mywebsite.com/newmywebsite/
    logonvalidate.aspx',
    [
    username=>"theusername",
    password=>"thepassword",
    ]
    );
    if ($response->is_success) { print $response->content; } else { print
    $response->status_line; }

    # the content at this point is the same as the logon.aspx page in step
    1 - why is the redirect not working here ?

    Thanks
    -emo
    eatmoreoats, Mar 4, 2008
    #1
    1. Advertising

  2. eatmoreoats

    Ben Morrow Guest

    Quoth eatmoreoats <>:
    > I'm trying to automate the log in to a website, which I've done before
    > successfully with lwp. However in this case I'm having problems and
    > need some help. After posting the username/password, a redirect
    > occurs to a new url where the username and password have been encoded.
    > With post redirects enabled, I end up back at the log in page, rather
    > than following the redirect.
    >
    > Here are the request and response headers, as observed with Firefox
    > plugins (HTTP LIve Headers, and Tamper Data). These are observed
    > during a successful login with my firefox browser.
    >
    > REQUEST HEADER
    >

    <snip>
    > Referer=http://www.mywebsite.com/newmywebsite/logon.aspx

    <snip>
    >
    > RESPONSE HEADER
    >
    > Status=Moved Temporarily - 302

    <snip>
    > Location=http://www.mywebsite.com/newmywebsite/somenewurl/
    > somenewscript.wotever?
    > cmd=login&username=YWR2YW5jZWludGVybmV0&password=c2FuaXR5

    <snip>
    >
    > Notice the Location in the response - if I take that url and do a get
    > request on it, I'm logged in. However, I can't seem to get the code to
    > follow the redirect down this url path. If I disable redirects, and
    > grab the Location header value, it is Location=http://
    > www.mywebsite.com/newmywebsite/somenewurl/somenewscript.wotever?
    > cmd=login&username=null.


    So whatever LWP is sending, the server doesn't like it and doesn't
    accept it as a valid logon.

    > I'd like the script to just follow the redirect to the new location
    > but its not clear why thats not working. Any ideas how to make this
    > work ?
    >
    > Here is the code example :
    >
    > use HTTP::Cookies;
    > use LWP::UserAgent;
    >
    > $ua = LWP::UserAgent->new;
    > $cookies = new HTTP::Cookies();
    > $ua->cookie_jar($cookies);
    > $ua->timeout(300);
    > $ua->requests_redirectable (['GET', 'HEAD', 'POST']);


    You shouldn't need this. requests_redirectable refers to the method used
    to follow the redirect, not the method the redirect is a reply to; and
    LWP treats 302 as 'GET this URL' rather than 'repeat the previous
    request with a different URL', in line with the behaviour of most (all?)
    actual browsers (this is strictly against the letter of the standard,
    but RFC2616 recognises that this is the actual implemented behaviour).

    > # step 1 - hit the login page first
    > $response = $ua->get('http://www.mywebsite.com/newmywebsite/
    > logon.aspx');
    >
    > # step 2 - post in the username/password
    > $response = $ua->post('http://www.mywebsite.com/newmywebsite/
    > logonvalidate.aspx',
    > [
    > username=>"theusername",
    > password=>"thepassword",
    > ]
    > );


    This will not set the Referer header. I would recommend using
    WWW::Mechanize instead, which tries as hard as possible to emulate an
    actual browser.

    Ben
    Ben Morrow, Mar 4, 2008
    #2
    1. Advertising

  3. eatmoreoats

    eatmoreoats Guest

    On Mar 4, 11:05 am, Ben Morrow <> wrote:
    > Quotheatmoreoats<>:
    >
    >
    >
    > > I'm trying to automate the log in to a website, which I've done before
    > > successfully with lwp. However in this case I'm having problems and
    > > need some help. After posting the username/password, a redirect
    > > occurs to a new url where the username and password have been encoded.
    > > With post redirects enabled, I end up back at the log in page, rather
    > > than following the redirect.

    >
    > > Here are the request and response headers, as observed with Firefox
    > > plugins (HTTP LIve Headers, and Tamper Data). These are observed
    > > during a successful login with my firefox browser.

    >
    > > REQUEST HEADER

    >
    > <snip>
    > > Referer=http://www.mywebsite.com/newmywebsite/logon.aspx

    > <snip>
    >
    > > RESPONSE HEADER

    >
    > > Status=Moved Temporarily - 302

    > <snip>
    > > Location=http://www.mywebsite.com/newmywebsite/somenewurl/
    > > somenewscript.wotever?
    > > cmd=login&username=YWR2YW5jZWludGVybmV0&password=c2FuaXR5

    > <snip>
    >
    > > Notice the Location in the response - if I take that url and do a get
    > > request on it, I'm logged in. However, I can't seem to get the code to
    > > follow the redirect down this url path. If I disable redirects, and
    > > grab the Location header value, it is Location=http://
    > >www.mywebsite.com/newmywebsite/somenewurl/somenewscript.wotever?
    > > cmd=login&username=null.

    >
    > So whatever LWP is sending, the server doesn't like it and doesn't
    > accept it as a valid logon.
    >
    > > I'd like the script to just follow the redirect to the new location
    > > but its not clear why thats not working. Any ideas how to make this
    > > work ?

    >
    > > Here is the code example :

    >
    > > use HTTP::Cookies;
    > > use LWP::UserAgent;

    >
    > > $ua = LWP::UserAgent->new;
    > > $cookies = new HTTP::Cookies();
    > > $ua->cookie_jar($cookies);
    > > $ua->timeout(300);
    > > $ua->requests_redirectable (['GET', 'HEAD', 'POST']);

    >
    > You shouldn't need this. requests_redirectable refers to the method used
    > to follow the redirect, not the method the redirect is a reply to; and
    > LWP treats 302 as 'GET this URL' rather than 'repeat the previous
    > request with a different URL', in line with the behaviour of most (all?)
    > actual browsers (this is strictly against the letter of the standard,
    > but RFC2616 recognises that this is the actual implemented behaviour).
    >
    > > # step 1 - hit the login page first
    > > $response = $ua->get('http://www.mywebsite.com/newmywebsite/
    > > logon.aspx');

    >
    > > # step 2 - post in the username/password
    > > $response = $ua->post('http://www.mywebsite.com/newmywebsite/
    > > logonvalidate.aspx',
    > > [
    > > username=>"theusername",
    > > password=>"thepassword",
    > > ]
    > > );

    >
    > This will not set the Referer header. I would recommend using
    > WWW::Mechanize instead, which tries as hard as possible to emulate an
    > actual browser.
    >
    > Ben


    Hi Ben,
    Thanks for the suggestion and I will give it a try. However, I did
    actually try setting all of the other header key-value pairs in the
    post request but without success.
    Dom
    eatmoreoats, Mar 4, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. La Jesus
    Replies:
    9
    Views:
    1,313
    Gunnar Hjalmarsson
    Oct 27, 2003
  2. Henrik_the_boss
    Replies:
    0
    Views:
    2,647
    Henrik_the_boss
    Nov 5, 2003
  3. AAaron123
    Replies:
    1
    Views:
    274
    Cowboy \(Gregory A. Beamer\)
    Oct 14, 2008
  4. Sal
    Replies:
    1
    Views:
    406
  5. Replies:
    8
    Views:
    126
    Paul Lalli
    Oct 24, 2005
Loading...

Share This Page