LWP::UserAgent and SSL is it impossible?

[Paul Lemmons]

I have spent most of the day trying to get this to work. I have visited
hundreds of web pages and it appears to be a common frustration but
nobody seems to be posting any solutions I am beginning to wonder if it
is impossible.

I want to script a login to a secure web site in perl. The web site is
using session cookies and SSL. It is not using basic authentication. No
matter what I try I always get "Authentication failed!" returned from
the server.

If I understand correctly how this should work, I would get a successful
response from this request along with a cookie that I would use for all
subsequent requests. I am not getting that cookie because I am not
getting success.

If this truly is impossible, please, someone say so so I can quit
beating my head against this. If you have actually gotten this to work I
would be thrilled if you could share your solution!

Thank you!!

===============================================
Here is my code. It is fairly straight forward:
===============================================

#!/usr/local/bin/perl
use HTTP::Cookies;
use HTTP::Request;
use HTTP::Request::Common;
use LWP::UserAgent;

$cookie_jar = new HTTP::Cookies(ignore_discard => TRUE);


$ua = new LWP::UserAgent;
$ua->agent('Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5)
Gecko/20031007 Firebird/0.7');
$ua->cookie_jar($cookie_jar);

$content =
join('&','Username=myuserid','Password=mypassword','URL=%2Fdefault.asp');

$req = new HTTP::Request 'POST' =>
'https://xxx.yyy.com/_mem_bin/verifpwd.asp';
$req->header('Host' => 'xxx.yyy.com');
$req->header('Keep-Alive' => '300');
$req->header('Connection' => 'Keep-Alive');
$req->header('Accept' =>
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1');
$req->header('Accept-Language' => 'en-us,en;q=0.5');
$req->header('Accept-Encoding' => 'gzip,deflate');
$req->header('Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7');
$req->header('Referer' =>
'https://xxx.yyy.com/public/login.asp?/default.asp');
$req->header('Content-Type' => 'application/x-www-form-urlencoded');
$req->header('Content' => $content);

print "-----------------------------------------------------\n";
print "HTTP::Request\n";
print "-----------------------------------------------------\n";
print $req->as_string();

$res = $ua->request($req);

$cookie_jar->extract_cookies($res);
$cookie_jar->save("cookie_jar");
print "-----------------------------------------------------\n";
print "Cookie after UA Request\n";
print "-----------------------------------------------------\n";
print $cookie_jar->as_string();

if ($res->is_success)
{
print "-----------------------------------------------------\n";
print "UA->Request\n";
print "-----------------------------------------------------\n";
print $res->request->as_string();
print "-----------------------------------------------------\n";
print "Contents\n";
print "-----------------------------------------------------\n";
print $res->content;
print "\n-----------------------------------------------------\n";
print "Done\n";
print "-----------------------------------------------------\n\n\n";
}
else
{
print $res->error_as_HTML;
}



===============================================
The output of the program is:
===============================================


-----------------------------------------------------
HTTP::Request
-----------------------------------------------------
POST https://xxx.yyy.com/_mem_bin/verifpwd.asp
Connection: Keep-Alive
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Host: xxx.yyy.com
Referer: https://xxx.yyy.com/public/login.asp?/default.asp
Content-Type: application/x-www-form-urlencoded
Content: Username=myuserid&Password=mypassword&URL=%2Fdefault.asp
Keep-Alive: 300


-----------------------------------------------------
Cookie after UA Request
-----------------------------------------------------
Set-Cookie3: ASPSESSIONIDASBBDDQA=JKCCBIMAFOFOBNLMJLHOLFAL; path="/";
domain=xxx.yyy.com; path_spec; discard; version=0
Set-Cookie3: TriedToLogin=1; path="/"; domain=xxx.yyy.com; path_spec;
discard; version=0
-----------------------------------------------------
UA->request
-----------------------------------------------------
POST https://xxx.yyy.com/_mem_bin/verifpwd.asp
Connection: Keep-Alive
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Host: xxx.yyy.com
Referer: https://xxx.yyy.com/public/login.asp?/default.asp
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5)
Gecko/20031007 Firebird/0.7
Content-Type: application/x-www-form-urlencoded
Content: Username=myuserid&Password=mypassword&URL=%2Fdefault.asp
Keep-Alive: 300

 

[Helgi Briem]

I want to script a login to a secure web site in perl. The web site is
using session cookies and SSL. It is not using basic authentication. No
matter what I try I always get "Authentication failed!" returned from
the server.

Install the Crypt::SSLeay module, then add the line

use Crypt::SSLeay;

to your program.

 

[ko]

I have spent most of the day trying to get this to work. I have visited
hundreds of web pages and it appears to be a common frustration but
nobody seems to be posting any solutions I am beginning to wonder if it
is impossible.

I want to script a login to a secure web site in perl. The web site is
using session cookies and SSL. It is not using basic authentication. No
matter what I try I always get "Authentication failed!" returned from
the server.

If I understand correctly how this should work, I would get a successful
response from this request along with a cookie that I would use for all
subsequent requests. I am not getting that cookie because I am not
getting success.

If this truly is impossible, please, someone say so so I can quit
beating my head against this. If you have actually gotten this to work I
would be thrilled if you could share your solution!

Thank you!!

[snip code]

I'm not sure if this will solve your problem, but since you're using the
as_string() method for debugging anyway, you might as well try the
previous() method too. As documented in HTTP::Response:

==DOCUMENTATION
$r->previous([$previousResponse])

....

The previous attribute is used to link together chains of responses. You
get chains of responses if the first response is redirect or unauthorized.
==END QUOTE

I would guess that since you're getting an 'Authentication failed' page,
you might also be getting a redirect/unauthorized response before that.
(the user agent automatically follows redirects) So something like the
following should give you the full chain of responses (in reverse
order), which might help you figure out what's missing:

*untested*

## same as your code - LAST response
print join("\n", "STATUS LINE:", $res->status_line), "\n\n";
print join("\n", "REQUEST:", $res->as_string), "\n\n";

## this will cover any response due to redirect or unauthorized
{
my $res = $res;
while (my $previous = $res->previous) {
print join("\n", "STATUS LINE:", $previous->status_line), "\n\n";
print join("\n", "REQUEST:", $previous->as_string), "\n\n";
$res = $previous;
}
}

use strict;
use warnings;

might also help

HTH - keith

 

[Peter Scott]

Install the Crypt::SSLeay module, then add the line

use Crypt::SSLeay;

You don't need to add that line to make HTTP over SSL work;
installing the module makes LWP work with the https schema
transparently. If the OP's problem was that he didn't have
Crypt::SSLeay installled, LWP would have told him so on
the request().

Perhaps there is a character in the password that should
undergo encoding. The poster would be better off sending
the content using, say, HTTP::Request::Common:OST
which relieves the user of having to worry about encoding.

 

[Pierre Asselin]

I want to script a login to a secure web site in perl. The web site is
using session cookies and SSL. It is not using basic authentication. No
matter what I try I always get "Authentication failed!" returned from
the server.

[ ... ]

$req = new HTTP::Request 'POST' =>
'https://xxx.yyy.com/_mem_bin/verifpwd.asp';

[ ... ]

$req->header('Content-Type' => 'application/x-www-form-urlencoded');
$req->header('Content' => $content);

My HTTP is rusty but, are you sure about this 'Content: ' header?
I thought the form data had to go in the request body, after the headers
and separated from them by a blank line. So,

$req->content($content);

You could also use the convenience functions in HTTP::Request::Common .