Perl - approach to query https website for code & parsing xmlresponse

D

deepalicanada

I am new to perl web programming. I have done perl scripting for unix
but none on webside.So not sure about what modules should i use to
query https website and parse xml response?

Problem - Trying to get a code from a https website
e.g. https://www.mytargetwebsite.com/servlet/Getcode.SendRequest?q=<QUERYNO>

Here <QUERYNO> is constant.

The website returns following response:

<Response timestamp="2 July, 2008 12:10 AM EDT">
<code> 9999999 </code>
</Response>

I can do this query manually through browser but when I tried taking a
shot through LWP, i got error that "Your browser sent a request that
this server could not understand". I'm not sure whether i am using
correct module/approach to solve this problem so help would be
appreciated.

Thanks,
Deep
 
Z

Zak B. Elep

I can do this query manually through browser but when I tried taking a
shot through LWP, i got error that "Your browser sent a request that
this server could not understand". I'm not sure whether i am using
correct module/approach to solve this problem so help would be
appreciated.
Click to expand...

Just how exactly did you do the call through LWP? Did you use
LWP::UserAgent to request the XML? If so, you probably want to do this:

my $ua = LWP::UserAgent->new();
my $res = $ua->post( "https://target/s/endpoint", q => 'query-number' );

I also noticed that you're requesting via HTTPS. Do you also have the
corresponding module for SSL installed (Crypt::SSLeay)
 
D

deep

I have the module installed. Here's the code with error.

use LWP::Debug qw(+);
use LWP::UserAgent;
use Crypt::SSLeay;
use URI;

$ENV{HTTPS_DEBUG} = 1;

my $url = URI->new('https://www.mytargetwebsite.com/servlet/
Getcode.SendRequest');
my $queryno = 999999;
$url->query_form('q' => $queryno);
print "\n $url \n";

my $ua = new LWP::UserAgent;


# Create a request
my $req = new HTTP::Request GET => $url;

# Pass request to the user agent and get a response back
my $res= $ua->request($req);

print "\nResult - $res->content";

print "\n Error - $res->status_line";




LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: POST <deleted actual website> -
https://www.mytargetwebsite.com/servlet/Getcode.SendRequest&q=999999
LWP::UserAgent::_need_proxy: Not proxied
LWP::protocol::http::request: ()
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
LWP::protocol::collect: read 208 bytes
SSL3 alert read:warning:close notify
LWP::UserAgent::request: Simple response: Bad Request
 
Z

Zak B. Elep

deep said:
# Create a request
my $req = new HTTP::Request GET => $url;
Click to expand...

The above causes the succeeding $ua->request() to connect correctly to
your server via HTTPS but feeding it with a `POST
https://site/request/script', which is not what you expect. I guess
this is taken from the Crypt::SSLeay documentation; I tried this myself
and reproduced the same bad request, even with GET. :(

Why make the HTTP::Request when LWP::UserAgent can do it for you via
$ua->post()?
 
D

deep

I tried that but got same results. I used following:

my $ua = new LWP::UserAgent;
my $res = $ua->post ($url);
print $res->as_string;
print "\nResult - $res->content";
print "\n Error - $res->status_line";


I got following output.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>200 OK</title>
</head><body>
<h1>OK</h1>
<p>Your browser sent a request that this server could not
understand.<br />
</p>
</body></html>

Result - HTTP::Response=HASH(0x1e93824)->content
Error - HTTP::Response=HASH(0x1e93824)->status_line
 
Z

Zak B. Elep

deep said:
I tried that but got same results. I used following:

my $ua = new LWP::UserAgent;
my $res = $ua->post ($url);
print $res->as_string;
print "\nResult - $res->content";
print "\n Error - $res->status_line";
Click to expand...

Here's my attempt against a live site (sourceforge), it works:

alteran:~ zakame$ perl -de 0 -MLWP::UserAgent

Loading DB routines from perl5db.pl version 1.28
Editor support available.

,----[ perl -de 0 -MLWP::UserAgent ]
| Enter h or `h h' for help, or `man perldebug' for more help.
|
| main::(-e:1): 0
| DB<1> $ua = LWP::UserAgent->new()
|
| DB<2> $url = 'https://sourceforge.net/forum/forum.php'
|
| DB<3> $res = $ua->post( $url, forum_id => '849067' )
|
| DB<4> x $res->as_string()
| 0 "HTTP/1.1 200 OK\cJConnection: close..."
|
| DB<5> x $res->status_line()
| 0 '200 OK'
`----
 
J

J. Gleixner

deep said:
I tried that but got same results. I used following:
Click to expand...
You tried what?
my $ua = new LWP::UserAgent;
my $res = $ua->post ($url);
Click to expand...

What's the URL?
print $res->as_string;
print "\nResult - $res->content";
print "\n Error - $res->status_line";
Click to expand...

Of course, that's not how you'd actually see the value of those methods,
which is why you're getting this wonderful output.
Result - HTTP::Response=HASH(0x1e93824)->content
Error - HTTP::Response=HASH(0x1e93824)->status_line
Click to expand...
print "Result - ", $res->content, "\n";
print "Error - ", $res->status_line, "\n";
Click to expand...

I have no idea what this has to do with 'parsing xml response', which
is in your subject.

Do you have Crypt::SSLeay installed?

You're sure you want to do a POST? You're not 'post'ing anything.
 
D

deep

On Jul 24, 1:20 pm, "J. Gleixner" <[email protected]>
wrote:
You tried what?
What's the URL?
Of course, that's not how you'd actually see the value of those
methods,which is why you're getting this wonderful output.
I have no idea what this has to do with 'parsing xml response', which
is in your subject.
Do you have Crypt::SSLeay installed?
You're sure you want to do a POST?  You're not 'post'ing anything.
 
Z

Zak B. Elep

Reposting back to clpmisc, it'll probably help someone else. Site
changed to preserve anonymity ;)

dc said:
When I run above through browser, I get following XML response:
- <BrokenResponse timestamp="25 July, 2012 01:16 PM EDT">
<code>Invalid request</code>
<url>https://www.brokensoapsite.example/</url>
</BrokenResponse>

But through code it gives me bad request instead of above XML response.
Click to expand...

I realized that what you're trying to do is to POST data to what looks
like a Web Service (in particular, a SOAP/XMLRPC-like service, except it
doesn't make an envelope;) we're using the wrong tool for the job.

I suggest looking at SOAP::Lite[1]. Here's a sample session to get you
started:

,----[ perl -de 0 -MSOAP::Lite ]
|
| Loading DB routines from perl5db.pl version 1.28
| Editor support available.
|
| Enter h or `h h' for help, or `man perldebug' for more help.
|
| main::(-e:1): 0
| DB<1> $url = 'https://www.brokensoapsite.example/borked.SendRequest'
|
| DB<2> $x = SOAP::Data->name( 'x' )->value( '11223344556677889900123' )
|
| DB<3> $y = SOAP::Data->name( 'y' )->value( '999999' )
|
| DB<4> $res = SOAP::Lite->new( proxy => $url )->call( $x, $y )
|
| DB<5> x $res->dataof('//BrokenResponse)->attr->{'timestamp'}
| 0 '26 July, 2012 08:10 AM EDT'
|
| DB<6> x $res->valueof('//BrokenResponse/code')
| 0 'Invalid request'
|
| DB<7> x $res->valueof('//BrokenResponse/url')
| 0 'https://www.brokensoapsite.example/'
`----


Footnotes:
[1] http://search.cpan.org/perldoc?SOAP::Lite
 
D

deep

For some reason my reply didn't get posted, so trying agin.

Hi Zak

That indeed worked. Thanks for your help in identifying fundamental
issue with the use of wrong module. It solved the problem.

regards,
Deep
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Help Changing Code for a Wordpress Website Menu Color 0
Post to https using perl script 7
Automating access to a HTTPS website 2
automate website login using perl 12
converting Java code to Perl (using LWP?) 4
Logging into and parsing a website using Perl 3
Best approach to locking down a single file on a public website? 1
optimize XML parsing 2

Members online

No members online now.
Total: 173 (members: 2, guests: 171)
Robots: 314

Forum statistics

Threads
474,262
Messages
2,571,049
Members
48,769
Latest member
Clifft

Latest Threads

Top