Perl - approach to query https website for code & parsing xmlresponse

Discussion in 'Perl Misc' started by deepalicanada@gmail.com, Jul 24, 2008.

  1. Guest

    I am new to perl web programming. I have done perl scripting for unix
    but none on webside.So not sure about what modules should i use to
    query https website and parse xml response?

    Problem - Trying to get a code from a https website
    e.g. https://www.mytargetwebsite.com/servlet/Getcode.SendRequest?q=<QUERYNO>

    Here <QUERYNO> is constant.

    The website returns following response:

    <Response timestamp="2 July, 2008 12:10 AM EDT">
    <code> 9999999 </code>
    </Response>

    I can do this query manually through browser but when I tried taking a
    shot through LWP, i got error that "Your browser sent a request that
    this server could not understand". I'm not sure whether i am using
    correct module/approach to solve this problem so help would be
    appreciated.

    Thanks,
    Deep
     
    , Jul 24, 2008
    #1
    1. Advertising

  2. Zak B. Elep Guest

    Re: Perl - approach to query https website for code & parsing xml response

    writes:

    > I can do this query manually through browser but when I tried taking a
    > shot through LWP, i got error that "Your browser sent a request that
    > this server could not understand". I'm not sure whether i am using
    > correct module/approach to solve this problem so help would be
    > appreciated.


    Just how exactly did you do the call through LWP? Did you use
    LWP::UserAgent to request the XML? If so, you probably want to do this:

    my $ua = LWP::UserAgent->new();
    my $res = $ua->post( "https://target/s/endpoint", q => 'query-number' );

    I also noticed that you're requesting via HTTPS. Do you also have the
    corresponding module for SSL installed (Crypt::SSLeay)

    --
    Zak B. Elep
    http://blog.zakame.net
     
    Zak B. Elep, Jul 24, 2008
    #2
    1. Advertising

  3. deep Guest

    I have the module installed. Here's the code with error.

    use LWP::Debug qw(+);
    use LWP::UserAgent;
    use Crypt::SSLeay;
    use URI;

    $ENV{HTTPS_DEBUG} = 1;

    my $url = URI->new('https://www.mytargetwebsite.com/servlet/
    Getcode.SendRequest');
    my $queryno = 999999;
    $url->query_form('q' => $queryno);
    print "\n $url \n";

    my $ua = new LWP::UserAgent;


    # Create a request
    my $req = new HTTP::Request GET => $url;

    # Pass request to the user agent and get a response back
    my $res= $ua->request($req);

    print "\nResult - $res->content";

    print "\n Error - $res->status_line";




    LWP::UserAgent::new: ()
    LWP::UserAgent::request: ()
    LWP::UserAgent::send_request: POST <deleted actual website> -
    https://www.mytargetwebsite.com/servlet/Getcode.SendRequest&q=999999
    LWP::UserAgent::_need_proxy: Not proxied
    LWP::protocol::http::request: ()
    SSL_connect:before/connect initialization
    SSL_connect:SSLv2/v3 write client hello A
    SSL_connect:SSLv3 read server hello A
    SSL_connect:SSLv3 read server certificate A
    SSL_connect:SSLv3 read server key exchange A
    SSL_connect:SSLv3 read server done A
    SSL_connect:SSLv3 write client key exchange A
    SSL_connect:SSLv3 write change cipher spec A
    SSL_connect:SSLv3 write finished A
    SSL_connect:SSLv3 flush data
    SSL_connect:SSLv3 read finished A
    LWP::protocol::collect: read 208 bytes
    SSL3 alert read:warning:close notify
    LWP::UserAgent::request: Simple response: Bad Request
     
    deep, Jul 24, 2008
    #3
  4. Zak B. Elep Guest

    Re: Perl - approach to query https website for code & parsing xml response

    deep <> writes:

    > # Create a request
    > my $req = new HTTP::Request GET => $url;


    The above causes the succeeding $ua->request() to connect correctly to
    your server via HTTPS but feeding it with a `POST
    https://site/request/script', which is not what you expect. I guess
    this is taken from the Crypt::SSLeay documentation; I tried this myself
    and reproduced the same bad request, even with GET. :(

    Why make the HTTP::Request when LWP::UserAgent can do it for you via
    $ua->post()?

    --
    Zak B. Elep
    http://blog.zakame.net
     
    Zak B. Elep, Jul 24, 2008
    #4
  5. deep Guest

    I tried that but got same results. I used following:

    my $ua = new LWP::UserAgent;
    my $res = $ua->post ($url);
    print $res->as_string;
    print "\nResult - $res->content";
    print "\n Error - $res->status_line";


    I got following output.
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>200 OK</title>
    </head><body>
    <h1>OK</h1>
    <p>Your browser sent a request that this server could not
    understand.<br />
    </p>
    </body></html>

    Result - HTTP::Response=HASH(0x1e93824)->content
    Error - HTTP::Response=HASH(0x1e93824)->status_line
     
    deep, Jul 24, 2008
    #5
  6. Zak B. Elep Guest

    Re: Perl - approach to query https website for code & parsing xml response

    deep <> writes:

    > I tried that but got same results. I used following:
    >
    > my $ua = new LWP::UserAgent;
    > my $res = $ua->post ($url);
    > print $res->as_string;
    > print "\nResult - $res->content";
    > print "\n Error - $res->status_line";


    Here's my attempt against a live site (sourceforge), it works:

    alteran:~ zakame$ perl -de 0 -MLWP::UserAgent

    Loading DB routines from perl5db.pl version 1.28
    Editor support available.

    ,----[ perl -de 0 -MLWP::UserAgent ]
    | Enter h or `h h' for help, or `man perldebug' for more help.
    |
    | main::(-e:1): 0
    | DB<1> $ua = LWP::UserAgent->new()
    |
    | DB<2> $url = 'https://sourceforge.net/forum/forum.php'
    |
    | DB<3> $res = $ua->post( $url, forum_id => '849067' )
    |
    | DB<4> x $res->as_string()
    | 0 "HTTP/1.1 200 OK\cJConnection: close..."
    |
    | DB<5> x $res->status_line()
    | 0 '200 OK'
    `----

    --
    I like the idea of 256 bits, though: 32 for the (Unicode) character leaves
    room for 224 Bucky bits, which ought to be enough for anyone.
    -- Roland Hutchinson, in alt.folklore.computers
     
    Zak B. Elep, Jul 24, 2008
    #6
  7. J. Gleixner Guest

    deep wrote:
    > I tried that but got same results. I used following:

    You tried what?

    >
    > my $ua = new LWP::UserAgent;
    > my $res = $ua->post ($url);


    What's the URL?

    > print $res->as_string;
    > print "\nResult - $res->content";
    > print "\n Error - $res->status_line";


    Of course, that's not how you'd actually see the value of those methods,
    which is why you're getting this wonderful output.

    > Result - HTTP::Response=HASH(0x1e93824)->content
    > Error - HTTP::Response=HASH(0x1e93824)->status_line


    > print "Result - ", $res->content, "\n";
    > print "Error - ", $res->status_line, "\n";


    I have no idea what this has to do with 'parsing xml response', which
    is in your subject.

    Do you have Crypt::SSLeay installed?

    You're sure you want to do a POST? You're not 'post'ing anything.
     
    J. Gleixner, Jul 24, 2008
    #7
  8. deep Guest

    On Jul 24, 1:20 pm, "J. Gleixner" <>
    wrote:
    You tried what?
    >> Tried post method as get method was not working as per Zak's comment.


    What's the URL?
    >> Had to exclude as it's a financial website.


    Of course, that's not how you'd actually see the value of those
    methods,which is why you're getting this wonderful output.
    >> Got your point. In the begining, I indicated that I'm new to web programming. That's why i'm posting on this forum.


    I have no idea what this has to do with 'parsing xml response', which
    is in your subject.
    >> Again, check the initial post. I have that in subject because if the code works properly, then the site will generate a XML response. Not sure how to read a XML response. And hence parsing xml response in the subject.


    Do you have Crypt::SSLeay installed?
    >> Yes


    You're sure you want to do a POST?  You're not 'post'ing anything.
    >> If I knew what I want then I wouldn't have posted.
     
    deep, Jul 24, 2008
    #8
  9. Zak B. Elep Guest

    Re: Perl - approach to query https website for code & parsing xml response

    Reposting back to clpmisc, it'll probably help someone else. Site
    changed to preserve anonymity ;)

    "dc" <> writes:

    > When I run above through browser, I get following XML response:
    > - <BrokenResponse timestamp="25 July, 2012 01:16 PM EDT">
    > <code>Invalid request</code>
    > <url>https://www.brokensoapsite.example/</url>
    > </BrokenResponse>
    >
    > But through code it gives me bad request instead of above XML response.


    I realized that what you're trying to do is to POST data to what looks
    like a Web Service (in particular, a SOAP/XMLRPC-like service, except it
    doesn't make an envelope;) we're using the wrong tool for the job.

    I suggest looking at SOAP::Lite[1]. Here's a sample session to get you
    started:

    ,----[ perl -de 0 -MSOAP::Lite ]
    |
    | Loading DB routines from perl5db.pl version 1.28
    | Editor support available.
    |
    | Enter h or `h h' for help, or `man perldebug' for more help.
    |
    | main::(-e:1): 0
    | DB<1> $url = 'https://www.brokensoapsite.example/borked.SendRequest'
    |
    | DB<2> $x = SOAP::Data->name( 'x' )->value( '11223344556677889900123' )
    |
    | DB<3> $y = SOAP::Data->name( 'y' )->value( '999999' )
    |
    | DB<4> $res = SOAP::Lite->new( proxy => $url )->call( $x, $y )
    |
    | DB<5> x $res->dataof('//BrokenResponse)->attr->{'timestamp'}
    | 0 '26 July, 2012 08:10 AM EDT'
    |
    | DB<6> x $res->valueof('//BrokenResponse/code')
    | 0 'Invalid request'
    |
    | DB<7> x $res->valueof('//BrokenResponse/url')
    | 0 'https://www.brokensoapsite.example/'
    `----


    Footnotes:
    [1] http://search.cpan.org/perldoc?SOAP::Lite

    --
    I like the idea of 256 bits, though: 32 for the (Unicode) character leaves
    room for 224 Bucky bits, which ought to be enough for anyone.
    -- Roland Hutchinson, in alt.folklore.computers
     
    Zak B. Elep, Jul 26, 2008
    #9
  10. deep Guest

    For some reason my reply didn't get posted, so trying agin.

    Hi Zak

    That indeed worked. Thanks for your help in identifying fundamental
    issue with the use of wrong module. It solved the problem.

    regards,
    Deep
     
    deep, Jul 28, 2008
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Boris Twila
    Replies:
    0
    Views:
    365
    Boris Twila
    Nov 14, 2006
  2. MS Techie
    Replies:
    1
    Views:
    844
    MS Techie
    Dec 23, 2008
  3. Axel
    Replies:
    8
    Views:
    1,111
    Adrienne Boswell
    Apr 27, 2009
  4. jotto
    Replies:
    4
    Views:
    401
    jotto
    Oct 2, 2006
  5. Naveen Dhanuka
    Replies:
    1
    Views:
    291
Loading...

Share This Page