HTTP::Cookie won't store sent cookie

Discussion in 'Perl Misc' started by Richard Lawrence, Mar 17, 2005.

  1. Hi all,

    My script requests http://foo.bar.com/ with code that looks a little
    like this:

    my $ua = LWP::UserAgent->new;
    my $cookie_jar = HTTP::Cookies->new(file => $cookie_path);
    $cookie_jar->load($cookie_path);
    $ua->cookie_jar($cookie_jar);

    my $req = HTTP::Request->new(GET => "http://foo.bar.com/");
    $cookie_jar->add_cookie_header($req);

    # Make request
    my $res = $ua->request($req);

    # HTML back
    if ($res->is_success)
    {
    $cookie_jar->extract_cookies($res);
    $cookie_jar->save();
    }

    This works great, however the site sends back this:

    Set-Cookie: name=fred; domain=.bar.com; path=/

    which for some reason doesn't get saved in the cookie jar.

    I'm not sure if this is because the set-cookie header is badly formed
    or non-standard but since Firefox and IE are both happy with it I
    really need to make my code happy with it.

    Have I done something wrong or is there a way to get this to work?

    Many thanks in advance,

    Richard
    Richard Lawrence, Mar 17, 2005
    #1
    1. Advertising

  2. Richard Lawrence wrote:
    > My script requests http://foo.bar.com/ with code that looks a little
    > like this:


    <code snipped>

    > This works great, however the site sends back this:
    >
    > Set-Cookie: name=fred; domain=.bar.com; path=/
    >
    > which for some reason doesn't get saved in the cookie jar.


    Have you possibly finished the printing of CGI headers prematurely? If
    you don't understand what I mean by that, please post a *short* but
    *complete* script that illustrates the issue.

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
    Gunnar Hjalmarsson, Mar 17, 2005
    #2
    1. Advertising

  3. Gunnar Hjalmarsson wrote:
    > Richard Lawrence wrote:
    > > My script requests http://foo.bar.com/ with code that looks a

    little
    > > like this:

    >
    > <code snipped>
    >
    > > This works great, however the site sends back this:
    > >
    > > Set-Cookie: name=fred; domain=.bar.com; path=/
    > >
    > > which for some reason doesn't get saved in the cookie jar.

    >
    > Have you possibly finished the printing of CGI headers prematurely?

    If
    > you don't understand what I mean by that, please post a *short* but
    > *complete* script that illustrates the issue.


    You're right in the fact that I don't understand :) Also following a
    bit more investigation, it doesn't look like what I originally thought
    was the problem isn't the problem.

    Here is a script that demonstrates the code although sadly I have to
    mask the site and the details sent. It's worth noting I have no control
    over the site I connect to and that the one I use below is different
    from the original one as it's easier to show the problem:

    #!/usr/bin/perl

    use strict;
    use warnings;

    use LWP::UserAgent;
    use HTTP::Cookies;
    use HTTP::Request;

    my $received_html;

    # Note: horrible wrapping caused by groups-google.com

    &get_page("http://www.somedomain.com/process_logon.jsp",
    "referring_page=sms.jsp&email_address=my%40email%2Eaddress&password=mypassword",
    "http://www.somedomain.com/process_logon.jsp");

    &get_page("http://www.somedomain.com/sms.jsp", "",
    "http://www.somedomain.com/process_logon.jsp");

    sub get_page
    {
    my ($url, $data, $referer) = @_;
    my $cookie_path = "./cookies";

    # If $data exists then it's POST, otherwise GET

    # Create the user agent

    my $ua = LWP::UserAgent->new(env_proxy => 1);
    $ua->agent("Mozilla/4.0");
    push @{ $ua->requests_redirectable }, 'POST';

    # Create the cookie jar

    my $cookie_jar = HTTP::Cookies->new(file => $cookie_path);
    $cookie_jar->load($cookie_path);
    $ua->cookie_jar($cookie_jar);

    # If we have a data= section, then this is a POST

    my $req;

    if ($data)
    {
    $req = HTTP::Request->new(POST => $url);
    $req->content_type('application/x-www-form-urlencoded');
    $req->content($data);
    }
    else
    {
    $req = HTTP::Request->new(GET => $url);
    }

    $req->header('Accept' => 'text/*');

    # Add a referer if required

    $req->referer($referer) if ($referer);

    # Add cookies

    $cookie_jar->add_cookie_header($req);

    print "-- Start of sent headers --\n";
    print $req->as_string();
    print "\n-- End of sent headers --\n";

    # Connect!

    my $res = $ua->request($req);

    $received_html = "";

    die "stop: Error sending request - " . $res->status_line . "\n" if
    (!$res->is_success);

    # Since we're here, we know that it went ok

    print "-- Start of received headers and HTML --\n";
    print $res->as_string . "\n";
    print "-- End of received headers and HTML --\n\n";

    $received_html = $res->content;

    # Extract cookies

    $cookie_jar->extract_cookies($res);
    $cookie_jar->save();
    }

    Here is the output I get:

    -- Start of sent headers --
    POST http://www.somedomain.com/process_logon.jsp
    Accept: text/*
    Referer: http://www.somedomain.com/process_logon.jsp
    Content-Type: application/x-www-form-urlencoded

    referring_page=sms.jsp&email_address=my%40email%2Eaddress&password=mypassword

    -- End of sent headers --
    -- Start of received headers and HTML --
    HTTP/1.1 200 OK
    Cache-Control: no-cache="set-cookie,set-cookie2"
    Connection: close
    Date: Fri, 18 Mar 2005 10:07:10 GMT
    Via: 1.1 www.somedomain.com
    Server: Microsoft-IIS/5.0
    Content-Length: 103
    Content-Type: text/html; charset=ISO-8859-1
    Expires: Thu, 01 Dec 1994 16:00:00 GMT
    Client-Date: Fri, 18 Mar 2005 10:07:22 GMT
    Client-Peer: 10.10.142.190:80
    Client-Response-Num: 1
    Refresh: 2;url="sms.jsp"
    Set-Cookie: registered=1
    Set-Cookie: jsessionid=2380421411141442392;path=/
    X-Powered-By: ASP.NET

    [html snipped]
    -- End of received headers and HTML --

    As you can see here, two cookies have been set however in the next
    send...

    -- Start of sent headers --
    GET http://www.somedomain.com/sms.jsp
    Accept: text/*
    Referer: http://www.somedomain.com/process_logon.jsp


    -- End of sent headers --

    These aren't sent back to the server.

    Many thanks for any suggestions!

    Richard
    Richard Lawrence, Mar 18, 2005
    #3
  4. Richard Lawrence wrote:
    > Gunnar Hjalmarsson wrote:
    >> Richard Lawrence wrote:
    >>> My script requests http://foo.bar.com/ with code that looks a little
    >>> like this:

    >>
    >> <code snipped>
    >>
    >>> This works great, however the site sends back this:
    >>>
    >>> Set-Cookie: name=fred; domain=.bar.com; path=/
    >>>
    >>> which for some reason doesn't get saved in the cookie jar.

    >>
    >> Have you possibly finished the printing of CGI headers prematurely? If
    >> you don't understand what I mean by that, please post a *short* but
    >> *complete* script that illustrates the issue.

    >
    > You're right in the fact that I don't understand :)


    Well, I obviously didn't read your question carefully enough, sorry.
    Please disregard that remark.

    A cookie without a valid expires parameter is not saved to file by
    default. That makes perfect sense, since such a cookie is just a session
    cookie which automatically expires at end of session. It gets not saved
    to disk by browsers either.

    Use the "ignore_discard" parameter to still have it saved to file:

    my $cookie_jar =
    HTTP::Cookies->new(file => $cookie_path, ignore_discard => 1);

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
    Gunnar Hjalmarsson, Mar 19, 2005
    #4
  5. Gunnar Hjalmarsson wrote:
    > Well, I obviously didn't read your question carefully enough, sorry.
    > Please disregard that remark.


    Not to worry :)

    > A cookie without a valid expires parameter is not saved to file by
    > default. That makes perfect sense, since such a cookie is just a

    session
    > cookie which automatically expires at end of session. It gets not

    saved
    > to disk by browsers either.
    >
    > Use the "ignore_discard" parameter to still have it saved to file:
    >
    > my $cookie_jar =
    > HTTP::Cookies->new(file => $cookie_path, ignore_discard => 1);


    Ahhh, I didn't realise I needed that because I failed to spot it was a
    session cookie. Works like a charm now.

    Thanks very much!

    Richard
    Richard Lawrence, Mar 21, 2005
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?Y2hzYWtkYQ==?=
    Replies:
    1
    Views:
    775
    Peter Rilling
    Apr 23, 2004
  2. Andreas Klemt
    Replies:
    8
    Views:
    10,664
    spalding
    Aug 17, 2004
  3. iotaivy
    Replies:
    0
    Views:
    146
    iotaivy
    Nov 10, 2005
  4. Paul  Wolpe
    Replies:
    0
    Views:
    165
    Paul Wolpe
    Oct 17, 2005
  5. HopfZ
    Replies:
    2
    Views:
    81
    Bart Van der Donck
    Jan 4, 2007
Loading...

Share This Page