ClientCookie

Discussion in 'Python' started by Michael Foord, Aug 11, 2004.

  1. Michael Foord, Aug 11, 2004
    #1
    1. Advertising

  2. Michael Foord

    John J. Lee Guest

    (Michael Foord) writes:

    > Has anyone used ClientCookie to store cookies ?
    > I'm going to play around with 'pickling cookies' - but I wondered if
    > anyone had any experience of this.

    [...]

    There shouldn't be any problem with that. You can pickle the
    CookieJar itself, or the cookies inside it ([c for c in cookiejar] --
    and use .set_cookie() to get them back into a new CookieJar).

    May I suggest instead using cookielib, from Python CVS? (note that
    POSTs with urllib2 are broken in 2.4a2, so don't use that)

    cookielib is a new module in 2.4, and is a cleaned-up version of the
    cookie-handling parts of ClientCookie.


    John
    John J. Lee, Aug 11, 2004
    #2
    1. Advertising

  3. Michael Foord

    John J. Lee Guest

    (Michael Foord) writes:

    > Has anyone used ClientCookie to store cookies ?
    > I'm going to play around with 'pickling cookies' - but I wondered if
    > anyone had any experience of this.
    >
    > (For session persistence in approx - my cgi proxy)


    Just thought to add:

    1. are you using a database for this? (you should be) Look at
    BSDDBCookieJar. BSDDBCookieJar isn't well-tested, but might be
    just the ticket for what you're doing. It's not in cookielib yet,
    but the one from ClientCookie 0.9.x should work fine with
    cookielib, and I'll make it available in a separate package RSN,
    along with the other stuff in ClientCookie but not in cookielib.

    2. here's a simpler way (in a sense) of serialising a CookieJar than
    pickling:

    cj = cookielib.CookieJar()
    ....
    s = [repr(c) for c in cj]


    and unserialising:

    cj = cookielib.CookieJar()
    for cs in s:
    cj.set_cookie(eval(cs))
    ....

    You'd still need to write something like BSDDBCookieJar, though, if
    I understand what you're doing.


    John
    John J. Lee, Aug 11, 2004
    #3
  4. (John J. Lee) wrote in message news:<>...
    > (Michael Foord) writes:
    >
    > > Has anyone used ClientCookie to store cookies ?
    > > I'm going to play around with 'pickling cookies' - but I wondered if
    > > anyone had any experience of this.

    > [...]
    >
    > There shouldn't be any problem with that. You can pickle the
    > CookieJar itself, or the cookies inside it ([c for c in cookiejar] --
    > and use .set_cookie() to get them back into a new CookieJar).
    >


    At the moment I can't get at the cookies at *all*. Can you see what
    I'm doing wrong.
    Here's my code (simplified) :


    import ClientCookie
    openfun = ClientCookie.urlopen

    cj = ClientCookie.CookieJar()
    req = ClientCookie.Request(theurl, txdata, txheaders)
    u = openfun(req)
    info = u.info()

    print '<PRE>' # This is ina CGI after all :)
    print info # This prints the headers from the server
    print
    print 'Cookies :'
    print cj
    for c in cj:
    print c
    print '</PRE>'


    Now if I set theurl to 'http://www.google.co.uk' I get the following
    response :
    (txdata=None, txheaders={ 'User-agent': 'Mozilla/4.0 (compatible; MSIE
    5.5; Windows NT)' })

    <PRE>
    Cache-Control: private
    Content-Type: text/html
    Set-Cookie: PREF=ID=0bac71b03c6b1aa8:LD=en:TM=1092300998:LM=1092300998:S=YasA-Kgirv2NPnd9;
    expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.co.uk
    Server: GWS/2.1
    Content-Length: 2901
    Date: Thu, 12 Aug 2004 08:56:38 GMT
    X-Cache: MISS from dav-serv.tbsmerchants.co.uk
    Proxy-Connection: close


    Cookies :
    <ClientCookie._ClientCookie.CookieJar[]>
    </PRE>

    And I'm getting this consistently - I can see a cookie in the header,
    but it never appears in the CookieJar - so loading and saving the
    CookieJar is of no avail.

    I'm sure I'm making an obvious mistake - but short of subclassing the
    CookieProcessor and doing it all manually (which seems overkill) - I'm
    a bit stumped.


    > May I suggest instead using cookielib, from Python CVS? (note that
    > POSTs with urllib2 are broken in 2.4a2, so don't use that)
    >
    > cookielib is a new module in 2.4, and is a cleaned-up version of the
    > cookie-handling parts of ClientCookie.
    >
    >


    I'll have a look in a bit... once I can get any kind of response !!

    Thanks for your help.

    Fuzzy


    http://www.voidspace.org.uk/atlantibots/pythonutils.html


    > John
    Michael Foord, Aug 12, 2004
    #4
  5. (John J. Lee) wrote in message news:<>...
    > (Michael Foord) writes:
    >
    > > Has anyone used ClientCookie to store cookies ?
    > > I'm going to play around with 'pickling cookies' - but I wondered if
    > > anyone had any experience of this.
    > >
    > > (For session persistence in approx - my cgi proxy)

    >
    > Just thought to add:
    >
    > 1. are you using a database for this? (you should be) Look at
    > BSDDBCookieJar. BSDDBCookieJar isn't well-tested, but might be
    > just the ticket for what you're doing. It's not in cookielib yet,
    > but the one from ClientCookie 0.9.x should work fine with
    > cookielib, and I'll make it available in a separate package RSN,
    > along with the other stuff in ClientCookie but not in cookielib.
    >


    The program I'm writing is a CGI. I'd like to have *minimum*
    dependencies.
    The version of Python on the server is 2.2 (I have no control over
    that) and having a dependence on CookieClient is enough for me. I
    think I can only use BSDDBCookieJar if the server has the Berkely
    Database installed ? I'd like other people to be able to use my CGI on
    a basic Python 2.2 install - so even if I have it on my server, I
    don't want to be dependent on it.

    What I'm aiming to provide is persistent cookie support for multiple
    users of the same CGI - so I'll probably assign each user an id number
    via a cookie I give to them and have a pickled CookieJar for each
    user.

    I'd also like to work towards cookie management as well - so each user
    can see/edit/control which cookies they have saved.

    At the moment getting it to work at all would be a bonus........


    Regards,

    Fuzzy

    http://www.voidspace.org.uk/atlantibots/pythonutils.html



    > 2. here's a simpler way (in a sense) of serialising a CookieJar than
    > pickling:
    >
    > cj = cookielib.CookieJar()
    > ...
    > s = [repr(c) for c in cj]
    >
    >
    > and unserialising:
    >
    > cj = cookielib.CookieJar()
    > for cs in s:
    > cj.set_cookie(eval(cs))
    > ...
    >
    > You'd still need to write something like BSDDBCookieJar, though, if
    > I understand what you're doing.
    >
    >
    > John
    Michael Foord, Aug 12, 2004
    #5
  6. (Michael Foord) wrote in message news:<>...
    > Has anyone used ClientCookie to store cookies ?
    > I'm going to play around with 'pickling cookies' - but I wondered if
    > anyone had any experience of this.
    >
    > (For session persistence in approx - my cgi proxy)
    >
    > Regards,
    >
    >
    > Fuzzy
    >
    > http://www.voidspace.org.uk/atlantibots/pythonutils.html


    Ok, I've got somewhere.
    I'm now using the 0.9 version and I found the debugging section in the
    docs.

    I'm using the example from the docs - but I'm pickling a list of the
    pickles rather than using the load and save methods (as the docs
    suggest) because I get an NotImplemented error for the save method (so
    what use is the load method ?).

    COOKIEFILE = 'cookies.pkl'
    import ClientCookie
    openfun = ClientCookie.urlopen

    cj = ClientCookie.CookieJar()
    opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cj))
    ClientCookie.install_opener(opener)

    if os.path.isfile(COOKIEFILE):
    cookies = open(COOKIEFILE, 'rb')
    for entry in pickle.load(cookies):
    cj.set_cookie(entry)
    cookies.close()


    cookies = open(COOKIEFILE, 'wb')
    pickle.dump([c for c in cj], cookies)
    cookies.close()

    req = ClientCookie.Request(theurl)
    req.add_header('User-agent', 'Mozilla/4.0 (compatible; MSIE 5.5;
    Windows NT)')
    u = openfun(req)
    info = u.info()

    print HR
    print '<PRE>'
    print info # print the headers
    print
    print 'Cookies :'
    a = 0
    for c in cj:
    a += 1
    print a, c.__repr__() # print each cookie
    print '</PRE>'

    and I'm now seeing persistent cookies..... hurrah.......
    Can I save each cookie as a line of text ? Would that work with the
    load method ?

    Anyway - now I need to learn about server side cookies so that I can
    give each user an 'id' and also write code to clean up unused cookie
    files...
    *Then* I need to do a seperate CGI that will let users 'manage' their
    cookies. (So I'll need to start understanding cookies a bit more)

    Great

    Regards,

    Fuzzy

    http://www.voidspace.org.uk/atlantibots/pythonutils.html
    Michael Foord, Aug 12, 2004
    #6
  7. Michael Foord

    John J. Lee Guest

    (Michael Foord) writes:
    [...]
    > The program I'm writing is a CGI. I'd like to have *minimum*
    > dependencies.
    > The version of Python on the server is 2.2 (I have no control over
    > that) and having a dependence on CookieClient is enough for me. I
    > think I can only use BSDDBCookieJar if the server has the Berkely
    > Database installed ?


    Yes.


    > I'd like other people to be able to use my CGI on
    > a basic Python 2.2 install - so even if I have it on my server, I
    > don't want to be dependent on it.

    [...]

    The BSDDB-wrapper is a fairly standard Python std. lib. module for
    unix machines. I don't know what small-scale commercial CGI hosting
    companies offer ATM, though.


    John
    John J. Lee, Aug 12, 2004
    #7
  8. Michael Foord

    John J. Lee Guest

    (Michael Foord) writes:
    [...]
    > At the moment I can't get at the cookies at *all*. Can you see what
    > I'm doing wrong.
    > Here's my code (simplified) :

    [...]
    > cj = ClientCookie.CookieJar()
    > req = ClientCookie.Request(theurl, txdata, txheaders)
    > u = openfun(req)

    [...]

    It's not magic! You aren't using cj anywhere!


    John
    John J. Lee, Aug 12, 2004
    #8
  9. Michael Foord

    John J. Lee Guest

    (Michael Foord) writes:
    [...]
    > Ok, I've got somewhere.
    > I'm now using the 0.9 version and I found the debugging section in the
    > docs.
    >
    > I'm using the example from the docs - but I'm pickling a list of the
    > pickles rather than using the load and save methods (as the docs
    > suggest) because I get an NotImplemented error for the save method (so
    > what use is the load method ?).


    I suppose you're trying to use FileCookieJar. That's an abstract base
    class. In other words, you're not supposed to use it directly.
    You're supposed to use one of its subclasses, such as LWPCookieJar,
    MSIECookieJar or MozillaCookieJar. The standard meaning of
    NotImplementedError is not "oops, I haven't got round to that", but
    rather "don't do that; use a subclass instead".

    MSIECookieJar can't save cookies because the format used by Windows
    isn't documented (there is an API function, but I don't use it).

    [...]
    > and I'm now seeing persistent cookies..... hurrah.......
    > Can I save each cookie as a line of text ? Would that work with the
    > load method ?


    Yes. Use LWPCookieJar.


    > Anyway - now I need to learn about server side cookies so that I can
    > give each user an 'id'


    Have a look at some of the Python web frameworks.


    > and also write code to clean up unused cookie
    > files...

    [...]

    Either

    - simply always have one file per user, and persist it by whatever
    means you like (LWPCookieJar, pickling,
    outfile.write('\n'.join([repr(c) for c in cookiejar])),
    whatever...)

    or

    - write a CookieJar that hides its implementation in terms of
    multiple files (and implements .clear_expired_cookies(), etc.).
    Probably not a good idea.


    John
    John J. Lee, Aug 12, 2004
    #9
  10. (John J. Lee) wrote in message news:<>...
    > (Michael Foord) writes:
    > [...]
    > > Ok, I've got somewhere.
    > > I'm now using the 0.9 version and I found the debugging section in the
    > > docs.
    > >
    > > I'm using the example from the docs - but I'm pickling a list of the
    > > pickles rather than using the load and save methods (as the docs
    > > suggest) because I get an NotImplemented error for the save method (so
    > > what use is the load method ?).

    >
    > I suppose you're trying to use FileCookieJar. That's an abstract base
    > class. In other words, you're not supposed to use it directly.
    > You're supposed to use one of its subclasses, such as LWPCookieJar,
    > MSIECookieJar or MozillaCookieJar. The standard meaning of
    > NotImplementedError is not "oops, I haven't got round to that", but
    > rather "don't do that; use a subclass instead".


    No - I was following hte docs - which just shows CookieJar IIRC :)
    Now that I'm using LWPCookieJar everything is wonderful and works fine...

    I'm nopw learning about cookies and authentication....

    Many Thanks


    Fuzzyman

    http://www.voidspace.org.uk/atlantibots/pythonutils.html
    Michael Foord, Aug 18, 2004
    #10
  11. Michael Foord

    John J. Lee Guest

    (Michael Foord) writes:
    > (John J. Lee) wrote in message news:<>...

    [...]
    > No - I was following hte docs - which just shows CookieJar IIRC :)
    > Now that I'm using LWPCookieJar everything is wonderful and works fine...

    [...]

    Whoops, sorry. Where is this mistake in the docs? I don't see it.


    John
    John J. Lee, Aug 19, 2004
    #11
  12. (John J. Lee) wrote in message news:<>...
    > (Michael Foord) writes:
    > > (John J. Lee) wrote in message news:<>...

    > [...]
    > > No - I was following hte docs - which just shows CookieJar IIRC :)
    > > Now that I'm using LWPCookieJar everything is wonderful and works fine...

    > [...]
    >
    > Whoops, sorry. Where is this mistake in the docs? I don't see it.
    >
    >
    > John


    Right...
    I'm using the document that starts :
    ClientCookie
    Note: this page describes the 0.9.x development version. See here for
    the stable 0.4.x version.


    There is no mention of LWPCookieJar *anywhere* in the docs.
    In the debugging section it talks about using the load and save
    method and gives a specific example involving the straight CookieJar
    class....... :

    When you .save() to or .load()/.revert() from a file, single-session
    cookies will expire unless you explicitly request otherwise with the
    ignore_discard argument. This may be your problem if you find cookies
    are going away after saving and loading.

    import ClientCookie
    cookies = ClientCookie.CookieJar()
    opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cookies))
    ClientCookie.install_opener(opener)
    r = ClientCookie.urlopen("http://foobar.com/")
    cookies.save("/some/file", ignore_discard=True, ignore_expires=True)



    These are the docs that came with my download...............
    LWPCookieJar - works fine.... but I'd never heard of it until you
    mentioned it.

    Regards,


    Fuzzy

    http://www.voidspace.org.uk/atlantibots/pythonutils.html
    Michael Foord, Aug 23, 2004
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mark Carter

    ClientCookie bug

    Mark Carter, Aug 14, 2003, in forum: Python
    Replies:
    28
    Views:
    806
    John J. Lee
    Aug 27, 2003
  2. Grant Edwards

    Why is ClientCookie/urllib2 using https?

    Grant Edwards, Sep 24, 2003, in forum: Python
    Replies:
    0
    Views:
    329
    Grant Edwards
    Sep 24, 2003
  3. Chuck Bearden
    Replies:
    2
    Views:
    788
    Chuck Bearden
    May 11, 2004
  4. Richie Hindle

    urllib2 / ClientCookie / Keep-Alive

    Richie Hindle, Oct 15, 2004, in forum: Python
    Replies:
    0
    Views:
    800
    Richie Hindle
    Oct 15, 2004
  5. Max M
    Replies:
    5
    Views:
    381
    John J Lee
    Oct 26, 2004
Loading...

Share This Page