ClientCookie

[Michael Foord]

Has anyone used ClientCookie to store cookies ?
I'm going to play around with 'pickling cookies' - but I wondered if
anyone had any experience of this.

(For session persistence in approx - my cgi proxy)

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[John J. Lee]

Has anyone used ClientCookie to store cookies ?
I'm going to play around with 'pickling cookies' - but I wondered if
anyone had any experience of this.

[...]

There shouldn't be any problem with that. You can pickle the
CookieJar itself, or the cookies inside it ([c for c in cookiejar] --
and use .set_cookie() to get them back into a new CookieJar).

May I suggest instead using cookielib, from Python CVS? (note that
POSTs with urllib2 are broken in 2.4a2, so don't use that)

cookielib is a new module in 2.4, and is a cleaned-up version of the
cookie-handling parts of ClientCookie.


John

 

[John J. Lee]

Has anyone used ClientCookie to store cookies ?
I'm going to play around with 'pickling cookies' - but I wondered if
anyone had any experience of this.

(For session persistence in approx - my cgi proxy)

Just thought to add:

1. are you using a database for this? (you should be) Look at
BSDDBCookieJar. BSDDBCookieJar isn't well-tested, but might be
just the ticket for what you're doing. It's not in cookielib yet,
but the one from ClientCookie 0.9.x should work fine with
cookielib, and I'll make it available in a separate package RSN,
along with the other stuff in ClientCookie but not in cookielib.

2. here's a simpler way (in a sense) of serialising a CookieJar than
pickling:

cj = cookielib.CookieJar()
....
s = [repr(c) for c in cj]


and unserialising:

cj = cookielib.CookieJar()
for cs in s:
cj.set_cookie(eval(cs))
....

You'd still need to write something like BSDDBCookieJar, though, if
I understand what you're doing.


John

 

[Michael Foord]

Has anyone used ClientCookie to store cookies ?
I'm going to play around with 'pickling cookies' - but I wondered if
anyone had any experience of this.

[...]

There shouldn't be any problem with that. You can pickle the
CookieJar itself, or the cookies inside it ([c for c in cookiejar] --
and use .set_cookie() to get them back into a new CookieJar).

At the moment I can't get at the cookies at *all*. Can you see what
I'm doing wrong.
Here's my code (simplified) :


import ClientCookie
openfun = ClientCookie.urlopen

cj = ClientCookie.CookieJar()
req = ClientCookie.Request(theurl, txdata, txheaders)
u = openfun(req)
info = u.info()

print '<PRE>' # This is ina CGI after all
print info # This prints the headers from the server
print
print 'Cookies :'
print cj
for c in cj:
print c
print '</PRE>'


Now if I set theurl to 'http://www.google.co.uk' I get the following
response :
(txdata=None, txheaders={ 'User-agent': 'Mozilla/4.0 (compatible; MSIE
5.5; Windows NT)' })

<PRE>
Cache-Control: private
Content-Type: text/html
Set-Cookie: PREF=ID=0bac71b03c6b1aa8:LD=en:TM=1092300998:LM=1092300998:S=YasA-Kgirv2NPnd9;
expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.co.uk
Server: GWS/2.1
Content-Length: 2901
Date: Thu, 12 Aug 2004 08:56:38 GMT
X-Cache: MISS from dav-serv.tbsmerchants.co.uk
Proxy-Connection: close


Cookies :
<ClientCookie._ClientCookie.CookieJar[]>
</PRE>

And I'm getting this consistently - I can see a cookie in the header,
but it never appears in the CookieJar - so loading and saving the
CookieJar is of no avail.

I'm sure I'm making an obvious mistake - but short of subclassing the
CookieProcessor and doing it all manually (which seems overkill) - I'm
a bit stumped.

May I suggest instead using cookielib, from Python CVS? (note that
POSTs with urllib2 are broken in 2.4a2, so don't use that)

cookielib is a new module in 2.4, and is a cleaned-up version of the
cookie-handling parts of ClientCookie.

I'll have a look in a bit... once I can get any kind of response !!

Thanks for your help.

Fuzzy


http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Michael Foord]

Just thought to add:

1. are you using a database for this? (you should be) Look at
BSDDBCookieJar. BSDDBCookieJar isn't well-tested, but might be
just the ticket for what you're doing. It's not in cookielib yet,
but the one from ClientCookie 0.9.x should work fine with
cookielib, and I'll make it available in a separate package RSN,
along with the other stuff in ClientCookie but not in cookielib.

The program I'm writing is a CGI. I'd like to have *minimum*
dependencies.
The version of Python on the server is 2.2 (I have no control over
that) and having a dependence on CookieClient is enough for me. I
think I can only use BSDDBCookieJar if the server has the Berkely
Database installed ? I'd like other people to be able to use my CGI on
a basic Python 2.2 install - so even if I have it on my server, I
don't want to be dependent on it.

What I'm aiming to provide is persistent cookie support for multiple
users of the same CGI - so I'll probably assign each user an id number
via a cookie I give to them and have a pickled CookieJar for each
user.

I'd also like to work towards cookie management as well - so each user
can see/edit/control which cookies they have saved.

At the moment getting it to work at all would be a bonus........


Regards,

Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

2. here's a simpler way (in a sense) of serialising a CookieJar than
pickling:

cj = cookielib.CookieJar()
...
s = [repr(c) for c in cj]


and unserialising:

cj = cookielib.CookieJar()
for cs in s:
cj.set_cookie(eval(cs))
...

You'd still need to write something like BSDDBCookieJar, though, if
I understand what you're doing.


John

 

[Michael Foord]

Has anyone used ClientCookie to store cookies ?
I'm going to play around with 'pickling cookies' - but I wondered if
anyone had any experience of this.

(For session persistence in approx - my cgi proxy)

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

Ok, I've got somewhere.
I'm now using the 0.9 version and I found the debugging section in the
docs.

I'm using the example from the docs - but I'm pickling a list of the
pickles rather than using the load and save methods (as the docs
suggest) because I get an NotImplemented error for the save method (so
what use is the load method ?).

COOKIEFILE = 'cookies.pkl'
import ClientCookie
openfun = ClientCookie.urlopen

cj = ClientCookie.CookieJar()
opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cj))
ClientCookie.install_opener(opener)

if os.path.isfile(COOKIEFILE):
cookies = open(COOKIEFILE, 'rb')
for entry in pickle.load(cookies):
cj.set_cookie(entry)
cookies.close()


cookies = open(COOKIEFILE, 'wb')
pickle.dump([c for c in cj], cookies)
cookies.close()

req = ClientCookie.Request(theurl)
req.add_header('User-agent', 'Mozilla/4.0 (compatible; MSIE 5.5;
Windows NT)')
u = openfun(req)
info = u.info()

print HR
print '<PRE>'
print info # print the headers
print
print 'Cookies :'
a = 0
for c in cj:
a += 1
print a, c.__repr__() # print each cookie
print '</PRE>'

and I'm now seeing persistent cookies..... hurrah.......
Can I save each cookie as a line of text ? Would that work with the
load method ?

Anyway - now I need to learn about server side cookies so that I can
give each user an 'id' and also write code to clean up unused cookie
files...
*Then* I need to do a seperate CGI that will let users 'manage' their
cookies. (So I'll need to start understanding cookies a bit more)

Great

Regards,

Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[John J. Lee]

The program I'm writing is a CGI. I'd like to have *minimum*
dependencies.
The version of Python on the server is 2.2 (I have no control over
that) and having a dependence on CookieClient is enough for me. I
think I can only use BSDDBCookieJar if the server has the Berkely
Database installed ?
Yes.


I'd like other people to be able to use my CGI on
a basic Python 2.2 install - so even if I have it on my server, I
don't want to be dependent on it.

[...]

The BSDDB-wrapper is a fairly standard Python std. lib. module for
unix machines. I don't know what small-scale commercial CGI hosting
companies offer ATM, though.


John

 

[John J. Lee]

At the moment I can't get at the cookies at *all*. Can you see what
I'm doing wrong.
Here's my code (simplified) : [...]
cj = ClientCookie.CookieJar()
req = ClientCookie.Request(theurl, txdata, txheaders)
u = openfun(req)

[...]

It's not magic! You aren't using cj anywhere!


John

 

[John J. Lee]

Ok, I've got somewhere.
I'm now using the 0.9 version and I found the debugging section in the
docs.

I'm using the example from the docs - but I'm pickling a list of the
pickles rather than using the load and save methods (as the docs
suggest) because I get an NotImplemented error for the save method (so
what use is the load method ?).

I suppose you're trying to use FileCookieJar. That's an abstract base
class. In other words, you're not supposed to use it directly.
You're supposed to use one of its subclasses, such as LWPCookieJar,
MSIECookieJar or MozillaCookieJar. The standard meaning of
NotImplementedError is not "oops, I haven't got round to that", but
rather "don't do that; use a subclass instead".

MSIECookieJar can't save cookies because the format used by Windows
isn't documented (there is an API function, but I don't use it).

[...]

and I'm now seeing persistent cookies..... hurrah.......
Can I save each cookie as a line of text ? Would that work with the
load method ?

Yes. Use LWPCookieJar.

Anyway - now I need to learn about server side cookies so that I can
give each user an 'id'

Have a look at some of the Python web frameworks.

and also write code to clean up unused cookie
files...

[...]

Either

- simply always have one file per user, and persist it by whatever
means you like (LWPCookieJar, pickling,
outfile.write('\n'.join([repr(c) for c in cookiejar])),
whatever...)

or

- write a CookieJar that hides its implementation in terms of
multiple files (and implements .clear_expired_cookies(), etc.).
Probably not a good idea.


John

 

[Michael Foord]

I suppose you're trying to use FileCookieJar. That's an abstract base
class. In other words, you're not supposed to use it directly.
You're supposed to use one of its subclasses, such as LWPCookieJar,
MSIECookieJar or MozillaCookieJar. The standard meaning of
NotImplementedError is not "oops, I haven't got round to that", but
rather "don't do that; use a subclass instead".

No - I was following hte docs - which just shows CookieJar IIRC
Now that I'm using LWPCookieJar everything is wonderful and works fine...

I'm nopw learning about cookies and authentication....

Many Thanks


Fuzzyman

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[John J. Lee]

(e-mail address removed) (John J. Lee) wrote in message news:<[email protected]>... [...]
No - I was following hte docs - which just shows CookieJar IIRC
Now that I'm using LWPCookieJar everything is wonderful and works fine...

[...]

Whoops, sorry. Where is this mistake in the docs? I don't see it.


John

 

[Michael Foord]

(e-mail address removed) (John J. Lee) wrote in message news:<[email protected]>... [...]
No - I was following hte docs - which just shows CookieJar IIRC
Now that I'm using LWPCookieJar everything is wonderful and works fine...

[...]

Whoops, sorry. Where is this mistake in the docs? I don't see it.


John

Right...
I'm using the document that starts :
ClientCookie
Note: this page describes the 0.9.x development version. See here for
the stable 0.4.x version.


There is no mention of LWPCookieJar *anywhere* in the docs.
In the debugging section it talks about using the load and save
method and gives a specific example involving the straight CookieJar
class....... :

When you .save() to or .load()/.revert() from a file, single-session
cookies will expire unless you explicitly request otherwise with the
ignore_discard argument. This may be your problem if you find cookies
are going away after saving and loading.

import ClientCookie
cookies = ClientCookie.CookieJar()
opener = ClientCookie.build_opener(ClientCookie.HTTPCookieProcessor(cookies))
ClientCookie.install_opener(opener)
r = ClientCookie.urlopen("http://foobar.com/")
cookies.save("/some/file", ignore_discard=True, ignore_expires=True)



These are the docs that came with my download...............
LWPCookieJar - works fine.... but I'd never heard of it until you
mentioned it.

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html