persistent deque

[castironpi]

I'd like a persistent deque, such that the instance operations all
commit atomicly to file system.

 

[inhahe]

I'd like a persistent deque, such that the instance operations all
commit atomicly to file system.

ok, i made your persistent deque. but there are two important notes
regarding this module: 1) I am a genius. 2) it doesn't actually work for
some reason. i.e., loading an instance from file won't work. i don't know
why. the pickle module is doing something i didn't expect.


from collections import deque
import os, pickle, random

class pdeque(deque):
def __init__(self, filename, initial):
deque.__init__(self, initial)
self.filename = filename
self.tempfilename = ''.join((random.choice("abcdefghijklmnopqrstuvwxyz")
for x in xrange(10)))+".tmp"
self.save()
def save(self):
pickle.dump(self, open(self.tempfilename,'wb'))
try: os.remove(self.filename)
except: pass
os.rename(self.tempfilename, self.filename)
@classmethod
def load(cls, filename):
return pickle.load(open(filename,'rb'))

for name in "append appendleft extend extendleft remove rotate".split():
func = getattr(deque, name)
def func2(instance, arg, func=func):
result = func(instance, arg)
instance.save()
return result
setattr(pdeque, name, func2)

for name in ("pop", "popleft", "clear"):
func = getattr(deque, name)
def func2(instance, func=func):
result = func(instance)
instance.save()
return result
setattr(pdeque, name, func2)


---

you use it just like a deque object except that when initializing the first
parameter is a filename for it to store in. to load the deque object again
later, call .load(filename) on the pdeque class or a pdeque object (except
that it doesn't work.)

i'm sure somebody here knows what obscure feature of pickling it is that's
doing this counterintuitively. (disclaimer: i've never used pickle to store
a custom type before)

 

[inhahe]

def __init__(self, filename, initial):

should be

def __init__(self, filename, initial=[]):

(that was the whole reason i put the filename first.)

sorry.

 

[inhahe]

oh yeah! thanks for pointing it out, i should have thought of that, i
*just* read about it today, or was it yesterday.

so what's the problem with pickle? i have a feeling, maybe i read it, that
when you pickle a derived class it pickles it as a member of the base class,
is that it?

i don't know how i would get around the problem, though, because i'd have to
know how to access the deque object that my class stores when i do
deque.__init__ in my constructor, so that i could pickle it and my class
variables separately.

def __init__(self, filename, initial):

should be

def __init__(self, filename, initial=[]):

(that was the whole reason i put the filename first.)

sorry.

Defaulting initial to empty list this way is asking for trouble. You
should
default it to None and check for None and reset to empty list.

-Larry

 

[inhahe]

i don't know how i would get around the problem, though, because i'd have
to know how to access the deque object that my class stores when i do
deque.__init__ in my constructor, so that i could pickle it and my class
variables separately.

i decided i could just pickle deque(self), which should return a regular
deque object with the data from self, and then in the load routine make a
pdeque object from it.

that, of couse, gives me another error. 'collections.deque' object has no
attribute 'write'. from the line 'self.write = file.write', which is in
pickle.py
pickling list(self) doesnt work either.

 

[inhahe]

something randomly made me realize why my second solution didn't work, so i
fixed it. now you have a working persistent deque.

1. if you call somepdequeobject.load(filename) (as opposed to
pdeque.load(filename)), do you want it to return a new object that it loaded
from file, or do you want it to change somepdequeobject to be the object in
the file? i'm not sure which way is canonical.

2. i'm not sure if i actually have to use a temp file and then copy it. i
don't know if pickle.dump as an "atomic commit".

3. cPickle would be faster but i think that means you might end up storing
something in your deque that can't be pickled, like a circular reference,
and dunno what else.

4. can someone tell me if the way i'm using the decorator is sane? i've
never used decorators before. it just seems ridiculous to a) define a lambda
that just throws away the parameter, and b) define a meaningless function to
pass to that lambda.

5. Of course, I am a genius.

from collections import deque
import os, pickle, random

class pdeque(deque):
def __init__(self, filename, initial=None):
if initial is None: initial = []
deque.__init__(self, initial)
self.filename = filename
self.tempfilename = ''.join((random.choice("abcdefghijklmnopqrstuvwxyz")
for x in xrange(10)))+".tmp"
self.save()
def save(self):
pickle.dump(deque(self), open(self.tempfilename,'wb'))
try: os.remove(self.filename)
except: pass
os.rename(self.tempfilename, self.filename)
@classmethod
def load(cls, filename):
return pdeque(filename, pickle.load(open(filename,'rb')))

#todo: change this so that if it's called from an object it loads
#the data into that object?

def makefunc(func):
def func2(instance, *args):
result = func(instance, *args)
instance.save()
return result
return lambda _: func2

for name, func in deque.__dict__.items():
if (not name.startswith("_")) or name in ('__delitem__', '__setitem__'):
@makefunc(func)
def f(): pass
setattr(pdeque, name, f)

 

[inhahe]

4. can someone tell me if the way i'm using the decorator is sane? i've
never used decorators before. it just seems ridiculous to a) define a
lambda that just throws away the parameter, and b) define a meaningless
function to pass to that lambda.

I thought about the fact that a decorator is merely syntactic sugar, so it
shouldn't have any closure magic that I can't make myself, and I realized
that I could have done it the following way:

def makefunc(func):
def func2(instance, *args):
result = func(instance, *args)
instance.save()
return result
return func2

for name, func in deque.__dict__.items():
if (not name.startswith("_")) or name in ('__delitem__', '__setitem__'):
setattr(pdeque, name, makefunc(func))

 

[Gabriel Genellina]

I thought about the fact that a decorator is merely syntactic sugar, so it
shouldn't have any closure magic that I can't make myself, and I realized
that I could have done it the following way:

def makefunc(func):
def func2(instance, *args):
result = func(instance, *args)
instance.save()
return result
return func2

Using functools.wraps is better because it helps to preserve the function name and signature:

def makefunc(func):
@wraps(func)
def wrapper(self, *args, **kwds):
result = func(self, *args, **kwds)
self.save()
return result
return wrapper