How to better pickle an extension type

[dgdev]

I would like to pickle an extension type (written in pyrex). I have
it working thus far by defining three methods:

class C:
# for pickling
__getstate__(self):
... # make 'state_obj'
return state_obj

__reduce__(self):
return C,(args,to,__init__),me.__getstate__()

# for unpickling
__setstate__(self,state_obj):
self.x=state_obj.x
...


This gets the class pickling and unpickling.

However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.

I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.

Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

 

[Alex Martelli]

I would like to pickle an extension type (written in pyrex). I have
it working thus far by defining three methods:

class C:
# for pickling
__getstate__(self):
... # make 'state_obj'
return state_obj

__reduce__(self):
return C,(args,to,__init__),me.__getstate__()

# for unpickling
__setstate__(self,state_obj):
self.x=state_obj.x
...


This gets the class pickling and unpickling.

However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.

I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.

Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

I don't understand why you have a problem -- __init__ is NOT called by
default upon loading an object w/__setstate__. Witness:
.... def __init__(self, *a): print 'init', a
.... def __getstate__(self): print 'gs'; return {}
.... def __setstate__(self, *a): print 'ss', a
.... ss ({},)


Perhaps you're not using protocol 2? You should be...


Alex

 

[Ziga Seilnacht]

I would like to pickle an extension type (written in pyrex). I have
it working thus far by defining three methods:

class C:
# for pickling
__getstate__(self):
... # make 'state_obj'
return state_obj

__reduce__(self):
return C,(args,to,__init__),me.__getstate__()

# for unpickling
__setstate__(self,state_obj):
self.x=state_obj.x
...

This gets the class pickling and unpickling.

However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.

I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.

Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

Why are you overwriting the __reduce__() method? The default
object.__reduce__() method, inherited by all new style classes,
already does what you want. If you really must overwrite it, and
you don't want __init__() to get called, then you should return a
reconstructor named __newobj__() as the first item of reduce
tuple. Something like this:
.... return cls.__new__(cls, *args)
........ def __init__(self):
.... print "I shouldn't be called at reconstruction"
.... def __reduce__(self):
.... try:
.... getnewargs = self.__getnewargs__
.... except AttributeError:
.... newargs = (self.__class__,)
.... else:
.... newargs = (self.__class__,) + getnewargs()
.... try:
.... getstate = self.__getstate__
.... except AttributeError:
.... # this ignores __slots__ complications
.... state = self.__dict__
.... else:
.... state = getstate()
.... # this ignores list and dict subclasses
.... return __newobj__, newargs, state
........ assert isinstance(pickle.loads(pickle.dumps(c, proto)), C)
....
Ziga

 

[dgdev]

Thanks for your replies.

The code I showed above was pyrex code, not python code. You are
correct that python objects do not require .__reduce__() to be
picklable, but apparently c extension types do (makes sense, they must
be more opaque to the python machinery).

I'll try the .__newobj__(), see if I can get it to do what I want...