How to better pickle an extension type

Discussion in 'Python' started by dgdev, Apr 16, 2007.

  1. dgdev

    dgdev Guest

    I would like to pickle an extension type (written in pyrex). I have
    it working thus far by defining three methods:

    class C:
    # for pickling
    __getstate__(self):
    ... # make 'state_obj'
    return state_obj

    __reduce__(self):
    return C,(args,to,__init__),me.__getstate__()

    # for unpickling
    __setstate__(self,state_obj):
    self.x=state_obj.x
    ...


    This gets the class pickling and unpickling.

    However, I'd like to not specify arguments for __init__ (as I do now
    in __reduce__), and so not have __init__ invoked during unpickling.

    I would like to have the pickling machinery somehow create an
    uninitialized object, and then call its __setstate__, where I can re-
    create it from 'state_obj'.

    Is there a kosher way to do so, that is without me having to have a
    special mode in the constructor for when the object is being created
    by the unpickler?
    dgdev, Apr 16, 2007
    #1
    1. Advertising

  2. dgdev <> wrote:

    > I would like to pickle an extension type (written in pyrex). I have
    > it working thus far by defining three methods:
    >
    > class C:
    > # for pickling
    > __getstate__(self):
    > ... # make 'state_obj'
    > return state_obj
    >
    > __reduce__(self):
    > return C,(args,to,__init__),me.__getstate__()
    >
    > # for unpickling
    > __setstate__(self,state_obj):
    > self.x=state_obj.x
    > ...
    >
    >
    > This gets the class pickling and unpickling.
    >
    > However, I'd like to not specify arguments for __init__ (as I do now
    > in __reduce__), and so not have __init__ invoked during unpickling.
    >
    > I would like to have the pickling machinery somehow create an
    > uninitialized object, and then call its __setstate__, where I can re-
    > create it from 'state_obj'.
    >
    > Is there a kosher way to do so, that is without me having to have a
    > special mode in the constructor for when the object is being created
    > by the unpickler?


    I don't understand why you have a problem -- __init__ is NOT called by
    default upon loading an object w/__setstate__. Witness:

    >>> class C(object):

    .... def __init__(self, *a): print 'init', a
    .... def __getstate__(self): print 'gs'; return {}
    .... def __setstate__(self, *a): print 'ss', a
    ....
    >>> c = C()

    init ()
    >>> s = cPickle.dumps(c, 2)

    gs
    >>> s

    '\x80\x02c__main__\nC\nq\x01)\x81q\x02}b.'
    >>> z = cPickle.loads(s)

    ss ({},)


    Perhaps you're not using protocol 2? You should be...


    Alex
    Alex Martelli, Apr 17, 2007
    #2
    1. Advertising

  3. dgdev wrote:
    > I would like to pickle an extension type (written in pyrex). I have
    > it working thus far by defining three methods:
    >
    > class C:
    > # for pickling
    > __getstate__(self):
    > ... # make 'state_obj'
    > return state_obj
    >
    > __reduce__(self):
    > return C,(args,to,__init__),me.__getstate__()
    >
    > # for unpickling
    > __setstate__(self,state_obj):
    > self.x=state_obj.x
    > ...
    >
    > This gets the class pickling and unpickling.
    >
    > However, I'd like to not specify arguments for __init__ (as I do now
    > in __reduce__), and so not have __init__ invoked during unpickling.
    >
    > I would like to have the pickling machinery somehow create an
    > uninitialized object, and then call its __setstate__, where I can re-
    > create it from 'state_obj'.
    >
    > Is there a kosher way to do so, that is without me having to have a
    > special mode in the constructor for when the object is being created
    > by the unpickler?


    Why are you overwriting the __reduce__() method? The default
    object.__reduce__() method, inherited by all new style classes,
    already does what you want. If you really must overwrite it, and
    you don't want __init__() to get called, then you should return a
    reconstructor named __newobj__() as the first item of reduce
    tuple. Something like this:

    >>> def __newobj__(cls, *args):

    .... return cls.__new__(cls, *args)
    ....
    >>> class C(object):

    .... def __init__(self):
    .... print "I shouldn't be called at reconstruction"
    .... def __reduce__(self):
    .... try:
    .... getnewargs = self.__getnewargs__
    .... except AttributeError:
    .... newargs = (self.__class__,)
    .... else:
    .... newargs = (self.__class__,) + getnewargs()
    .... try:
    .... getstate = self.__getstate__
    .... except AttributeError:
    .... # this ignores __slots__ complications
    .... state = self.__dict__
    .... else:
    .... state = getstate()
    .... # this ignores list and dict subclasses
    .... return __newobj__, newargs, state
    ....
    >>> c = C()

    I shouldn't be called at reconstruction
    >>> import pickle
    >>> for proto in range(3):

    .... assert isinstance(pickle.loads(pickle.dumps(c, proto)), C)
    ....
    >>>


    Ziga
    Ziga Seilnacht, Apr 17, 2007
    #3
  4. dgdev

    dgdev Guest

    Thanks for your replies.

    The code I showed above was pyrex code, not python code. You are
    correct that python objects do not require .__reduce__() to be
    picklable, but apparently c extension types do (makes sense, they must
    be more opaque to the python machinery).

    I'll try the .__newobj__(), see if I can get it to do what I want...

    On Apr 17, 2:50 am, Ziga Seilnacht <> wrote:
    > dgdev wrote:
    > > I would like topicklean extension type (written inpyrex). I have
    > > it working thus far by defining three methods:

    >
    > > class C:
    > > # for pickling
    > > __getstate__(self):
    > > ... # make 'state_obj'
    > > return state_obj

    >
    > > __reduce__(self):
    > > return C,(args,to,__init__),me.__getstate__()

    >
    > > # for unpickling
    > > __setstate__(self,state_obj):
    > > self.x=state_obj.x
    > > ...

    >
    > > This gets the class pickling and unpickling.

    >
    > > However, I'd like to not specify arguments for __init__ (as I do now
    > > in __reduce__), and so not have __init__ invoked during unpickling.

    >
    > > I would like to have the pickling machinery somehow create an
    > > uninitialized object, and then call its __setstate__, where I can re-
    > > create it from 'state_obj'.

    >
    > > Is there a kosher way to do so, that is without me having to have a
    > > special mode in the constructor for when the object is being created
    > > by the unpickler?

    >
    > Why are you overwriting the __reduce__() method? The default
    > object.__reduce__() method, inherited by all new style classes,
    > already does what you want. If you really must overwrite it, and
    > you don't want __init__() to get called, then you should return a
    > reconstructor named __newobj__() as the first item of reduce
    > tuple. Something like this:
    >
    > >>> def __newobj__(cls, *args):

    >
    > ... return cls.__new__(cls, *args)
    > ...>>> class C(object):
    >
    > ... def __init__(self):
    > ... print "I shouldn't be called at reconstruction"
    > ... def __reduce__(self):
    > ... try:
    > ... getnewargs = self.__getnewargs__
    > ... except AttributeError:
    > ... newargs = (self.__class__,)
    > ... else:
    > ... newargs = (self.__class__,) + getnewargs()
    > ... try:
    > ... getstate = self.__getstate__
    > ... except AttributeError:
    > ... # this ignores __slots__ complications
    > ... state = self.__dict__
    > ... else:
    > ... state = getstate()
    > ... # this ignores list and dict subclasses
    > ... return __newobj__, newargs, state
    > ...>>> c = C()
    >
    > I shouldn't be called at reconstruction>>> importpickle
    > >>> for proto in range(3):

    >
    > ... assert isinstance(pickle.loads(pickle.dumps(c, proto)), C)
    > ...
    >
    >
    >
    > Ziga
    dgdev, Apr 18, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. a pickle's pickle

    , Aug 2, 2005, in forum: Python
    Replies:
    4
    Views:
    373
  2. Peter Bencsik
    Replies:
    2
    Views:
    813
  3. Michele Simionato
    Replies:
    2
    Views:
    1,862
    Michele Simionato
    May 23, 2008
  4. Paul  Moore
    Replies:
    2
    Views:
    259
    Paul Moore
    Nov 20, 2008
  5. Replies:
    1
    Views:
    566
    Tennessee
    Oct 14, 2009
Loading...

Share This Page