Pickling and unpickling inherited attributes

Discussion in 'Python' started by Alex, Oct 30, 2005.

  1. Alex

    Alex Guest

    I have a serious problem and I hope there is some solution. It is
    easier to illustrate with a simple code:

    >>> class Parent(object):

    __slots__=['A', 'B']
    def __init__(self, a, b):
    self.A=a; self.B=b
    def __getstate__(self):
    return self.A, self.B
    def __setstate__(self, tup):
    self.A, self.B=tup


    >>> class Child(Parent):

    __slots__=['C',]
    def __init__(self, c):
    self.C=c
    def __getstate__(self):
    return self.C,
    def __setstate__(self, tup):
    self.C, =tup


    >>> obj=Child(1)
    >>> obj.A=2
    >>> obj.B=3
    >>> obj.A

    2
    >>> obj.B

    3
    >>> obj.C

    1
    >>> objct.Z=4


    Traceback (most recent call last):
    File "<pyshell#60>", line 1, in -toplevel-
    objct.Z=4
    AttributeError: 'Child' object has no attribute 'Z'

    So far so good.. Object obj inherited attributes (A and B) from the
    parent class and refuses to take any new ones. But look what happens
    when I try to pickle and unpickle it:

    >>> import cPickle
    >>> File=open('test', 'w')
    >>> cPickle.dump(obj, File)
    >>> File.close()
    >>>
    >>> file1=open('test', 'r')
    >>> objct=cPickle.load(file1)
    >>> file1.close()
    >>> objct.A


    Traceback (most recent call last):
    File "<pyshell#55>", line 1, in -toplevel-
    objct.A
    AttributeError: A
    >>> objct.C

    1

    Its own attribute (C) value is restored but the value of an inherited
    attribute (A) is not. I tried pickling protocol 2, and module pickle
    instead of cPickle, all with the same result.

    What can be done?! Or maybe nothing can be done?

    I would greatly appreciate an advice. A lot of code is written, but now
    we have a need for pickling and unpickling objects and discovered this
    problem.
    Alex, Oct 30, 2005
    #1
    1. Advertising

  2. Alex

    Alex Guest

    Sorry, I copied and pasted a wrong piece from shell at one part of the
    code:

    objct.Z=4 was in fact obj.Z=4 and it did refuse to accept Z (because it
    is not in __slots__).

    But the question remains: why the value of attribute A is not preserved
    during pickling and unpickling and what can be done about it, if
    anything?
    Alex, Oct 30, 2005
    #2
    1. Advertising

  3. Alex <> wrote:

    > I have a serious problem and I hope there is some solution. It is
    > easier to illustrate with a simple code:
    >
    > >>> class Parent(object):

    > __slots__=['A', 'B']
    > def __init__(self, a, b):
    > self.A=a; self.B=b
    > def __getstate__(self):
    > return self.A, self.B
    > def __setstate__(self, tup):
    > self.A, self.B=tup
    >
    >
    > >>> class Child(Parent):

    > __slots__=['C',]
    > def __init__(self, c):
    > self.C=c
    > def __getstate__(self):
    > return self.C,
    > def __setstate__(self, tup):
    > self.C, =tup


    Child.__getstate__ and __setstate__ need to cooperate with those of
    Parent, not just override them as yours do. __getstate__ is easy:

    in Child:

    def __getstate__(self):
    return super(Child, self).__getstate__ + (self.C,)

    i.e., each class will successively append its parts of state to the
    resulting overall tuple. It's harder to do for __setstate__ without
    building into it a strong dependency on how many items of the tuple the
    Parent is taking for its own use; you need to establish some protocol of
    your own for that purpose, such as:

    in Parent:

    def __setstate__(self, tup):
    self.A, self.B = tup[:2]
    self._tup = tup[2:]

    in Child:

    def __setstate__(self, tup):
    super(Child, self).__setstate__(tup)
    self.C, = self._tup[:1]
    self._tup = self._tup[1:]

    (the last statement is needed in case Child was further subclassed by a
    Grandchild class following the same set-state protocol).

    Working with lists rather than tuples might in fact be simpler. But
    simplest might be if __setstate__ was allowed to return something
    different than None, so it could return "whatever's left of the tuple"
    rather than relying on that nasty self._tup thingy (which is going to be
    left there, empty, in the end -- though, each class could conditionally
    delete it if it was empty). However, while I believe it is not
    currently enforced that __setstate__ might return a non-None value to be
    ignored, I'm not sure it's _allowed_, either, so I wouldn't try (lest
    this application should break in strange ways in future Python
    versions).


    Alex
    Alex Martelli, Oct 31, 2005
    #3
  4. Alex

    Sam Pointon Guest

    The reason the state of obj.A and obj.B aren't preserved is because
    your __getstate__ and __setstate__ don't preserve them - they only save
    obj.C, and you don't make a call to the parent class's respective
    methods. Here's what I mean:

    >>> import pickle
    >>> class Child(Parent):


    __slots__=['C',]
    def __init__(self, c):
    self.C=c
    def __getstate__(self):
    return Parent.__getstate__(self) + (self.C)
    def __setstate__(self, tup):
    self.C = tup.pop()
    Parent.__setstate__(self, tup)

    >>> obj = Child('foo')
    >>> obj.A = 'bar'
    >>> obj.B = 'baz'
    >>> objct = pickle.loads(pickle.dumps(obj))
    >>> objct.A

    'bar'
    >>> objct.B

    'baz'
    >>> objct.C

    'foo'
    Sam Pointon, Oct 31, 2005
    #4
  5. Alex

    Sam Pointon Guest

    Of course, in __setstate__, there should be a tup = list(tup) as well -
    sheer forgetfulness and a confusing name in one.
    Sam Pointon, Oct 31, 2005
    #5
  6. Alex

    Alex Guest

    Thanks so much! It makes perfect sense and I am sure it will work now.
    Alex, Oct 31, 2005
    #6
  7. Alex

    Steve Holden Guest

    Alex wrote:
    > I have a serious problem and I hope there is some solution. It is
    > easier to illustrate with a simple code:
    >
    >
    >>>>class Parent(object):

    >
    > __slots__=['A', 'B']
    > def __init__(self, a, b):
    > self.A=a; self.B=b
    > def __getstate__(self):
    > return self.A, self.B
    > def __setstate__(self, tup):
    > self.A, self.B=tup
    >
    >
    >
    >>>>class Child(Parent):

    >
    > __slots__=['C',]
    > def __init__(self, c):
    > self.C=c
    > def __getstate__(self):
    > return self.C,
    > def __setstate__(self, tup):
    > self.C, =tup
    >
    >
    >
    >>>>obj=Child(1)
    >>>>obj.A=2
    >>>>obj.B=3
    >>>>obj.A

    >
    > 2
    >
    >>>>obj.B

    >
    > 3
    >
    >>>>obj.C

    >
    > 1
    >
    >>>>objct.Z=4

    >
    >
    > Traceback (most recent call last):
    > File "<pyshell#60>", line 1, in -toplevel-
    > objct.Z=4
    > AttributeError: 'Child' object has no attribute 'Z'
    >
    > So far so good.. Object obj inherited attributes (A and B) from the
    > parent class and refuses to take any new ones. But look what happens
    > when I try to pickle and unpickle it:
    >
    >
    >>>>import cPickle
    >>>>File=open('test', 'w')
    >>>>cPickle.dump(obj, File)
    >>>>File.close()
    >>>>
    >>>>file1=open('test', 'r')
    >>>>objct=cPickle.load(file1)
    >>>>file1.close()
    >>>>objct.A

    >
    >
    > Traceback (most recent call last):
    > File "<pyshell#55>", line 1, in -toplevel-
    > objct.A
    > AttributeError: A
    >
    >>>>objct.C

    >
    > 1
    >
    > Its own attribute (C) value is restored but the value of an inherited
    > attribute (A) is not. I tried pickling protocol 2, and module pickle
    > instead of cPickle, all with the same result.
    >
    > What can be done?! Or maybe nothing can be done?
    >
    > I would greatly appreciate an advice. A lot of code is written, but now
    > we have a need for pickling and unpickling objects and discovered this
    > problem.
    >

    You have explicitly told the objects' class definition to only store the
    C attribute as a part of the object state.

    If you change the definition of child to:

    class Child(Parent):

    __slots__=['C',]
    def __init__(self, c):
    self.C=c
    def __getstate__(self):
    return self.A, self.B, self.C,
    def __setstate__(self, tup):
    self.A, self.B, self.C, =tup

    everything works as you expect. But I presume you want the subclass to
    require no knowledge of the superclass state?

    In that case you might consider rewriting Child as

    class Child(Parent):

    __slots__ = ['C'] # only tuples need trailing comma
    def __init__(self, c):
    self.C = c
    def __getstate__(self):
    return Parent.__getstate__(self) + (self.C, )
    def __setstate__(self, t):
    self.C = t[-1]
    Parent.__setstate__(self, t[:-1])

    This would work adequately. May I ask your use case for __slots__? I
    presume there is a specific reason why you can't just code the classes as

    class Parent(object):
    def __init__(self, a, b):
    self.A = a
    self.B = b

    class Child(Parent):
    def __init__(self, c):
    self.C = c

    since these definitions will pickle and unpickle perfectly.

    A couple of other notes:

    First, it's always safest to open pickle files in binary mode, as this
    will eliminate the chance of platform differences making a pickle
    written on one architecture or platform being unreadable by another.

    Secondly, it's usual to have the subclass explicitly call the superclass
    to initialize that portion of the instance. So it would be more
    typically Pythonic to write

    class Child(Parent):
    def __init__(self, c):
    Parent.__init__(self, a, b)
    self.C = c

    This allows you to pass the a and b values into the creator call in the
    following way:

    obj = Child(2, 3, 1)

    rather than explicitly setting attributes of the Child instance in line.
    If your inheritance patterns are complex you may find it useful to
    investigate the super() function.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC www.holdenweb.com
    PyCon TX 2006 www.python.org/pycon/
    Steve Holden, Oct 31, 2005
    #7
  8. Alex

    Alex Guest

    Thanks to both Alex Martelli and Steve Holden.. We need __slots__ for
    other reasons, too long to explain, but basically to prevent assignment
    of extra attributes.

    I ended up changing child classes this way:

    def __getstate__(self):
    return(Parent.__getstate__(self), self.C)
    def __setstate__(self, data):
    Parent.__setstate__(self, data[0])
    self.C=data[1:]

    This seems to work fine.
    Alex, Nov 1, 2005
    #8
  9. Alex

    Steve Holden Guest

    Alex wrote:
    > Thanks to both Alex Martelli and Steve Holden.. We need __slots__ for
    > other reasons, too long to explain, but basically to prevent assignment
    > of extra attributes.
    >
    > I ended up changing child classes this way:
    >
    > def __getstate__(self):
    > return(Parent.__getstate__(self), self.C)
    > def __setstate__(self, data):
    > Parent.__setstate__(self, data[0])
    > self.C=data[1:]
    >
    > This seems to work fine.
    >

    OK, but do be aware that slots was intended solely as a
    memory-conservation measure where large numbers of objects are being
    created. You are therefore subverting its true intent by using it to
    limit the assignment of extra attributes (and there has been much recent
    discussion on this list, though not in this thread, about what to do
    instead.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC www.holdenweb.com
    PyCon TX 2006 www.python.org/pycon/
    Steve Holden, Nov 1, 2005
    #9
  10. Alex

    Alex Guest


    > >

    > OK, but do be aware that slots was intended solely as a
    > memory-conservation measure where large numbers of objects are being
    > created. You are therefore subverting its true intent by using it to
    > limit the assignment of extra attributes (and there has been much recent
    > discussion on this list, though not in this thread, about what to do
    > instead.
    >
    > regards
    > Steve
    > --



    Oh, that too... We have tens of thousands of these objects in the
    memory at the same moment, so memory conservation is another reason for
    slots.
    Alex, Nov 2, 2005
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Bob
    Replies:
    1
    Views:
    353
    Martin v. =?iso-8859-15?q?L=F6wis?=
    Sep 11, 2003
  2. Thomas Guettler

    Pickling object inherited from dict

    Thomas Guettler, Oct 27, 2003, in forum: Python
    Replies:
    1
    Views:
    293
    Thomas Guettler
    Oct 28, 2003
  3. Thomas Guettler

    Pickling Objects inherited from dict (Bug?)

    Thomas Guettler, Nov 5, 2003, in forum: Python
    Replies:
    1
    Views:
    297
    Michael Hudson
    Nov 5, 2003
  4. Erwin S. Andreasen
    Replies:
    3
    Views:
    323
    Erwin S. Andreasen
    Feb 24, 2006
  5. Replies:
    3
    Views:
    297
    Peter Hansen
    Apr 6, 2006
Loading...

Share This Page