Pickling and unpickling inherited attributes

[Alex]

I have a serious problem and I hope there is some solution. It is
easier to illustrate with a simple code:
__slots__=['A', 'B']
def __init__(self, a, b):
self.A=a; self.B=b
def __getstate__(self):
return self.A, self.B
def __setstate__(self, tup):
self.A, self.B=tup

__slots__=['C',]
def __init__(self, c):
self.C=c
def __getstate__(self):
return self.C,
def __setstate__(self, tup):
self.C, =tup


Traceback (most recent call last):
File "<pyshell#60>", line 1, in -toplevel-
objct.Z=4
AttributeError: 'Child' object has no attribute 'Z'

So far so good.. Object obj inherited attributes (A and B) from the
parent class and refuses to take any new ones. But look what happens
when I try to pickle and unpickle it:

Traceback (most recent call last):
File "<pyshell#55>", line 1, in -toplevel-
objct.A
AttributeError: A1

Its own attribute (C) value is restored but the value of an inherited
attribute (A) is not. I tried pickling protocol 2, and module pickle
instead of cPickle, all with the same result.

What can be done?! Or maybe nothing can be done?

I would greatly appreciate an advice. A lot of code is written, but now
we have a need for pickling and unpickling objects and discovered this
problem.

 

[Alex]

Sorry, I copied and pasted a wrong piece from shell at one part of the
code:

objct.Z=4 was in fact obj.Z=4 and it did refuse to accept Z (because it
is not in __slots__).

But the question remains: why the value of attribute A is not preserved
during pickling and unpickling and what can be done about it, if
anything?

 

[Alex Martelli]

I have a serious problem and I hope there is some solution. It is
easier to illustrate with a simple code:
__slots__=['A', 'B']
def __init__(self, a, b):
self.A=a; self.B=b
def __getstate__(self):
return self.A, self.B
def __setstate__(self, tup):
self.A, self.B=tup

__slots__=['C',]
def __init__(self, c):
self.C=c
def __getstate__(self):
return self.C,
def __setstate__(self, tup):
self.C, =tup

Child.__getstate__ and __setstate__ need to cooperate with those of
Parent, not just override them as yours do. __getstate__ is easy:

in Child:

def __getstate__(self):
return super(Child, self).__getstate__ + (self.C,)

i.e., each class will successively append its parts of state to the
resulting overall tuple. It's harder to do for __setstate__ without
building into it a strong dependency on how many items of the tuple the
Parent is taking for its own use; you need to establish some protocol of
your own for that purpose, such as:

in Parent:

def __setstate__(self, tup):
self.A, self.B = tup[:2]
self._tup = tup[2:]

in Child:

def __setstate__(self, tup):
super(Child, self).__setstate__(tup)
self.C, = self._tup[:1]
self._tup = self._tup[1:]

(the last statement is needed in case Child was further subclassed by a
Grandchild class following the same set-state protocol).

Working with lists rather than tuples might in fact be simpler. But
simplest might be if __setstate__ was allowed to return something
different than None, so it could return "whatever's left of the tuple"
rather than relying on that nasty self._tup thingy (which is going to be
left there, empty, in the end -- though, each class could conditionally
delete it if it was empty). However, while I believe it is not
currently enforced that __setstate__ might return a non-None value to be
ignored, I'm not sure it's _allowed_, either, so I wouldn't try (lest
this application should break in strange ways in future Python
versions).


Alex

 

[Sam Pointon]

The reason the state of obj.A and obj.B aren't preserved is because
your __getstate__ and __setstate__ don't preserve them - they only save
obj.C, and you don't make a call to the parent class's respective
methods. Here's what I mean:

__slots__=['C',]
def __init__(self, c):
self.C=c
def __getstate__(self):
return Parent.__getstate__(self) + (self.C)
def __setstate__(self, tup):
self.C = tup.pop()
Parent.__setstate__(self, tup)
'foo'

 

[Sam Pointon]

Of course, in __setstate__, there should be a tup = list(tup) as well -
sheer forgetfulness and a confusing name in one.

 

[Alex]

Thanks so much! It makes perfect sense and I am sure it will work now.

 

[Steve Holden]

I have a serious problem and I hope there is some solution. It is
easier to illustrate with a simple code:


__slots__=['A', 'B']
def __init__(self, a, b):
self.A=a; self.B=b
def __getstate__(self):
return self.A, self.B
def __setstate__(self, tup):
self.A, self.B=tup



__slots__=['C',]
def __init__(self, c):
self.C=c
def __getstate__(self):
return self.C,
def __setstate__(self, tup):
self.C, =tup

2


3


1

Traceback (most recent call last):
File "<pyshell#60>", line 1, in -toplevel-
objct.Z=4
AttributeError: 'Child' object has no attribute 'Z'

So far so good.. Object obj inherited attributes (A and B) from the
parent class and refuses to take any new ones. But look what happens
when I try to pickle and unpickle it:



Traceback (most recent call last):
File "<pyshell#55>", line 1, in -toplevel-
objct.A
AttributeError: A

1

Its own attribute (C) value is restored but the value of an inherited
attribute (A) is not. I tried pickling protocol 2, and module pickle
instead of cPickle, all with the same result.

What can be done?! Or maybe nothing can be done?

I would greatly appreciate an advice. A lot of code is written, but now
we have a need for pickling and unpickling objects and discovered this
problem.

You have explicitly told the objects' class definition to only store the
C attribute as a part of the object state.

If you change the definition of child to:

class Child(Parent):

__slots__=['C',]
def __init__(self, c):
self.C=c
def __getstate__(self):
return self.A, self.B, self.C,
def __setstate__(self, tup):
self.A, self.B, self.C, =tup

everything works as you expect. But I presume you want the subclass to
require no knowledge of the superclass state?

In that case you might consider rewriting Child as

class Child(Parent):

__slots__ = ['C'] # only tuples need trailing comma
def __init__(self, c):
self.C = c
def __getstate__(self):
return Parent.__getstate__(self) + (self.C, )
def __setstate__(self, t):
self.C = t[-1]
Parent.__setstate__(self, t[:-1])

This would work adequately. May I ask your use case for __slots__? I
presume there is a specific reason why you can't just code the classes as

class Parent(object):
def __init__(self, a, b):
self.A = a
self.B = b

class Child(Parent):
def __init__(self, c):
self.C = c

since these definitions will pickle and unpickle perfectly.

A couple of other notes:

First, it's always safest to open pickle files in binary mode, as this
will eliminate the chance of platform differences making a pickle
written on one architecture or platform being unreadable by another.

Secondly, it's usual to have the subclass explicitly call the superclass
to initialize that portion of the instance. So it would be more
typically Pythonic to write

class Child(Parent):
def __init__(self, c):
Parent.__init__(self, a, b)
self.C = c

This allows you to pass the a and b values into the creator call in the
following way:

obj = Child(2, 3, 1)

rather than explicitly setting attributes of the Child instance in line.
If your inheritance patterns are complex you may find it useful to
investigate the super() function.

regards
Steve

 

[Alex]

Thanks to both Alex Martelli and Steve Holden.. We need __slots__ for
other reasons, too long to explain, but basically to prevent assignment
of extra attributes.

I ended up changing child classes this way:

def __getstate__(self):
return(Parent.__getstate__(self), self.C)
def __setstate__(self, data):
Parent.__setstate__(self, data[0])
self.C=data[1:]

This seems to work fine.

 

[Steve Holden]

Thanks to both Alex Martelli and Steve Holden.. We need __slots__ for
other reasons, too long to explain, but basically to prevent assignment
of extra attributes.

I ended up changing child classes this way:

def __getstate__(self):
return(Parent.__getstate__(self), self.C)
def __setstate__(self, data):
Parent.__setstate__(self, data[0])
self.C=data[1:]

This seems to work fine.

OK, but do be aware that slots was intended solely as a
memory-conservation measure where large numbers of objects are being
created. You are therefore subverting its true intent by using it to
limit the assignment of extra attributes (and there has been much recent
discussion on this list, though not in this thread, about what to do
instead.

regards
Steve

 

[Alex]

OK, but do be aware that slots was intended solely as a
memory-conservation measure where large numbers of objects are being
created. You are therefore subverting its true intent by using it to
limit the assignment of extra attributes (and there has been much recent
discussion on this list, though not in this thread, about what to do
instead.

regards
Steve
--

Oh, that too... We have tens of thousands of these objects in the
memory at the same moment, so memory conservation is another reason for
slots.