__getattr__, __setattr__ and pickle

  • Thread starter Bruno Desthuilliers
  • Start date
B

Bruno Desthuilliers

mwojc a écrit :
Hi!
My class with implemented __getattr__ and __setattr__ methods cannot be
pickled because of the Error:

======================================================================
ERROR: testPickle (__main__.TestDeffnet2WithBiases)
----------------------------------------------------------------------
Traceback (most recent call last):
File "deffnet.py", line 246, in testPickle
cPickle.dump(self.denet, file)
TypeError: 'NoneType' object is not callable

----------------------------------------------------------------------

Is there an obvious reason i don't know, which prevents pickling with those
methods (if i comment them out the pickling test passes)?

I'm using Python 2.4.4 on Gentoo Linux. The mentioned methods goes as
follows:

def __setattr__(self, name, value):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
net.weights = value[w1:w2]
j += 1
else:
self.__dict__[name] = value

def __getattr__(self, name):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
j += 1
return self._weights

__getattr__ should raise an AttributeError when name != 'weight' instead
of (implicitely) returning None. pickle looks for a couple special
method in your object[1], and it looks like it doesn't bother to check
if what it found was really callable.

[1] cf http://docs.python.org/lib/pickle-inst.html


FWIW, you'd be better using a property instead of __getattr__ /
__setattr__ if possible. And while we're at it, you dont need to
manually take care of your index in the for loop - you can use
enumerate(iterable) instead:

for j, net in enumerate(self.nets):
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
return self._weights


HTH
 
M

mwojc

Hi!
My class with implemented __getattr__ and __setattr__ methods cannot be
pickled because of the Error:

======================================================================
ERROR: testPickle (__main__.TestDeffnet2WithBiases)
----------------------------------------------------------------------
Traceback (most recent call last):
File "deffnet.py", line 246, in testPickle
cPickle.dump(self.denet, file)
TypeError: 'NoneType' object is not callable

----------------------------------------------------------------------

Is there an obvious reason i don't know, which prevents pickling with those
methods (if i comment them out the pickling test passes)?

I'm using Python 2.4.4 on Gentoo Linux. The mentioned methods goes as
follows:

def __setattr__(self, name, value):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
net.weights = value[w1:w2]
j += 1
else:
self.__dict__[name] = value

def __getattr__(self, name):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
j += 1
return self._weights

Greetings,
 
M

mwojc

Bruno said:
mwojc a écrit :
Hi!
My class with implemented __getattr__ and __setattr__ methods cannot be
pickled because of the Error:

======================================================================
ERROR: testPickle (__main__.TestDeffnet2WithBiases)
----------------------------------------------------------------------
Traceback (most recent call last):
File "deffnet.py", line 246, in testPickle
cPickle.dump(self.denet, file)
TypeError: 'NoneType' object is not callable

----------------------------------------------------------------------

Is there an obvious reason i don't know, which prevents pickling with
those methods (if i comment them out the pickling test passes)?

I'm using Python 2.4.4 on Gentoo Linux. The mentioned methods goes as
follows:

def __setattr__(self, name, value):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
net.weights = value[w1:w2]
j += 1
else:
self.__dict__[name] = value

def __getattr__(self, name):
if name == 'weights':
j = 0
for net in self.nets:
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
j += 1
return self._weights

__getattr__ should raise an AttributeError when name != 'weight' instead
of (implicitely) returning None. pickle looks for a couple special
method in your object[1], and it looks like it doesn't bother to check
if what it found was really callable.

Yes, raising AttributeError helped. Thanks!
FWIW, you'd be better using a property instead of __getattr__ /
__setattr__ if possible.

You're probably right again, in this case it's better to use property.
And while we're at it, you dont need to
manually take care of your index in the for loop - you can use
enumerate(iterable) instead:

for j, net in enumerate(self.nets):
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
return self._weights
Sometimes i use manual handling of index because i'm convinced that
enumeration is a bit slower than this. But i'm not really sure about it...

Thanks again.

Greetings,
 
B

Bruno Desthuilliers

mwojc a écrit :
Bruno Desthuilliers wrote: (snip)

You're probably right again, in this case it's better to use property.

Since you seem to have concerns wrt/ execution time, properties might be
a little bit faster than __getattr__/__setattr__ hooks - cf below...
And while we're at it, you dont need to
manually take care of your index in the for loop - you can use
enumerate(iterable) instead:

for j, net in enumerate(self.nets):
w1 = self.wmarks[j]
w2 = self.wmarks[j+1]
self._weights[w1:w2] = net.weights
return self._weights
Sometimes i use manual handling of index because i'm convinced that
enumeration is a bit slower than this. But i'm not really sure about it...

It's easy to try out:

bruno@bibi ~ $ python
Python 2.5.1 (r251:54863, Apr 6 2008, 17:20:35)
[GCC 4.1.2 (Gentoo 4.1.2 p1.0.2)] on linux2
Type "help", "copyright", "credits" or "license" for more information..... j = 0
.... for i in xrange(cnt):
.... x = i
.... j += 1
........ for j, i in enumerate(xrange(cnt)):
.... x = i
....
Looks like enum is a bit faster by a mostly constant factor here.

And while we're at it:
.... @apply
.... def prop():
.... def fget(self): return self._prop
.... def fset(self, val): self._prop = val
.... return property(**locals())
.... def __init__(self, val): self.prop=val
........ def __getattr__(self, name):
.... if name == 'prop':
.... return self._prop
.... raise AttributeError("yadda")
.... def __setattr__(self, name, val):
.... if name == 'prop':
.... self.__dict__['_prop'] = val
.... else:
.... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
.... self.__dict__[name] = val
.... # correct implementation:
.... # super(Hook, self).__setattr__(name, value)
.... def __init__(self, val): self.prop=val
........ p = Prop('test')
.... for i in xrange(cnt):
.... p.prop = i
.... x = p.prop
........ h = Hook('test')
.... for i in xrange(cnt):
.... h.prop = i
.... x = h.prop
....7.0815410614013672

Looks like properties are significatively faster than the
__getattr__/__setattr__ hook too... Which is probably explained by the
following facts:

Looking for binding descriptors (like read/write properties) is the
first very stage of the attribute resolution algorithm (since they must
be looked up before the instance's __dict__ to avoid setting an instance
attribute which would then shadow the property).

OTHO, __getattr__ is always looked for last - which means it takes
longer to resolve __getattr__ than to resolve a read access to a
read/write property.

wrt/ __setattr__ - which is always called for attribute assignement,
whatever -, overloading it (instead of relying on the optimized builtin
implementation) is not only tricky (BTW, your implementation is broken -
try to add a read/write property to your class and enjoy...[1]), but
also cause a penalty for *all* attributes 'write' access.

[1] oh, yes, should I mention it ? The correct implementation is even
slower:.... def __getattr__(self, name):
.... if name == 'prop':
.... return self._prop
.... raise AttributeError("yadda")
.... def __setattr__(self, name, val):
.... if name == 'prop':
.... self.__dict__['_prop'] = val
.... else:
.... #self.__dict__[name] = val
.... super(Hook, self).__setattr__(name, value)
.... def __init__(self, val): self.prop=val
....

HTH
 
M

mwojc

Bruno said:
... @apply
... def prop():
... def fget(self): return self._prop
... def fset(self, val): self._prop = val
... return property(**locals())
... def __init__(self, val): self.prop=val
...... def __getattr__(self, name):
... if name == 'prop':
... return self._prop
... raise AttributeError("yadda")
... def __setattr__(self, name, val):
... if name == 'prop':
... self.__dict__['_prop'] = val
... else:
... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
... self.__dict__[name] = val
... # correct implementation:
... # super(Hook, self).__setattr__(name, value)
... def __init__(self, val): self.prop=val

Hi!
Finally i ended up with all the suggestions you gave me. The speed is
important to me besause i get/set my attribute a lot... The only doubts i
have are with the @apply decorator, because 'apply' seems to be depreciated
since python 2.3... And another thing: why writing for all
attributes 'self.__dict__[name] = val' in __setattr__ is incorrect?

Thanks a lot!

Greetings,
 
M

Michele Simionato

Hi!
My class with implemented __getattr__ and __setattr__ methods cannot be
pickled because of the Error:

Another option is to define __getstate__ on your class:

def __getstate__(self): return vars(self)


M.S.
 
B

Bruno Desthuilliers

mwojc a écrit :
Bruno said:
class Prop(object):
... @apply
... def prop():
... def fget(self): return self._prop
... def fset(self, val): self._prop = val
... return property(**locals())
... def __init__(self, val): self.prop=val
...
class Hook(object):
... def __getattr__(self, name):
... if name == 'prop':
... return self._prop
... raise AttributeError("yadda")
... def __setattr__(self, name, val):
... if name == 'prop':
... self.__dict__['_prop'] = val
... else:
... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
... self.__dict__[name] = val
... # correct implementation:
... # super(Hook, self).__setattr__(name, value)
... def __init__(self, val): self.prop=val

Hi!
Finally i ended up with all the suggestions you gave me. The speed is
important to me besause i get/set my attribute a lot... The only doubts i
have are with the @apply decorator, because 'apply' seems to be depreciated
since python 2.3...

As far as I can tell, it should stay until Python 3.0. But you don't
have to write your properties that way anyway - it's just something I
personnaly find to be a convenient shortcut, period. FWIW, the above
code is just a copy/paste from an interactive session, not an example of
"clean" Python coding.
And another thing: why writing for all
attributes 'self.__dict__[name] = val' in __setattr__ is incorrect?

Because __setattr__ is *always* called. With your above solution,
read/write properties setters won't be called.

Python 2.5.1 (r251:54863, Mar 7 2008, 03:41:45)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information. @apply
def bar():
def fget(self):
print "bar get"
return self._bar
def fset(self, val):
print "bar set"
self._bar = val
def __getattr__(self, name):
print "__getattr__", name
if name == 'baaz':
return self._baaz
raise AttributeError("yadda")
def __setattr__(self, name, val):
print "__setattr__", name
self.__dict__[name] = val__setattr__ bar
__setattr__ baaz{'baaz': 'baaz', 'bar': 'bar'}
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top