__getattr__, __setattr__ and pickle

Discussion in 'Python' started by Bruno Desthuilliers, Aug 12, 2008.

  1. mwojc a écrit :
    > Hi!
    > My class with implemented __getattr__ and __setattr__ methods cannot be
    > pickled because of the Error:
    >
    > ======================================================================
    > ERROR: testPickle (__main__.TestDeffnet2WithBiases)
    > ----------------------------------------------------------------------
    > Traceback (most recent call last):
    > File "deffnet.py", line 246, in testPickle
    > cPickle.dump(self.denet, file)
    > TypeError: 'NoneType' object is not callable
    >
    > ----------------------------------------------------------------------
    >
    > Is there an obvious reason i don't know, which prevents pickling with those
    > methods (if i comment them out the pickling test passes)?
    >
    > I'm using Python 2.4.4 on Gentoo Linux. The mentioned methods goes as
    > follows:
    >
    > def __setattr__(self, name, value):
    > if name == 'weights':
    > j = 0
    > for net in self.nets:
    > w1 = self.wmarks[j]
    > w2 = self.wmarks[j+1]
    > net.weights = value[w1:w2]
    > j += 1
    > else:
    > self.__dict__[name] = value
    >
    > def __getattr__(self, name):
    > if name == 'weights':
    > j = 0
    > for net in self.nets:
    > w1 = self.wmarks[j]
    > w2 = self.wmarks[j+1]
    > self._weights[w1:w2] = net.weights
    > j += 1
    > return self._weights


    __getattr__ should raise an AttributeError when name != 'weight' instead
    of (implicitely) returning None. pickle looks for a couple special
    method in your object[1], and it looks like it doesn't bother to check
    if what it found was really callable.

    [1] cf http://docs.python.org/lib/pickle-inst.html


    FWIW, you'd be better using a property instead of __getattr__ /
    __setattr__ if possible. And while we're at it, you dont need to
    manually take care of your index in the for loop - you can use
    enumerate(iterable) instead:

    for j, net in enumerate(self.nets):
    w1 = self.wmarks[j]
    w2 = self.wmarks[j+1]
    self._weights[w1:w2] = net.weights
    return self._weights


    HTH




    > Greetings,
    Bruno Desthuilliers, Aug 12, 2008
    #1
    1. Advertising

  2. Bruno Desthuilliers

    mwojc Guest

    Hi!
    My class with implemented __getattr__ and __setattr__ methods cannot be
    pickled because of the Error:

    ======================================================================
    ERROR: testPickle (__main__.TestDeffnet2WithBiases)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "deffnet.py", line 246, in testPickle
    cPickle.dump(self.denet, file)
    TypeError: 'NoneType' object is not callable

    ----------------------------------------------------------------------

    Is there an obvious reason i don't know, which prevents pickling with those
    methods (if i comment them out the pickling test passes)?

    I'm using Python 2.4.4 on Gentoo Linux. The mentioned methods goes as
    follows:

    def __setattr__(self, name, value):
    if name == 'weights':
    j = 0
    for net in self.nets:
    w1 = self.wmarks[j]
    w2 = self.wmarks[j+1]
    net.weights = value[w1:w2]
    j += 1
    else:
    self.__dict__[name] = value

    def __getattr__(self, name):
    if name == 'weights':
    j = 0
    for net in self.nets:
    w1 = self.wmarks[j]
    w2 = self.wmarks[j+1]
    self._weights[w1:w2] = net.weights
    j += 1
    return self._weights

    Greetings,
    --
    Marek
    mwojc, Aug 12, 2008
    #2
    1. Advertising

  3. Bruno Desthuilliers

    mwojc Guest

    Bruno Desthuilliers wrote:

    > mwojc a écrit :
    >> Hi!
    >> My class with implemented __getattr__ and __setattr__ methods cannot be
    >> pickled because of the Error:
    >>
    >> ======================================================================
    >> ERROR: testPickle (__main__.TestDeffnet2WithBiases)
    >> ----------------------------------------------------------------------
    >> Traceback (most recent call last):
    >> File "deffnet.py", line 246, in testPickle
    >> cPickle.dump(self.denet, file)
    >> TypeError: 'NoneType' object is not callable
    >>
    >> ----------------------------------------------------------------------
    >>
    >> Is there an obvious reason i don't know, which prevents pickling with
    >> those methods (if i comment them out the pickling test passes)?
    >>
    >> I'm using Python 2.4.4 on Gentoo Linux. The mentioned methods goes as
    >> follows:
    >>
    >> def __setattr__(self, name, value):
    >> if name == 'weights':
    >> j = 0
    >> for net in self.nets:
    >> w1 = self.wmarks[j]
    >> w2 = self.wmarks[j+1]
    >> net.weights = value[w1:w2]
    >> j += 1
    >> else:
    >> self.__dict__[name] = value
    >>
    >> def __getattr__(self, name):
    >> if name == 'weights':
    >> j = 0
    >> for net in self.nets:
    >> w1 = self.wmarks[j]
    >> w2 = self.wmarks[j+1]
    >> self._weights[w1:w2] = net.weights
    >> j += 1
    >> return self._weights

    >
    > __getattr__ should raise an AttributeError when name != 'weight' instead
    > of (implicitely) returning None. pickle looks for a couple special
    > method in your object[1], and it looks like it doesn't bother to check
    > if what it found was really callable.


    Yes, raising AttributeError helped. Thanks!

    >
    > FWIW, you'd be better using a property instead of __getattr__ /
    > __setattr__ if possible.


    You're probably right again, in this case it's better to use property.

    > And while we're at it, you dont need to
    > manually take care of your index in the for loop - you can use
    > enumerate(iterable) instead:
    >
    > for j, net in enumerate(self.nets):
    > w1 = self.wmarks[j]
    > w2 = self.wmarks[j+1]
    > self._weights[w1:w2] = net.weights
    > return self._weights
    >

    Sometimes i use manual handling of index because i'm convinced that
    enumeration is a bit slower than this. But i'm not really sure about it...

    Thanks again.

    Greetings,
    --
    Marek
    mwojc, Aug 12, 2008
    #3
  4. mwojc a écrit :
    > Bruno Desthuilliers wrote:

    (snip)
    >> FWIW, you'd be better using a property instead of __getattr__ /
    >> __setattr__ if possible.

    >
    > You're probably right again, in this case it's better to use property.


    Since you seem to have concerns wrt/ execution time, properties might be
    a little bit faster than __getattr__/__setattr__ hooks - cf below...

    >> And while we're at it, you dont need to
    >> manually take care of your index in the for loop - you can use
    >> enumerate(iterable) instead:
    >>
    >> for j, net in enumerate(self.nets):
    >> w1 = self.wmarks[j]
    >> w2 = self.wmarks[j+1]
    >> self._weights[w1:w2] = net.weights
    >> return self._weights
    >>

    > Sometimes i use manual handling of index because i'm convinced that
    > enumeration is a bit slower than this. But i'm not really sure about it...


    It's easy to try out:

    bruno@bibi ~ $ python
    Python 2.5.1 (r251:54863, Apr 6 2008, 17:20:35)
    [GCC 4.1.2 (Gentoo 4.1.2 p1.0.2)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> def manual(cnt):

    .... j = 0
    .... for i in xrange(cnt):
    .... x = i
    .... j += 1
    ....
    >>> def auto(cnt):

    .... for j, i in enumerate(xrange(cnt)):
    .... x = i
    ....
    >>> from timeit import Timer


    >>> tm = Timer("manual(10000)", "from __main__ import manual")
    >>> ta = Timer("auto(10000)", "from __main__ import auto")
    >>> tm.timeit(1000)

    3.3354489803314209
    >>> tm.timeit(1000)

    3.3376359939575195
    >>> tm.timeit(1000)

    3.3400180339813232
    >>> ta.timeit(1000)

    2.8350770473480225
    >>> ta.timeit(1000)

    2.8400650024414062
    >>> ta.timeit(1000)

    2.8361449241638184
    >>>


    Looks like enum is a bit faster by a mostly constant factor here.

    And while we're at it:

    >>> class Prop(object):

    .... @apply
    .... def prop():
    .... def fget(self): return self._prop
    .... def fset(self, val): self._prop = val
    .... return property(**locals())
    .... def __init__(self, val): self.prop=val
    ....
    >>> class Hook(object):

    .... def __getattr__(self, name):
    .... if name == 'prop':
    .... return self._prop
    .... raise AttributeError("yadda")
    .... def __setattr__(self, name, val):
    .... if name == 'prop':
    .... self.__dict__['_prop'] = val
    .... else:
    .... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
    .... self.__dict__[name] = val
    .... # correct implementation:
    .... # super(Hook, self).__setattr__(name, value)
    .... def __init__(self, val): self.prop=val
    ....
    >>> def testprop(cnt):

    .... p = Prop('test')
    .... for i in xrange(cnt):
    .... p.prop = i
    .... x = p.prop
    ....
    >>> def testhook(cnt):

    .... h = Hook('test')
    .... for i in xrange(cnt):
    .... h.prop = i
    .... x = h.prop
    ....
    >>> tp = Timer("testprop(1000)", "from __main__ import testprop")
    >>> th = Timer("testhook(1000)", "from __main__ import testhook")
    >>> tp.timeit(1000)

    3.0640909671783447
    >>> tp.timeit(1000)

    3.0650019645690918
    >>> th.timeit(1000)

    7.0889511108398438
    >>> th.timeit(1000)

    7.0815410614013672

    Looks like properties are significatively faster than the
    __getattr__/__setattr__ hook too... Which is probably explained by the
    following facts:

    Looking for binding descriptors (like read/write properties) is the
    first very stage of the attribute resolution algorithm (since they must
    be looked up before the instance's __dict__ to avoid setting an instance
    attribute which would then shadow the property).

    OTHO, __getattr__ is always looked for last - which means it takes
    longer to resolve __getattr__ than to resolve a read access to a
    read/write property.

    wrt/ __setattr__ - which is always called for attribute assignement,
    whatever -, overloading it (instead of relying on the optimized builtin
    implementation) is not only tricky (BTW, your implementation is broken -
    try to add a read/write property to your class and enjoy...[1]), but
    also cause a penalty for *all* attributes 'write' access.

    [1] oh, yes, should I mention it ? The correct implementation is even
    slower:
    >>> class Hook(object):

    .... def __getattr__(self, name):
    .... if name == 'prop':
    .... return self._prop
    .... raise AttributeError("yadda")
    .... def __setattr__(self, name, val):
    .... if name == 'prop':
    .... self.__dict__['_prop'] = val
    .... else:
    .... #self.__dict__[name] = val
    .... super(Hook, self).__setattr__(name, value)
    .... def __init__(self, val): self.prop=val
    ....
    >>> th.timeit(1000)

    7.1943540573120117
    >>> th.timeit(1000)

    7.1930480003356934
    >>>



    HTH
    Bruno Desthuilliers, Aug 12, 2008
    #4
  5. Bruno Desthuilliers

    mwojc Guest

    Bruno Desthuilliers wrote:

    > >>> class Prop(object):

    > ... @apply
    > ... def prop():
    > ... def fget(self): return self._prop
    > ... def fset(self, val): self._prop = val
    > ... return property(**locals())
    > ... def __init__(self, val): self.prop=val
    > ...
    > >>> class Hook(object):

    > ... def __getattr__(self, name):
    > ... if name == 'prop':
    > ... return self._prop
    > ... raise AttributeError("yadda")
    > ... def __setattr__(self, name, val):
    > ... if name == 'prop':
    > ... self.__dict__['_prop'] = val
    > ... else:
    > ... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
    > ... self.__dict__[name] = val
    > ... # correct implementation:
    > ... # super(Hook, self).__setattr__(name, value)
    > ... def __init__(self, val): self.prop=val


    Hi!
    Finally i ended up with all the suggestions you gave me. The speed is
    important to me besause i get/set my attribute a lot... The only doubts i
    have are with the @apply decorator, because 'apply' seems to be depreciated
    since python 2.3... And another thing: why writing for all
    attributes 'self.__dict__[name] = val' in __setattr__ is incorrect?

    Thanks a lot!

    Greetings,
    --
    Marek
    mwojc, Aug 13, 2008
    #5
  6. On Aug 12, 7:28 pm, mwojc <> wrote:
    > Hi!
    > My class with implemented __getattr__ and __setattr__ methods cannot be
    > pickled because of the Error:


    Another option is to define __getstate__ on your class:

    def __getstate__(self): return vars(self)


    M.S.
    Michele Simionato, Aug 13, 2008
    #6
  7. mwojc a écrit :
    > Bruno Desthuilliers wrote:
    >
    >> >>> class Prop(object):

    >> ... @apply
    >> ... def prop():
    >> ... def fget(self): return self._prop
    >> ... def fset(self, val): self._prop = val
    >> ... return property(**locals())
    >> ... def __init__(self, val): self.prop=val
    >> ...
    >> >>> class Hook(object):

    >> ... def __getattr__(self, name):
    >> ... if name == 'prop':
    >> ... return self._prop
    >> ... raise AttributeError("yadda")
    >> ... def __setattr__(self, name, val):
    >> ... if name == 'prop':
    >> ... self.__dict__['_prop'] = val
    >> ... else:
    >> ... # XXX : INCORRECT IMPLEMENTATION, DONT DO THIS !
    >> ... self.__dict__[name] = val
    >> ... # correct implementation:
    >> ... # super(Hook, self).__setattr__(name, value)
    >> ... def __init__(self, val): self.prop=val

    >
    > Hi!
    > Finally i ended up with all the suggestions you gave me. The speed is
    > important to me besause i get/set my attribute a lot... The only doubts i
    > have are with the @apply decorator, because 'apply' seems to be depreciated
    > since python 2.3...


    As far as I can tell, it should stay until Python 3.0. But you don't
    have to write your properties that way anyway - it's just something I
    personnaly find to be a convenient shortcut, period. FWIW, the above
    code is just a copy/paste from an interactive session, not an example of
    "clean" Python coding.

    > And another thing: why writing for all
    > attributes 'self.__dict__[name] = val' in __setattr__ is incorrect?


    Because __setattr__ is *always* called. With your above solution,
    read/write properties setters won't be called.

    Python 2.5.1 (r251:54863, Mar 7 2008, 03:41:45)
    [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> class Foo(object):

    @apply
    def bar():
    def fget(self):
    print "bar get"
    return self._bar
    def fset(self, val):
    print "bar set"
    self._bar = val
    def __getattr__(self, name):
    print "__getattr__", name
    if name == 'baaz':
    return self._baaz
    raise AttributeError("yadda")
    def __setattr__(self, name, val):
    print "__setattr__", name
    self.__dict__[name] = val
    >>> foo = Foo('bar', 'baaz')

    __setattr__ bar
    __setattr__ baaz
    >>> foo.baaz

    'baaz'
    >>> foo.bar

    'bar'
    >>> foo.__dict__

    {'baaz': 'baaz', 'bar': 'bar'}
    Bruno Desthuilliers, Aug 13, 2008
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Anand
    Replies:
    0
    Views:
    407
    Anand
    Dec 18, 2003
  2. a pickle's pickle

    , Aug 2, 2005, in forum: Python
    Replies:
    4
    Views:
    382
  3. Thomas Heller

    __getattr__, __setattr__

    Thomas Heller, Oct 20, 2005, in forum: Python
    Replies:
    3
    Views:
    678
    Thomas Heller
    Oct 20, 2005
  4. Michele Simionato
    Replies:
    2
    Views:
    1,871
    Michele Simionato
    May 23, 2008
  5. aum
    Replies:
    1
    Views:
    134
Loading...

Share This Page