Dynamic use of property() fails

[andrew cooke]

Hi,

This is my first attempt at new classes and dynamic python, so I am
probably doing something very stupid... After reading the how-to for
descriptors at http://users.rcn.com/python/download/Descriptor.htm I
decided I would make an object that returns attributes on read, but on
setting calls an arbitrary function.

My code looks like:
class ActiveDAO(object):
def __init__(self):
self.__values__ = {}
def add_field(self, name, value, on_change):
self.__values__[name] = value
def get(self): return self.__values__[name]
def set(self, new_value): self.__values__[name] =
on_change(new_value)
def delete(self): raise AttributeError
self.__dict__[name] = property(get, set, delete)

However, when I try to use this (in a test) with code like:
dao = ActiveDAO()
dao.add_field("name", "value", lambda _: None)
assertEqual(dao.name, "value")

I get a failure because lookup of the attribute is returning
"<property object at 0x6b8910>".

That is quite reasonable, but I was under the expression that some
magic was supposed to happen, as described in the document referenced
above!

Please can someone explain why there is no magic? (

Thanks,
Andrew

PS I am using Python 2.5 on Linux

 

[Gabriel Genellina]

This is my first attempt at new classes and dynamic python, so I am
probably doing something very stupid... After reading the how-to for
descriptors at http://users.rcn.com/python/download/Descriptor.htm I
decided I would make an object that returns attributes on read, but on
setting calls an arbitrary function.

My code looks like:
class ActiveDAO(object):
def __init__(self):
self.__values__ = {}
def add_field(self, name, value, on_change):
self.__values__[name] = value
def get(self): return self.__values__[name]
def set(self, new_value): self.__values__[name] =
on_change(new_value)
def delete(self): raise AttributeError
self.__dict__[name] = property(get, set, delete)

However, when I try to use this (in a test) with code like:
dao = ActiveDAO()
dao.add_field("name", "value", lambda _: None)
assertEqual(dao.name, "value")

I get a failure because lookup of the attribute is returning
"<property object at 0x6b8910>".

That is quite reasonable, but I was under the expression that some
magic was supposed to happen, as described in the document referenced
above!

The "magic" happens when the descriptor is found in the *class*, not in
the instance. I think it's detailed in Hettinger's document.
Do you actually want "per-instance" defined properties?

__special__ names are reserved for Python internal usage; don't use them.
Implementation-only attributes ("private" ones) are spelled with a single
underscore.

 

[andrew cooke]

[Gabriel:]

The "magic" happens when the descriptor is found in the *class*, not in
the instance. I think it's detailed in Hettinger's document.
Do you actually want "per-instance" defined properties?

ah! ok. yes, you're right: i want instance values but class
properties, so i'll rethink things. thanks very much!

__special__ names are reserved for Python internal usage; don't use them.
Implementation-only attributes ("private" ones) are spelled with a single
underscore.

ah, yes, sorry about that.

thanks again for the quick reply,
andrew

 

[Hrvoje Niksic]

The "magic" happens when the descriptor is found in the *class*, not
in the instance. I think it's detailed in Hettinger's document.

The document is wrong here:

Alternatively, it is more common for a descriptor to be invoked
automatically upon attribute access. For example, obj.d looks up d
in the dictionary of obj. If d defines the method __get__, then
d.__get__(obj) is invoked according to the precedence rules listed
below.

This sounds plausible and might have led Andrew to believe that his
code should work. It should instead say "in the dictionary of obj's
type" or something to that effect. I asked Raymond about it some time
ago, and he agreed that it's an error, but he apparently didn't get
around to fixing it.

The actual code examples in the document are, of course, correct.

 

[Hrvoje Niksic]

This is my first attempt at new classes and dynamic python, so I am
probably doing something very stupid... After reading the how-to for
descriptors at http://users.rcn.com/python/download/Descriptor.htm I
decided I would make an object that returns attributes on read, but on
setting calls an arbitrary function.

My code looks like:
class ActiveDAO(object):
def __init__(self):
self.__values__ = {}
def add_field(self, name, value, on_change):
self.__values__[name] = value
def get(self): return self.__values__[name]
def set(self, new_value): self.__values__[name] =
on_change(new_value)
def delete(self): raise AttributeError
self.__dict__[name] = property(get, set, delete)

As others explained, descriptors are called for descriptors found in
class attributes, not in ones in instance attributes. Calling them
for the latter would be dangerous because it might accidentally invoke
magic whenever you store the "wrong" kind of object in an instance
attribute. Also, in many cases (slots), instance property access is
*implemented* using class property descriptors, so calling descriptors
on objects retrieved from the instance would mean that the descriptor
would have to be invoked twice.

However, if you know what you're doing, you can simply customize your
class's __getattribute__ to do what *you* want for your objects. For
example:

def __getattribute__(self, name):
dict = object.__getattribute__(self, '__dict__') # self.__dict__ would infloop
if name in dict:
o = dict[name]
# call the descriptor even if found in an object in __dict__
if hasattr(o, '__get__'):
return o.__get__(self, type(self))
return o
return object.__getattribute__(self, name)

With that addition:

dao = ActiveDAO()
dao.add_field('name', 'value', lambda _: None)
dao.name 'value'
dao.__dict__['name']

<property object at 0xb7d53284>

 

[Bruno Desthuilliers]

andrew cooke a écrit :

Hi,

This is my first attempt at new classes and dynamic python, so I am
probably doing something very stupid... After reading the how-to for
descriptors at http://users.rcn.com/python/download/Descriptor.htm I
decided I would make an object that returns attributes on read, but on
setting calls an arbitrary function.

My code looks like:
class ActiveDAO(object):
def __init__(self):
self.__values__ = {}

__names__ are reserved for the Python implementation itself. Use _names
for 'protected' attributes.

def add_field(self, name, value, on_change):
self.__values__[name] = value
def get(self): return self.__values__[name]
def set(self, new_value): self.__values__[name] =
on_change(new_value)
def delete(self): raise AttributeError
self.__dict__[name] = property(get, set, delete)

However, when I try to use this (in a test) with code like:
dao = ActiveDAO()
dao.add_field("name", "value", lambda _: None)
assertEqual(dao.name, "value")

I get a failure because lookup of the attribute is returning
"<property object at 0x6b8910>".

That is quite reasonable, but I was under the expression that some
magic was supposed to happen, as described in the document referenced
above!

Please can someone explain why there is no magic? (

Others already answered this.

The canonical solution is to use a custom descriptor instead of a property:


class Field(object):
def __init__(self, name, onchange):
self.name = name
self.onchange = onchange

def __get__(self, instance, cls):
if instance is None:
# called on the class
return self
# called on instance
return instance._values[self.name]

def __set__(self, instance, value):
instance._values[name] = self.onchange(value)



class ActiveDAO(object):
def __init__(self):
self._values = []


class Person(ActiveDAO):
name = Field('firstname', lambda v: v.strip().capitalize())
age = Field('age', lambda v : int(v))

Now you may want to search here or in the cookbook to learn how to:
- dynamically create new classes
- avoid having to repeat the name of the field (usually done using a
metaclass and a two-stages initialisation of Field objects)


HTH

 

[Bruno Desthuilliers]

Hrvoje Niksic a écrit :
(snip)

As others explained, descriptors are called for descriptors found in
class attributes, not in ones in instance attributes. (snip)

However, if you know what you're doing, you can simply customize your
class's __getattribute__ to do what *you* want for your objects.

<op>
But bear in mind that, beside possible unwanted side-effectn, you'll get
a non-negligible performance hit - __getattribute__ being, as the name
implies, invoked on each and every attribute lookup.

FWIW, I tried this approach once, and quickly came back to a simpler
solution.

</op>
(snip code)

 

[andrew cooke]

The canonical solution is to use a custom descriptor instead of a property:

class Field(object):
def __init__(self, name, onchange):
self.name = name
self.onchange = onchange

def __get__(self, instance, cls):
if instance is None:
# called on the class
return self
# called on instance
return instance._values[self.name]

def __set__(self, instance, value):
instance._values[name] = self.onchange(value)

class ActiveDAO(object):
def __init__(self):
self._values = []

class Person(ActiveDAO):
name = Field('firstname', lambda v: v.strip().capitalize())
age = Field('age', lambda v : int(v))

i tried code very similar after reading the first replies and found
that it did not work as expected on setting. for example, in

person = Person()
person.age = 27

"age" is set in the instance's dictionary (as 27; the descriptor is
not called), which then shadows the definition of age in the class
dictionary.

my understanding was that the descriptor is only called in the class
context, so would be called if, say, a subclass tried to redefine
age. but maybe i am still confused.

i am about to go to sleep. i guess i will try your code exactly
tomorrow, but it looks very close to mine which showed this problem.
are you sure your solution works?

thanks,
andrew

 

[andrew cooke]

ignore that - i was mistaken (my test was too complex).

the problem seems to be that the attribute is deleted even though
__delete__ is defined.

i'll look at it tomorrow.

thanks again,
andrew

 

[andrew cooke]

OK, fixed my bug - it does work. Now sleep... Thanks again, Andrew

 

[Bruno Desthuilliers]

andrew cooke a écrit :

(snip code)
i tried code very similar after reading the first replies and found
that it did not work as expected on setting. for example, in

person = Person()
person.age = 27

"age" is set in the instance's dictionary (as 27; the descriptor is
not called), which then shadows the definition of age in the class
dictionary.

Are you sure your Person class is a new-style one (inheriting, directly
or not, from object) ? The descriptor protocol doesn't work properly on
old-style classes, with *exactly* the symptom you describe here (setter
is not invoked, property get shadowed by an instance attribute)

my understanding was that the descriptor is only called in the class
context,

The descriptor protocol is only invoked on class attributes.

so would be called if, say, a subclass tried to redefine
age. but maybe i am still confused.
Possibly.

i am about to go to sleep. i guess i will try your code exactly
tomorrow, but it looks very close to mine which showed this problem.
are you sure your solution works?

Yes - minus a couple (unrelated) typos ('name' instead of 'self.name' in
Field.__set__, and 'self._values = []' instead of 'self._values = {}' in
ActiveDAO.__init__). Here's the corrected one:

class Field(object):
def __init__(self, name, onchange):
self.name = name
self.onchange = onchange

def __get__(self, instance, cls):
if instance is None:
# called on the class
return self
# called on instance
return instance._values[self.name]

def __set__(self, instance, value):
instance._values[self.name] = self.onchange(value)

class ActiveDAO(object):
def __init__(self):
self._values = {}

class Person(ActiveDAO):
name = Field('name', lambda v: v.strip().capitalize())
age = Field('age', lambda v : int(v))

 

[Hrvoje Niksic]

<op>
But bear in mind that, beside possible unwanted side-effectn, you'll
get a non-negligible performance hit - __getattribute__ being, as the
name implies, invoked on each and every attribute lookup.

That's unavoidable, though -- whatever you do to customize your class
in Python, the result will be slower than the C code built into
Python. Fortunately, not every object is performance-critical. In
this case, __getattribute__ buys you per-instance customization not
otherwise available. The code you posted is probably more efficient,
but at the cost of losing the ability to customize specific instances
of the class.

 

[Bruno Desthuilliers]

Hrvoje Niksic a écrit :

That's unavoidable, though -- whatever you do to customize your class
in Python, the result will be slower than the C code built into
Python.

This one customization hook is probably the most sensible still.

Fortunately, not every object is performance-critical. In
this case, __getattribute__ buys you per-instance customization not
otherwise available. The code you posted is probably more efficient,
but at the cost of losing the ability to customize specific instances
of the class.

You could as well just customize __setattr__/__getattr__ (which is what
was done before new-style classes and descriptors). But my own
experience is that the very few times I thought I had a need for
per-instance descriptors, I found other solutions that were certainly
easier to grasp and maintain on the long term. Not that I would not be
able to deal with per-instance descriptors, but so far the cost
outweighted the benefits IMHO. Now YMMV of course !-)