Class attributes, instances and metaclass __getattribute__

[Pedro Werneck]

Hi all


I noticed something strange here while explaining decorators to someone.
Not any real use code, but I think it's worth mentioning.

When I access a class attribute, on a class with a custom metaclass with
a __getattribute__ method, the method is used when acessing some
attribute directly with the class object, but not when you do it from
the instance.

.... def __getattribute__(cls, attr):
.... print cls, attr
.... return type.__getattribute__(cls, attr)
.... .... __metaclass__ = M
.... </code>


Someone at freenode #python channel involved with python-dev sprint
suggested it might be a bug, worth mentioning... to me it seems like a
decision to avoid some problems with method and descriptors creation,
since someone using metaclasses and custom __getattribute__ at the same
time is asking for trouble, but... I googled for it and tried to find
something on the list but, nothing. From the source it seems like a
generic wrapper is used. What's the real case here ?


Regards,

 

[Michele Simionato]

When I access a class attribute, on a class with a custom metaclass with
a __getattribute__ method, the method is used when acessing some
attribute directly with the class object, but not when you do it from
the instance.


... def __getattribute__(cls, attr):
... print cls, attr
... return type.__getattribute__(cls, attr)
...
... __metaclass__ = M
...
</code>


Someone at freenode #python channel involved with python-dev sprint
suggested it might be a bug, worth mentioning... to me it seems like a
decision to avoid some problems with method and descriptors creation,
since someone using metaclasses and custom __getattribute__ at the same
time is asking for trouble, but... I googled for it and tried to find
something on the list but, nothing. From the source it seems like a
generic wrapper is used. What's the real case here ?


Regards,

To me, it seems consistent. As said in
http://www-128.ibm.com/developerworks/linux/library/l-pymeta2/

"""The availability of metaclass attributes is not transitive; in other
words, the attributes of a metaclass are available to its instances,
but not to the instances of the instances. Just this is the main
difference between metaclasses and superclasses."""

Since this happens for real attributes, it looks natural that the same
should
happen for 'virtual' attributes implemented via '__getattr__' or
'__getattribute__'.

Michele Simionato

 

[Pedro Werneck]

Hi

To me, it seems consistent. As said in
http://www-128.ibm.com/developerworks/linux/library/l-pymeta2/

"""The availability of metaclass attributes is not transitive; in
other words, the attributes of a metaclass are available to its
instances, but not to the instances of the instances. Just this is the
main difference between metaclasses and superclasses."""

Well... I'm not talking about metaclass attributes... that's perfectly
consistent, agreed.

I'm saying that when the class implements a custom __getattribute__,
when you try to access the instance attributes from itself, it uses it.
But if the class is a metaclass, instances of its instances have acess
to the attribute anyway, but don't use the custom __getattribute__ you
implemented.

Like the example I mentioned on the previous mail, or (I think in this
case it's more obvious):
.... def __getattribute__(cls, attr):
.... print 'using N.__getattribute for "%s"'%(attr)
.... return type.__getattribute__(cls, attr)
.... .... __metaclass__ = N
.... .... __metaclass__ = M
.... using N.__getattribute for "x"
'foo''foo'


So, in both cases I have access to the class attribute; but in one case
it's using the bound M.__getattribute__ implemented at the base
metaclass, but not in the other. It was supposed to use it after the
bound 'C.__getattribute__' raises AttributeError as expected, but it
doesn't...

It's inconsistent, but seems a reasonable decision to me since someone
can easily mess with descriptors doing with this... but, someone else
involved with Python dev I talked about thinks it may be a bug.

And, I'm curious anyway... is it possible to customize attribute access
in this case in any other way ? What really happens here ?

From the typeobject.c source code, seems like when the metaclass

implements __getattribute__, it uses type.__getattribute__ in this case,
but I'm not sure.

Since this happens for real attributes, it looks natural that the same
should happen for 'virtual' attributes implemented via '__getattr__'
or '__getattribute__'.

Well... as I think it's clear now, the case you mentioned is not exactly
what I'm talking about, but it's good you mentioned about 'virtual'
attributes, because in this case we have another problem, it's
inconsistent too and there's no reason for it...
.... def __getattr__(cls, attr):
.... if attr == 'x':
.... return 'foo'
.... .... __metaclass__ = M
.... ....
AttributeError: 'C' object has no attribute 'x'
So... both 'x' and 'y' are class attributes, but 'x' is a virtual
attribute implemented with M.__getattr__. From the instance I have
access to 'y' but not to 'x'.



Regards,

 

[Ziga Seilnacht]

Hi
[snip]
Well... I'm not talking about metaclass attributes... that's perfectly
consistent, agreed.

I'm saying that when the class implements a custom __getattribute__,
when you try to access the instance attributes from itself, it uses it.
But if the class is a metaclass, instances of its instances have acess
to the attribute anyway, but don't use the custom __getattribute__ you
implemented.

Attribute lookup for instances of a class never calls metaclass'
__getattribute__() method. This method is called only when you
access attributes directly on the class.

[snip]

And, I'm curious anyway... is it possible to customize attribute access
in this case in any other way ? What really happens here ?

There are two distinct methods involved in your example; attribute
lookup for classes is controled by metaclass' __getattribute__()
method, while instance attribute lookup is controled by class'
__getattribute__() method.

They are basicaly the same, but they never use ``type(obj).attr`` to
access the class' attributes. The code for these methods would look
something like this in Python:

class Object(object):
"""
Emulates object's and type's behaviour in attribute lookup.
"""

def __getattribute__(self, name):
cls = type(self)

# you normally access this as self.__dict__
try:
dict_descriptor = cls.__dict__['__dict__']
except KeyError:
# uses __slots__ without dict
mydict = {}
else:
mydict = dict_descriptor.__get__(self, cls)

# Can't use cls.name because we would get descriptors
# (methods and similar) that are provided by class'
# metaclass and are not meant to be accessible from
# instances.
classdicts = [c.__dict__ for c in cls.__mro__]

# We have to look in class attributes first, since it can
# be a data descriptor, in which case we have to ignore
# the value in the instance's dict.
for d in classdicts:
if name in d:
classattr = d[name]
break
else:
# None of the classes provides this attribute; perform
# the normal lookup in instance's dict.
try:
return mydict[name]
except KeyError:
# Finally if everything else failed, look for the
# __getattr__ hook.
for d in classdicts:
if '__getattr__' in d:
return d['__getattr__'](self, name)
msg = "%r object has no attribute %r"
raise AttributeError(msg % (cls.__name__, name))

# Check if class' attribute is a descriptor.
if hasattr(classattr, '__get__'):
# If it is a non-data descriptor, then the value in
# instance's dict takes precedence
if not hasattr(classattr, '__set__') and name in mydict:
return mydict[name]
return classattr.__get__(self, cls)

# Finally, look into instance's dict.
return mydict.get(name, classattr)

As you can see, it completely avoids calling metaclass'
__getattribute__()
method. If it wouldn't do that, then the metaclass' attributes would
'leak' to instances of its classes. For example, __name__, __mro__
and mro() are some of the descriptors provided by type to every class,
but they are not accesible through instances of these classes,
and shouldn't be, otherwise they could mask some errors in user's code.

Ziga

 

[Pedro Werneck]

[snip]

Well... I'm not talking about metaclass attributes... that's
perfectly consistent, agreed.

I'm saying that when the class implements a custom __getattribute__,
when you try to access the instance attributes from itself, it uses
it. But if the class is a metaclass, instances of its instances have
acess to the attribute anyway, but don't use the custom
__getattribute__ you implemented.

Attribute lookup for instances of a class never calls metaclass'
__getattribute__() method. This method is called only when you
access attributes directly on the class.

Well... thanks for the answer.

As I said on the first mail, I noticed this when I was explaining
descriptors and methods to someone else... I implemented a pure-python
Method class to show him exactly how it works.

But, since their __get__ call is available at the class too, my first
thought was that it was implemented at the metaclass __getattribute__,
and when an instance tries to get a class attribute it would fail on its
own __getattribute__, use the bound method at its class and make the
call.

After implementing a test metaclass I noticed it doesn't work this way,
and even if it's a bit inconsistent (especially in the case of 'virtual'
attributes), it seemed to me the reasonable thing to do, exactly for
what you mentioned on your code... someone could easily break a lot of
stuff doing it the wrong way, instead if not using __dict__.

I mailed the list because someone else thought it might be a bug and I
was in doubt... now it's clear it was the right thing to do.


Regards,