Understanding descriptors

[Brian Allen Vanderburg II]

I'm trying to better understand descriptors and I've got a few questions
still after reading some sites. Here is what I 'think', but please let
me know if any of this is wrong as I'm sure it probably is.


First when accessing an attribute on a class or instance it must be
found. For an instance, it's __dict__ is first search. If not found
the class and base class __dict__ are searched. For a class, the
__dict__ and base classes __dict__ are search.

If assigning, and the attribute is found and is not a descriptor or if
the attribute is not found, then the assignment will occur in the
__dict__ of the class or instance. If it is found and is a descriptor,
then __set__ will be call.

For reading, if the attribute is found and is a descriptor, __get__ will
be called, passing the object (if it is an instance) and class. If it
is not a descriptor, the attribute will be returned directly.

Class methods are just functions:

class C(object):
def F(self):
pass

C.__dict__['F'] # function object ...

But functions are descriptors:

C.__dict__['F'].__get__ # method wrapper ...

def f1():
pass

f1.__get__ # method wrapper ...

When a lookup is done it uses this descriptor to make a bound or unbound
method:

c=C()

C.F # unbound method object, expects explicit instance when calling the
function
c.F # bound method object provides instance implicitly when calling the
function

This is also done when adding to the classes:

C.f1 = f1

f1 # function
C.f1 # unbound method
c.f1 # bound method

To prevent this it has to be decorated so the descriptor doesn't cause
the binding:

C.f2 = staticmethod(f1)
C.f2 # functon
c.f2 # function

Here is a question, why don't instance attributes do the same thing?

c.f3 = f1
c.f3 # function, not bound method

So it is not calling the __get__ method for c.f3 After it finds c.f3 in
c.__dict__, and since it has a getter, shouldn't it call the __get__ to
return the bound method. It is good that it doesn't I know, but I just
want to know why it doesn't from an implementation view.


Brian Vanderburg II

 

[Aahz]

[...]

When a lookup is done it uses this descriptor to make a bound or unbound
method:

c=C()

C.F # unbound method object, expects explicit instance when calling the
function
c.F # bound method object provides instance implicitly when calling the
function

This is also done when adding to the classes:

C.f1 = f1

f1 # function
C.f1 # unbound method
c.f1 # bound method

To prevent this it has to be decorated so the descriptor doesn't cause
the binding:

C.f2 = staticmethod(f1)
C.f2 # functon
c.f2 # function

Here is a question, why don't instance attributes do the same thing?

c.f3 = f1
c.f3 # function, not bound method

So it is not calling the __get__ method for c.f3 After it finds c.f3 in
c.__dict__, and since it has a getter, shouldn't it call the __get__ to
return the bound method. It is good that it doesn't I know, but I just
want to know why it doesn't from an implementation view.

This is a bit beyond my expertise, but nobody else has responded, so I'll
take a stab at it:

Basically, this is part of the design philosphy that keeps everything
working the same way. When an attribute is found on an instance, it
gets returned unmodified. Period. In other words, the fact that f3
happens to return a function object gets ignored. You can force f3 to
work like a bound method by defining it as one:

def f3(self):
pass

If you use a new-style class or Python 3.0, the same design is evident
when you try to use e.g. str() to call the __str__() special method: the
instance gets ignored for looking up the method.

 

[Bruno Desthuilliers]

Brian Allen Vanderburg II a écrit :

I'm trying to better understand descriptors and I've got a few questions
still after reading some sites. Here is what I 'think', but please let
me know if any of this is wrong as I'm sure it probably is.


First when accessing an attribute on a class or instance it must be
found. For an instance, it's __dict__ is first search.

Actually, this is not true - binding descriptors are searched (in the
class and it's mro) _before_ the instance's __dict__ :
.... @apply
.... def bar():
.... def fget(self):
.... return self._bar
.... def fset(self, v):
.... self._bar = v
.... return property(**locals())
....

>>> f = Foo()
>>> f.bar = 42
>>> f.bar 42
>>> f.__dict__['bar'] = "boo"
>>> f.bar 42
>>> f.__dict__['bar'] 'boo'
>>>

Note that this works a bit differently for non-binding descriptors
(descriptors that only implement the __get__ method), which are looked
up after the instance's __dict__

So the lookup chain is:

1/ lookup the class and bases for a binding descriptor
2/ then lookup the instance's __dict__
3/ then lookup the class and bases for a non-binding descriptor or plain
attribute
4/ then class __getattr__

Also and FWIW, there's a "step zero" : calls __getattribute__. All the
above lookup mechanism is actually implemented by object.__getattribute__.

If not found
the class and base class __dict__ are searched. For a class, the
__dict__ and base classes __dict__ are search.

If assigning, and the attribute is found and is not a descriptor

a binding descriptor

or if
the attribute is not found, then the assignment will occur in the
__dict__ of the class or instance. If it is found and is a descriptor,

a binding descriptor

then __set__ will be call.

For reading, if the attribute is found and is a descriptor, __get__ will
be called, passing the object (if it is an instance) and class. If it
is not a descriptor, the attribute will be returned directly.

Class methods are just functions:

In Python, "classmethod" has a definite meaning - it's a method that
takes the class (and not the instance) as first argument. So your
assertion should be "Methods are just functions which are attributes of
the class". Which is still not quite true. Sure, the def statement
always create function objects, regardless of whether it happens within
a class statement or not. OTHO, what you get when looking up the
corresponding attribute is definitly a method object. So the correct
formulation here is (IMHO of course) that methods are implemented by
functions that are attributes of the class. Also, this is not restricted
to functions : any callable object correctly implementing the descriptor
protocol can be used (cf the classmethod and staticmethod objects).

class C(object):
def F(self):
pass

C.__dict__['F'] # function object ...
yes.

But functions are descriptors:
yes.

C.__dict__['F'].__get__ # method wrapper ...

def f1():
pass

f1.__get__ # method wrapper ...

When a lookup is done it uses this descriptor to make a bound or unbound
method:
yes.

c=C()

C.F # unbound method object, expects explicit instance when calling the
function
c.F # bound method object provides instance implicitly when calling the
function

This is also done when adding to the classes:

Indeed. Whether the def statement happens within the class statement or
not is irrelevant here.

C.f1 = f1

f1 # function
C.f1 # unbound method
c.f1 # bound method

To prevent this it has to be decorated so the descriptor doesn't cause
the binding:

C.f2 = staticmethod(f1)
C.f2 # functon
c.f2 # function
yes.

Here is a question, why don't instance attributes do the same thing?

Because the descriptor protocol is only invoked on class attributes.

c.f3 = f1
c.f3 # function, not bound method

So it is not calling the __get__ method for c.f3 After it finds c.f3 in
c.__dict__, and since it has a getter, shouldn't it call the __get__ to
return the bound method.

Nope, cf above.

It is good that it doesn't I know, but I just
want to know why it doesn't from an implementation view.

If by "from an implementation view", you mean "how it works", then the
answer is above. If your question was about "why this design choice",
I'll leave the answer to someone else...

 

[Brian Allen Vanderburg II]

So the lookup chain is:

1/ lookup the class and bases for a binding descriptor
2/ then lookup the instance's __dict__
3/ then lookup the class and bases for a non-binding descriptor or
plain attribute
4/ then class __getattr__

Also and FWIW, there's a "step zero" : calls __getattribute__. All the
above lookup mechanism is actually implemented by
object.__getattribute__.

Okay, so instance attributes never use their __get__/__set__/etc when
looking up.

A binding descriptor is one that has a __set__ (even if it doesn't do
anything) and it takes priority over instance variables. Properties are
binding descriptors even if they don't have a set function specified. A
non-binding descriptor doesn't have __set__ and instance variables take
priority over them.

For reading:

1. Lookup in the class/bases for a binding descriptor and if found use
its __get__
2. If instance, look up in instance __dict__ and if found return it
3. Lookup in the class/bases
a. if found and a descriptor use it's __get__
b. if found and not a descriptor return it
4. Use __getattr__ (if instance?)

For writing:

1. If instance
a. lookup in the class/bases for a binding descriptor and if found
use its __set__
b. write to instance __dict__
2. If class, write in class __dict__


I think I understand it now. Thanks.

Brian Vanderburg II