Understanding descriptors

Discussion in 'Python' started by Brian Allen Vanderburg II, Jan 29, 2009.

  1. I'm trying to better understand descriptors and I've got a few questions
    still after reading some sites. Here is what I 'think', but please let
    me know if any of this is wrong as I'm sure it probably is.


    First when accessing an attribute on a class or instance it must be
    found. For an instance, it's __dict__ is first search. If not found
    the class and base class __dict__ are searched. For a class, the
    __dict__ and base classes __dict__ are search.

    If assigning, and the attribute is found and is not a descriptor or if
    the attribute is not found, then the assignment will occur in the
    __dict__ of the class or instance. If it is found and is a descriptor,
    then __set__ will be call.

    For reading, if the attribute is found and is a descriptor, __get__ will
    be called, passing the object (if it is an instance) and class. If it
    is not a descriptor, the attribute will be returned directly.

    Class methods are just functions:

    class C(object):
    def F(self):
    pass

    C.__dict__['F'] # function object ...

    But functions are descriptors:

    C.__dict__['F'].__get__ # method wrapper ...

    def f1():
    pass

    f1.__get__ # method wrapper ...

    When a lookup is done it uses this descriptor to make a bound or unbound
    method:

    c=C()

    C.F # unbound method object, expects explicit instance when calling the
    function
    c.F # bound method object provides instance implicitly when calling the
    function

    This is also done when adding to the classes:

    C.f1 = f1

    f1 # function
    C.f1 # unbound method
    c.f1 # bound method

    To prevent this it has to be decorated so the descriptor doesn't cause
    the binding:

    C.f2 = staticmethod(f1)
    C.f2 # functon
    c.f2 # function

    Here is a question, why don't instance attributes do the same thing?

    c.f3 = f1
    c.f3 # function, not bound method

    So it is not calling the __get__ method for c.f3 After it finds c.f3 in
    c.__dict__, and since it has a getter, shouldn't it call the __get__ to
    return the bound method. It is good that it doesn't I know, but I just
    want to know why it doesn't from an implementation view.


    Brian Vanderburg II
     
    Brian Allen Vanderburg II, Jan 29, 2009
    #1
    1. Advertising

  2. Brian Allen Vanderburg II

    Aahz Guest

    In article <>,
    Brian Allen Vanderburg II <> wrote:
    >
    > [...]
    >
    >When a lookup is done it uses this descriptor to make a bound or unbound
    >method:
    >
    >c=C()
    >
    >C.F # unbound method object, expects explicit instance when calling the
    >function
    >c.F # bound method object provides instance implicitly when calling the
    >function
    >
    >This is also done when adding to the classes:
    >
    >C.f1 = f1
    >
    >f1 # function
    >C.f1 # unbound method
    >c.f1 # bound method
    >
    >To prevent this it has to be decorated so the descriptor doesn't cause
    >the binding:
    >
    >C.f2 = staticmethod(f1)
    >C.f2 # functon
    >c.f2 # function
    >
    >Here is a question, why don't instance attributes do the same thing?
    >
    >c.f3 = f1
    >c.f3 # function, not bound method
    >
    >So it is not calling the __get__ method for c.f3 After it finds c.f3 in
    >c.__dict__, and since it has a getter, shouldn't it call the __get__ to
    >return the bound method. It is good that it doesn't I know, but I just
    >want to know why it doesn't from an implementation view.


    This is a bit beyond my expertise, but nobody else has responded, so I'll
    take a stab at it:

    Basically, this is part of the design philosphy that keeps everything
    working the same way. When an attribute is found on an instance, it
    gets returned unmodified. Period. In other words, the fact that f3
    happens to return a function object gets ignored. You can force f3 to
    work like a bound method by defining it as one:

    def f3(self):
    pass

    If you use a new-style class or Python 3.0, the same design is evident
    when you try to use e.g. str() to call the __str__() special method: the
    instance gets ignored for looking up the method.
    --
    Aahz () <*> http://www.pythoncraft.com/

    Weinberg's Second Law: If builders built buildings the way programmers wrote
    programs, then the first woodpecker that came along would destroy civilization.
     
    Aahz, Feb 4, 2009
    #2
    1. Advertising

  3. Brian Allen Vanderburg II a écrit :
    > I'm trying to better understand descriptors and I've got a few questions
    > still after reading some sites. Here is what I 'think', but please let
    > me know if any of this is wrong as I'm sure it probably is.
    >
    >
    > First when accessing an attribute on a class or instance it must be
    > found. For an instance, it's __dict__ is first search.


    Actually, this is not true - binding descriptors are searched (in the
    class and it's mro) _before_ the instance's __dict__ :

    >>> class Foo(object):

    .... @apply
    .... def bar():
    .... def fget(self):
    .... return self._bar
    .... def fset(self, v):
    .... self._bar = v
    .... return property(**locals())
    ....
    >>> f = Foo()
    >>> f.bar = 42
    >>> f.bar

    42
    >>> f.__dict__['bar'] = "boo"
    >>> f.bar

    42
    >>> f.__dict__['bar']

    'boo'
    >>>


    Note that this works a bit differently for non-binding descriptors
    (descriptors that only implement the __get__ method), which are looked
    up after the instance's __dict__

    So the lookup chain is:

    1/ lookup the class and bases for a binding descriptor
    2/ then lookup the instance's __dict__
    3/ then lookup the class and bases for a non-binding descriptor or plain
    attribute
    4/ then class __getattr__

    Also and FWIW, there's a "step zero" : calls __getattribute__. All the
    above lookup mechanism is actually implemented by object.__getattribute__.

    > If not found
    > the class and base class __dict__ are searched. For a class, the
    > __dict__ and base classes __dict__ are search.
    >
    > If assigning, and the attribute is found and is not a descriptor


    a binding descriptor

    > or if
    > the attribute is not found, then the assignment will occur in the
    > __dict__ of the class or instance. If it is found and is a descriptor,


    a binding descriptor

    > then __set__ will be call.



    > For reading, if the attribute is found and is a descriptor, __get__ will
    > be called, passing the object (if it is an instance) and class. If it
    > is not a descriptor, the attribute will be returned directly.
    >
    > Class methods are just functions:


    In Python, "classmethod" has a definite meaning - it's a method that
    takes the class (and not the instance) as first argument. So your
    assertion should be "Methods are just functions which are attributes of
    the class". Which is still not quite true. Sure, the def statement
    always create function objects, regardless of whether it happens within
    a class statement or not. OTHO, what you get when looking up the
    corresponding attribute is definitly a method object. So the correct
    formulation here is (IMHO of course) that methods are implemented by
    functions that are attributes of the class. Also, this is not restricted
    to functions : any callable object correctly implementing the descriptor
    protocol can be used (cf the classmethod and staticmethod objects).

    > class C(object):
    > def F(self):
    > pass
    >
    > C.__dict__['F'] # function object ...


    yes.

    > But functions are descriptors:


    yes.

    > C.__dict__['F'].__get__ # method wrapper ...
    >
    > def f1():
    > pass
    >
    > f1.__get__ # method wrapper ...
    >
    > When a lookup is done it uses this descriptor to make a bound or unbound
    > method:


    yes.

    > c=C()
    >
    > C.F # unbound method object, expects explicit instance when calling the
    > function
    > c.F # bound method object provides instance implicitly when calling the
    > function
    >
    > This is also done when adding to the classes:


    Indeed. Whether the def statement happens within the class statement or
    not is irrelevant here.

    > C.f1 = f1
    >
    > f1 # function
    > C.f1 # unbound method
    > c.f1 # bound method
    >
    > To prevent this it has to be decorated so the descriptor doesn't cause
    > the binding:
    >
    > C.f2 = staticmethod(f1)
    > C.f2 # functon
    > c.f2 # function


    yes.

    > Here is a question, why don't instance attributes do the same thing?


    Because the descriptor protocol is only invoked on class attributes.

    > c.f3 = f1
    > c.f3 # function, not bound method
    >
    > So it is not calling the __get__ method for c.f3 After it finds c.f3 in
    > c.__dict__, and since it has a getter, shouldn't it call the __get__ to
    > return the bound method.


    Nope, cf above.

    > It is good that it doesn't I know, but I just
    > want to know why it doesn't from an implementation view.


    If by "from an implementation view", you mean "how it works", then the
    answer is above. If your question was about "why this design choice",
    I'll leave the answer to someone else...
     
    Bruno Desthuilliers, Feb 5, 2009
    #3
  4. lid wrote:
    >
    > So the lookup chain is:
    >
    > 1/ lookup the class and bases for a binding descriptor
    > 2/ then lookup the instance's __dict__
    > 3/ then lookup the class and bases for a non-binding descriptor or
    > plain attribute
    > 4/ then class __getattr__
    >
    > Also and FWIW, there's a "step zero" : calls __getattribute__. All the
    > above lookup mechanism is actually implemented by
    > object.__getattribute__.


    Okay, so instance attributes never use their __get__/__set__/etc when
    looking up.

    A binding descriptor is one that has a __set__ (even if it doesn't do
    anything) and it takes priority over instance variables. Properties are
    binding descriptors even if they don't have a set function specified. A
    non-binding descriptor doesn't have __set__ and instance variables take
    priority over them.

    For reading:

    1. Lookup in the class/bases for a binding descriptor and if found use
    its __get__
    2. If instance, look up in instance __dict__ and if found return it
    3. Lookup in the class/bases
    a. if found and a descriptor use it's __get__
    b. if found and not a descriptor return it
    4. Use __getattr__ (if instance?)

    For writing:

    1. If instance
    a. lookup in the class/bases for a binding descriptor and if found
    use its __set__
    b. write to instance __dict__
    2. If class, write in class __dict__


    I think I understand it now. Thanks.

    Brian Vanderburg II
     
    Brian Allen Vanderburg II, Feb 5, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gordon Beaton

    Re: Running out of file descriptors

    Gordon Beaton, Jul 24, 2003, in forum: Java
    Replies:
    0
    Views:
    398
    Gordon Beaton
    Jul 24, 2003
  2. Oz Levanon

    File Descriptors Leak

    Oz Levanon, Nov 25, 2003, in forum: Java
    Replies:
    3
    Views:
    585
    Robert Olofsson
    Nov 25, 2003
  3. SmokingGun
    Replies:
    0
    Views:
    802
    SmokingGun
    Feb 9, 2004
  4. rooban
    Replies:
    0
    Views:
    606
    rooban
    Jul 21, 2004
  5. Replies:
    0
    Views:
    372
Loading...

Share This Page