Implied instance attribute creation when referencing a class attribute

Discussion in 'Python' started by Russell Warren, Jan 16, 2006.

  1. I just ran across a case which seems like an odd exception to either
    what I understand as the "normal" variable lookup scheme in an
    instance/object heirarchy, or to the rules regarding variable usage
    before creation. Check this out:

    >>> class foo(object):

    .... I = 1
    .... def __init__(self):
    .... print self.__dict__
    .... self.I += 1
    .... print self.__dict__
    ....
    >>> a=foo()

    {}
    {'I': 2}
    >>> foo.I

    1
    >>> a.I

    2
    >>> del a.I
    >>> a.I

    1
    >>> del a.I

    Traceback (most recent call last):
    File "<string>", line 1, in <string>
    AttributeError: I
    >>> non_existent_var += 1

    Traceback (most recent call last):
    File "<string>", line 1, in <string>
    NameError: name 'non_existent_var' is not defined


    In this case, 'self.I += 1' clearly has inserted a surprise
    behind-the-scenes step of 'self.I = foo.I', and it is this which I find
    interesting.

    As I understand it, asking for self.I at this point should check
    self.__dict__ for an 'I' entry, and if it doesn't find it, head on up
    to foo.__dict__ and look for it.

    So... I initially *thought* there were two possibilities for what would
    happen with the 'self.I += 1':
    1. 'self.I += 1' would get a hold of 'foo.I' and increment it
    2. I'd get an AttributeError

    Both were wrong. I thought maybe an AttributeError because trying to
    modify 'self.I' at that point in the code is a bit fuzzy... ie: am I
    really trying to deal with foo.I (in which case, I should properly use
    foo.I) or am I trying to reference an instance attribute named I (in
    which case I should really create it explicitly first or get an error
    as with the non_existent_var example above... maybe with 'self.I =
    foo.I').

    Python is obviously assuming the latter and is "helping" you by
    automatically doing the 'self.I = foo.I' for you. Now that I know this
    I (hopefully) won't make this mistake again, but doing this seems
    equivalent to taking my 'non_existent_var += 1' example above and
    having the interpreter interpret as "oh, you must want to deal with an
    integer, so I'll just create one for you with 'non_existent_var = 0'
    first". Fortunately this is not done, so why do it with the instance
    attribute reference?

    Does anyone have any solid reasoning behind the Python behavior? It
    might help drive it home more so than just taking it as "that's the way
    it is" and remembering it.

    It gets even more confusing for me because the behaviour could be
    viewed as being opposite when dealing with mutable class members. eg:

    >>> class foo(object):

    .... M = [1,2,3]
    .... def __init__(self):
    .... self.M.append(len(self.M) + 1)
    .... print self.M
    ....
    >>> a=foo()

    [1, 2, 3, 4]
    >>> foo.M

    [1, 2, 3, 4]
    >>> del a.M

    Traceback (most recent call last):
    File "<string>", line 1, in <string>
    AttributeError: 'foo' object attribute 'M' is read-only

    By opposite I mean that with immutable objects, a sloppy self.I
    reference doesn't get you to the base class object, whereas with a
    mutable one you do get to the base object (although I do recognize that
    in both cases if you just remember that the interpreter will always
    stuff in a 'self.x = BaseClass.x' it works as expected in both the
    immutable and mutable case).

    After all that, I guess it boils down to me thinking that the code
    *should* interpret the attempted instance modification with one of the
    two possibilities I mentioned above (although after typing this I'm now
    leaning more towards an AttributeError rather than allowing 'self.I' to
    be synonymous with 'foo.I' if no local override).

    Russ

    PS: Apologies if I mangled the "proper" terminology for talking about
    this... hopefully it makes sense.
     
    Russell Warren, Jan 16, 2006
    #1
    1. Advertising

  2. Russell Warren

    Chris Mellon Guest

    Re: Implied instance attribute creation when referencing a classattribute

    On 16 Jan 2006 14:11:25 -0800, Russell Warren <> wrote:
    > I just ran across a case which seems like an odd exception to either
    > what I understand as the "normal" variable lookup scheme in an
    > instance/object heirarchy, or to the rules regarding variable usage
    > before creation. Check this out:
    >
    > >>> class foo(object):

    > ... I = 1
    > ... def __init__(self):
    > ... print self.__dict__
    > ... self.I += 1
    > ... print self.__dict__
    > ...
    > >>> a=foo()

    > {}
    > {'I': 2}
    > >>> foo.I

    > 1
    > >>> a.I

    > 2
    > >>> del a.I
    > >>> a.I

    > 1
    > >>> del a.I

    > Traceback (most recent call last):
    > File "<string>", line 1, in <string>
    > AttributeError: I
    > >>> non_existent_var += 1

    > Traceback (most recent call last):
    > File "<string>", line 1, in <string>
    > NameError: name 'non_existent_var' is not defined
    >
    >
    > In this case, 'self.I += 1' clearly has inserted a surprise
    > behind-the-scenes step of 'self.I = foo.I', and it is this which I find
    > interesting.
    >
    > As I understand it, asking for self.I at this point should check
    > self.__dict__ for an 'I' entry, and if it doesn't find it, head on up
    > to foo.__dict__ and look for it.
    >
    > So... I initially *thought* there were two possibilities for what would
    > happen with the 'self.I += 1':
    > 1. 'self.I += 1' would get a hold of 'foo.I' and increment it
    > 2. I'd get an AttributeError
    >
    > Both were wrong. I thought maybe an AttributeError because trying to
    > modify 'self.I' at that point in the code is a bit fuzzy... ie: am I
    > really trying to deal with foo.I (in which case, I should properly use
    > foo.I) or am I trying to reference an instance attribute named I (in
    > which case I should really create it explicitly first or get an error
    > as with the non_existent_var example above... maybe with 'self.I =
    > foo.I').
    >
    > Python is obviously assuming the latter and is "helping" you by
    > automatically doing the 'self.I = foo.I' for you. Now that I know this
    > I (hopefully) won't make this mistake again, but doing this seems
    > equivalent to taking my 'non_existent_var += 1' example above and
    > having the interpreter interpret as "oh, you must want to deal with an
    > integer, so I'll just create one for you with 'non_existent_var = 0'
    > first". Fortunately this is not done, so why do it with the instance
    > attribute reference?
    >
    > Does anyone have any solid reasoning behind the Python behavior? It
    > might help drive it home more so than just taking it as "that's the way
    > it is" and remembering it.
    >
    > It gets even more confusing for me because the behaviour could be
    > viewed as being opposite when dealing with mutable class members. eg:
    >
    > >>> class foo(object):

    > ... M = [1,2,3]
    > ... def __init__(self):
    > ... self.M.append(len(self.M) + 1)
    > ... print self.M
    > ...
    > >>> a=foo()

    > [1, 2, 3, 4]
    > >>> foo.M

    > [1, 2, 3, 4]
    > >>> del a.M

    > Traceback (most recent call last):
    > File "<string>", line 1, in <string>
    > AttributeError: 'foo' object attribute 'M' is read-only
    >
    > By opposite I mean that with immutable objects, a sloppy self.I
    > reference doesn't get you to the base class object, whereas with a
    > mutable one you do get to the base object (although I do recognize that
    > in both cases if you just remember that the interpreter will always
    > stuff in a 'self.x = BaseClass.x' it works as expected in both the
    > immutable and mutable case).
    >
    > After all that, I guess it boils down to me thinking that the code
    > *should* interpret the attempted instance modification with one of the
    > two possibilities I mentioned above (although after typing this I'm now
    > leaning more towards an AttributeError rather than allowing 'self.I' to
    > be synonymous with 'foo.I' if no local override).


    I can see how this can be confusing, but I think the confusion here is
    yours, not Pythons ;)

    self.I += 10 is an *assignment*. Like any assignment, it causes the
    attribute in question to be created. You wouldn't be suprised if you'd
    done self.I='foo' and had it "create" the instance attribute, would
    you?

    If you write out the longhand for += it becomes totally obvious what
    is happening and why it makes sense:

    temp = self.I (ends up returning value bound to foo.I)
    temp = temp+10 #if temp were mutable, this would modify it instead of
    creating a new object
    self.I = temp (binds name "I" in namespace "self" to the value bound
    to temp, which will shadow the class attribute)

    Note that nowhere is Python "inserting self.I = foo.I" for you - you
    are (implicitly) retrieving the class attribute, and then inserting
    that value into the instance attribute.

    So your case 1 is actually exactly what is happening! Python is
    getting a hold of foo.I and incrementing it (which you can see
    happening when you use a mutable object). The bit you're missing is
    after that, where you're then binding that value into the instance
    attribute.


    >
    > Russ
    >
    > PS: Apologies if I mangled the "proper" terminology for talking about
    > this... hopefully it makes sense.
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    Chris Mellon, Jan 16, 2006
    #2
    1. Advertising

  3. > I can see how this can be confusing, but I think the confusion here is
    > yours, not Pythons ;)


    This is very possible, but I don't think in the way you describe!

    > self.I += 10 is an *assignment*. Like any assignment, it causes the
    > attribute in question to be created


    .... no it isn't. The += is an operator. Look at the example I
    included with non_existent_var above. If that doesn't do it for you,
    pop open a clean python shell and do this one:

    >>> x += 2

    Traceback (most recent call last):
    File "<string>", line 1, in <string>
    NameError: name 'x' is not defined

    Note that x doesn't exists and it does not create it. You can't
    normally operate on something before it is created - Python won't
    create it for you (which is why I was surprised by the class attribute
    behavior in the first post).

    > If you write out the longhand for += it becomes totally obvious what
    > is happening and why it makes sense:


    Not true as above. The longhand for 'self.I += 1' is 'self.I = self.I
    + 1', which normally needs self.I to exist due to the RHS of this.

    > So your case 1 is actually exactly what is happening! Python is
    > getting a hold of foo.I and incrementing it


    Nope. My case 1 would have the 'self.I += 1' modifying the actual
    class attribute, not some new instance attribute and this is definitely
    NOT happening. Maybe my example was bad? Try this one instead:

    >>> class foo(object):

    .... I = 1
    .... def __init__(self):
    .... self.I += 123455
    ....
    >>> a=foo()
    >>> a.I

    123456
    >>> foo.I

    1
    >>> del a.I
    >>> a.I

    1

    Note that we ended up binding a new "I" to the 'a' instance with the
    'self.I += 1' statement, and it started with the value of 1 (the value
    of the base class attribute). I tried to make it clear in the example
    by wiping out the local copy, which then reveals the base class
    attribute when you go for it again.

    The fact that there is a local I being made with the value of the base
    class attribute says that Python is essentially adding the line 'self.I
    = foo.I' as in the code below.

    >>> class foo(object):

    .... I = 123455
    .... def __init__(self):
    .... self.I = foo.I # unnecessary since python seems to do it in
    the next line
    .... self.I += 1
    ....
    >>> a=foo()
    >>> b=foo()
    >>> c=foo()
    >>> print c.I, foo.I

    123456 1

    For kicks I added the b and c creations to show that at no time did the
    += operator get a hold of the foo base class as you state. It stayed
    untouched at 1 the whole time. To do that you need to reference foo
    itself as in the following case:

    >>> class foo(object):

    .... I = 0
    .... def __init__(self):
    .... foo.I += 1
    .... self.I = foo.I
    ....
    >>> a=foo()
    >>> b=foo()
    >>> c=foo()
    >>> print a.I, b.I, c.I, foo.I

    1 2 3 3
    >>> del a.I
    >>> a.I

    3

    Here it of course *did* increment the base foo attribute since it was
    directly referenced. 'a.I' stays as 1 here because I rebound a new
    instance attribute I on top with a copy of the base foo.I value due to
    it being immutable (a bit weird to use the same name, but I'm trying to
    show something) and it is what is retrieved first by Python (local
    dictionary first, if not found it goes to the base class). When I
    clear I from the local __dict__ with the del, you see that future
    self.I references skip out to the base class attribute since there is
    no instance I attribute anymore.

    A bit of a sidetrack there... still curious why python decides to
    auto-create the variable for you in this particular case. Any other
    takers?

    Russ
     
    Russell Warren, Jan 17, 2006
    #3
  4. Russell Warren

    David Wahler Guest

    Russell Warren wrote:
    > Not true as above. The longhand for 'self.I += 1' is 'self.I = self.I
    > + 1', which normally needs self.I to exist due to the RHS of this.


    Try this:

    >>> class foo(object):

    .... I = 1
    .... def __init__(self):
    .... print self.__dict__
    .... self.I = self.I + 1
    .... print self.__dict__
    ....
    >>> a=foo()

    {}
    {'I': 2}


    Notice that the result is the same! The catch is that the two
    occurrences of "self.I" occur in different contexts -- on the left-hand
    side of an assignment, and in an expression -- and are therefore
    subject to different lookup rules. Specifically, evaluation of "self.I"
    is delegated from instances to their classes and superclasses, while
    assignment is not.

    As an expression, "self.I" first tries and fails to look up
    self.__dict__['I']; then, finding foo.__dict__['I'] to be present, it
    returns that (1) instead. When the result of the expression is then
    assigned to self.I, no delegation takes place and the value 2 is stored
    in self.__dict__['I'].

    A note of caution: you might be tempted to think that with objects such
    as lists, which implement the __iadd__ method, no assignment would take
    place. This is actually not the case -- it works exactly the same way!
    To use another variation of your example:


    >>> class foo(object):

    .... lst = ['a','b','c']
    .... def __init__(self):
    .... print self.__dict__
    .... self.lst += [1,2,3]
    .... print self.__dict__
    ....
    >>> a=foo()

    {}
    {'lst': ['a','b','c',1,2,3]}
    >>> foo.l

    ['a','b','c',1,2,3]
    >>> id(foo.lst) == id(a.lst)

    True

    The list is mutated in-place, but there is _also_ an implicit
    assignment to self.lst. In other words, it expands to:

    self.lst = self.lst.__iadd__([1,2,3])

    Normally this implicit assignment doesn't matter, since the context the
    variable is stored into is the same one it was retrieved from. It's
    only in situations like yours where it becomes important.

    -- David
     
    David Wahler, Jan 17, 2006
    #4
  5. D'oh... I just realized why this is happening. It is clear in the
    longhand as you say, but I don't think in the way you descibed it (or
    I'm so far gone right now I have lost it).

    self.I += 1

    is the same as

    self.I = self.I + 1

    and when python tries figures out what the 'self.I' is on the right
    hand side. it of course ends up having to move up to the base class
    foo.__dict__ because there is no 'I' in self.__dict__ yet. So it ends
    up effectively being:

    self.I = foo.I + 1

    which explains where the "self.I = foo.I' that I was claiming was being
    done magically comes from.

    What my head was thinking was that the 'self.I' lookup would move up to
    get foo.__dict__['I'], and that I would effectively get 'foo.I += 1',
    but this is a bit of a brain fart and is just plain wrong.

    I should have seen that earlier... oh well. I'm happy that it is
    perfectly clear where it comes from, now. It still does look odd when
    you do a simplistic comparison of the behaviour of 'x += 1' and 'self.I
    += 1', but I suppose that that's just the way the lookup scheme
    crumbles. An unfortunate (and rare?) quirk, I guess.

    It still might be nice were python to just block out this potential
    confusion with an Exception... it seems that class vs instance
    attribute referencing is confusing enough for people without having
    this type of potential confusion lurking around the syntax. It seems
    like such a simple thing, but to understand the outcomes requires
    knowing how the name lookup scheme works, how mutable/immutable objects
    are dealt with, and what the += keystroke-saver/macro operator is
    actually doing. That this is stuff that someone coding in python
    should understand could certainly be argued, though...

    Russ
     
    Russell Warren, Jan 17, 2006
    #5
  6. Thanks for the additional examples, David (didn't see this before my
    last post). All of it makes sense now, including those examples.

    Russ
     
    Russell Warren, Jan 17, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kent Paul Dolan
    Replies:
    22
    Views:
    1,144
    goergebush
    Dec 22, 2005
  2. Replies:
    2
    Views:
    375
  3. Don Lancaster

    Implied Document form problem.

    Don Lancaster, May 3, 2007, in forum: Javascript
    Replies:
    2
    Views:
    165
    -Lost
    May 4, 2007
  4. Andrew Poulos

    implied eval

    Andrew Poulos, Feb 2, 2009, in forum: Javascript
    Replies:
    8
    Views:
    137
  5. Richard Maher

    Interface with implied Constructor

    Richard Maher, Jul 9, 2013, in forum: Java
    Replies:
    58
    Views:
    778
    Steven Simpson
    Jul 27, 2013
Loading...

Share This Page