Implied instance attribute creation when referencing a class attribute

[Russell Warren]

I just ran across a case which seems like an odd exception to either
what I understand as the "normal" variable lookup scheme in an
instance/object heirarchy, or to the rules regarding variable usage
before creation. Check this out:
.... I = 1
.... def __init__(self):
.... print self.__dict__
.... self.I += 1
.... print self.__dict__
....{}
{'I': 2}Traceback (most recent call last):
Traceback (most recent call last):
File "<string>", line 1, in <string>
NameError: name 'non_existent_var' is not defined


In this case, 'self.I += 1' clearly has inserted a surprise
behind-the-scenes step of 'self.I = foo.I', and it is this which I find
interesting.

As I understand it, asking for self.I at this point should check
self.__dict__ for an 'I' entry, and if it doesn't find it, head on up
to foo.__dict__ and look for it.

So... I initially *thought* there were two possibilities for what would
happen with the 'self.I += 1':
1. 'self.I += 1' would get a hold of 'foo.I' and increment it
2. I'd get an AttributeError

Both were wrong. I thought maybe an AttributeError because trying to
modify 'self.I' at that point in the code is a bit fuzzy... ie: am I
really trying to deal with foo.I (in which case, I should properly use
foo.I) or am I trying to reference an instance attribute named I (in
which case I should really create it explicitly first or get an error
as with the non_existent_var example above... maybe with 'self.I =
foo.I').

Python is obviously assuming the latter and is "helping" you by
automatically doing the 'self.I = foo.I' for you. Now that I know this
I (hopefully) won't make this mistake again, but doing this seems
equivalent to taking my 'non_existent_var += 1' example above and
having the interpreter interpret as "oh, you must want to deal with an
integer, so I'll just create one for you with 'non_existent_var = 0'
first". Fortunately this is not done, so why do it with the instance
attribute reference?

Does anyone have any solid reasoning behind the Python behavior? It
might help drive it home more so than just taking it as "that's the way
it is" and remembering it.

It gets even more confusing for me because the behaviour could be
viewed as being opposite when dealing with mutable class members. eg:
.... M = [1,2,3]
.... def __init__(self):
.... self.M.append(len(self.M) + 1)
.... print self.M
....

a=foo() [1, 2, 3, 4]
foo.M [1, 2, 3, 4]
del a.M

Traceback (most recent call last):
File "<string>", line 1, in <string>
AttributeError: 'foo' object attribute 'M' is read-only

By opposite I mean that with immutable objects, a sloppy self.I
reference doesn't get you to the base class object, whereas with a
mutable one you do get to the base object (although I do recognize that
in both cases if you just remember that the interpreter will always
stuff in a 'self.x = BaseClass.x' it works as expected in both the
immutable and mutable case).

After all that, I guess it boils down to me thinking that the code
*should* interpret the attempted instance modification with one of the
two possibilities I mentioned above (although after typing this I'm now
leaning more towards an AttributeError rather than allowing 'self.I' to
be synonymous with 'foo.I' if no local override).

Russ

PS: Apologies if I mangled the "proper" terminology for talking about
this... hopefully it makes sense.

 

[Chris Mellon]

I just ran across a case which seems like an odd exception to either
what I understand as the "normal" variable lookup scheme in an
instance/object heirarchy, or to the rules regarding variable usage
before creation. Check this out:
... I = 1
... def __init__(self):
... print self.__dict__
... self.I += 1
... print self.__dict__
...{}
{'I': 2}Traceback (most recent call last):
Traceback (most recent call last):
File "<string>", line 1, in <string>
NameError: name 'non_existent_var' is not defined


In this case, 'self.I += 1' clearly has inserted a surprise
behind-the-scenes step of 'self.I = foo.I', and it is this which I find
interesting.

As I understand it, asking for self.I at this point should check
self.__dict__ for an 'I' entry, and if it doesn't find it, head on up
to foo.__dict__ and look for it.

So... I initially *thought* there were two possibilities for what would
happen with the 'self.I += 1':
1. 'self.I += 1' would get a hold of 'foo.I' and increment it
2. I'd get an AttributeError

Both were wrong. I thought maybe an AttributeError because trying to
modify 'self.I' at that point in the code is a bit fuzzy... ie: am I
really trying to deal with foo.I (in which case, I should properly use
foo.I) or am I trying to reference an instance attribute named I (in
which case I should really create it explicitly first or get an error
as with the non_existent_var example above... maybe with 'self.I =
foo.I').

Python is obviously assuming the latter and is "helping" you by
automatically doing the 'self.I = foo.I' for you. Now that I know this
I (hopefully) won't make this mistake again, but doing this seems
equivalent to taking my 'non_existent_var += 1' example above and
having the interpreter interpret as "oh, you must want to deal with an
integer, so I'll just create one for you with 'non_existent_var = 0'
first". Fortunately this is not done, so why do it with the instance
attribute reference?

Does anyone have any solid reasoning behind the Python behavior? It
might help drive it home more so than just taking it as "that's the way
it is" and remembering it.

It gets even more confusing for me because the behaviour could be
viewed as being opposite when dealing with mutable class members. eg:
... M = [1,2,3]
... def __init__(self):
... self.M.append(len(self.M) + 1)
... print self.M
...

a=foo() [1, 2, 3, 4]
foo.M [1, 2, 3, 4]
del a.M

Traceback (most recent call last):
File "<string>", line 1, in <string>
AttributeError: 'foo' object attribute 'M' is read-only

By opposite I mean that with immutable objects, a sloppy self.I
reference doesn't get you to the base class object, whereas with a
mutable one you do get to the base object (although I do recognize that
in both cases if you just remember that the interpreter will always
stuff in a 'self.x = BaseClass.x' it works as expected in both the
immutable and mutable case).

After all that, I guess it boils down to me thinking that the code
*should* interpret the attempted instance modification with one of the
two possibilities I mentioned above (although after typing this I'm now
leaning more towards an AttributeError rather than allowing 'self.I' to
be synonymous with 'foo.I' if no local override).

I can see how this can be confusing, but I think the confusion here is
yours, not Pythons

self.I += 10 is an *assignment*. Like any assignment, it causes the
attribute in question to be created. You wouldn't be suprised if you'd
done self.I='foo' and had it "create" the instance attribute, would
you?

If you write out the longhand for += it becomes totally obvious what
is happening and why it makes sense:

temp = self.I (ends up returning value bound to foo.I)
temp = temp+10 #if temp were mutable, this would modify it instead of
creating a new object
self.I = temp (binds name "I" in namespace "self" to the value bound
to temp, which will shadow the class attribute)

Note that nowhere is Python "inserting self.I = foo.I" for you - you
are (implicitly) retrieving the class attribute, and then inserting
that value into the instance attribute.

So your case 1 is actually exactly what is happening! Python is
getting a hold of foo.I and incrementing it (which you can see
happening when you use a mutable object). The bit you're missing is
after that, where you're then binding that value into the instance
attribute.

 

[Russell Warren]

I can see how this can be confusing, but I think the confusion here is

yours, not Pythons

This is very possible, but I don't think in the way you describe!

self.I += 10 is an *assignment*. Like any assignment, it causes the
attribute in question to be created

.... no it isn't. The += is an operator. Look at the example I
included with non_existent_var above. If that doesn't do it for you,
pop open a clean python shell and do this one:
Traceback (most recent call last):
File "<string>", line 1, in <string>
NameError: name 'x' is not defined

Note that x doesn't exists and it does not create it. You can't
normally operate on something before it is created - Python won't
create it for you (which is why I was surprised by the class attribute
behavior in the first post).

If you write out the longhand for += it becomes totally obvious what
is happening and why it makes sense:

Not true as above. The longhand for 'self.I += 1' is 'self.I = self.I
+ 1', which normally needs self.I to exist due to the RHS of this.

So your case 1 is actually exactly what is happening! Python is
getting a hold of foo.I and incrementing it

Nope. My case 1 would have the 'self.I += 1' modifying the actual
class attribute, not some new instance attribute and this is definitely
NOT happening. Maybe my example was bad? Try this one instead:
.... I = 1
.... def __init__(self):
.... self.I += 123455
....1

Note that we ended up binding a new "I" to the 'a' instance with the
'self.I += 1' statement, and it started with the value of 1 (the value
of the base class attribute). I tried to make it clear in the example
by wiping out the local copy, which then reveals the base class
attribute when you go for it again.

The fact that there is a local I being made with the value of the base
class attribute says that Python is essentially adding the line 'self.I
= foo.I' as in the code below.
.... I = 123455
.... def __init__(self):
.... self.I = foo.I # unnecessary since python seems to do it in
the next line
.... self.I += 1
....123456 1

For kicks I added the b and c creations to show that at no time did the
+= operator get a hold of the foo base class as you state. It stayed
untouched at 1 the whole time. To do that you need to reference foo
itself as in the following case:
.... I = 0
.... def __init__(self):
.... foo.I += 1
.... self.I = foo.I
....3

Here it of course *did* increment the base foo attribute since it was
directly referenced. 'a.I' stays as 1 here because I rebound a new
instance attribute I on top with a copy of the base foo.I value due to
it being immutable (a bit weird to use the same name, but I'm trying to
show something) and it is what is retrieved first by Python (local
dictionary first, if not found it goes to the base class). When I
clear I from the local __dict__ with the del, you see that future
self.I references skip out to the base class attribute since there is
no instance I attribute anymore.

A bit of a sidetrack there... still curious why python decides to
auto-create the variable for you in this particular case. Any other
takers?

Russ

 

[David Wahler]

Not true as above. The longhand for 'self.I += 1' is 'self.I = self.I
+ 1', which normally needs self.I to exist due to the RHS of this.

Try this:
.... I = 1
.... def __init__(self):
.... print self.__dict__
.... self.I = self.I + 1
.... print self.__dict__
....{}
{'I': 2}


Notice that the result is the same! The catch is that the two
occurrences of "self.I" occur in different contexts -- on the left-hand
side of an assignment, and in an expression -- and are therefore
subject to different lookup rules. Specifically, evaluation of "self.I"
is delegated from instances to their classes and superclasses, while
assignment is not.

As an expression, "self.I" first tries and fails to look up
self.__dict__['I']; then, finding foo.__dict__['I'] to be present, it
returns that (1) instead. When the result of the expression is then
assigned to self.I, no delegation takes place and the value 2 is stored
in self.__dict__['I'].

A note of caution: you might be tempted to think that with objects such
as lists, which implement the __iadd__ method, no assignment would take
place. This is actually not the case -- it works exactly the same way!
To use another variation of your example:

.... lst = ['a','b','c']
.... def __init__(self):
.... print self.__dict__
.... self.lst += [1,2,3]
.... print self.__dict__
....{}
{'lst': ['a','b','c',1,2,3]}

foo.l ['a','b','c',1,2,3]
id(foo.lst) == id(a.lst)

True

The list is mutated in-place, but there is _also_ an implicit
assignment to self.lst. In other words, it expands to:

self.lst = self.lst.__iadd__([1,2,3])

Normally this implicit assignment doesn't matter, since the context the
variable is stored into is the same one it was retrieved from. It's
only in situations like yours where it becomes important.

-- David

 

[Russell Warren]

D'oh... I just realized why this is happening. It is clear in the
longhand as you say, but I don't think in the way you descibed it (or
I'm so far gone right now I have lost it).

self.I += 1

is the same as

self.I = self.I + 1

and when python tries figures out what the 'self.I' is on the right
hand side. it of course ends up having to move up to the base class
foo.__dict__ because there is no 'I' in self.__dict__ yet. So it ends
up effectively being:

self.I = foo.I + 1

which explains where the "self.I = foo.I' that I was claiming was being
done magically comes from.

What my head was thinking was that the 'self.I' lookup would move up to
get foo.__dict__['I'], and that I would effectively get 'foo.I += 1',
but this is a bit of a brain fart and is just plain wrong.

I should have seen that earlier... oh well. I'm happy that it is
perfectly clear where it comes from, now. It still does look odd when
you do a simplistic comparison of the behaviour of 'x += 1' and 'self.I
+= 1', but I suppose that that's just the way the lookup scheme
crumbles. An unfortunate (and rare?) quirk, I guess.

It still might be nice were python to just block out this potential
confusion with an Exception... it seems that class vs instance
attribute referencing is confusing enough for people without having
this type of potential confusion lurking around the syntax. It seems
like such a simple thing, but to understand the outcomes requires
knowing how the name lookup scheme works, how mutable/immutable objects
are dealt with, and what the += keystroke-saver/macro operator is
actually doing. That this is stuff that someone coding in python
should understand could certainly be argued, though...

Russ

 

[Russell Warren]

Thanks for the additional examples, David (didn't see this before my
last post). All of it makes sense now, including those examples.

Russ