Unexpected behavior of read only attributes and super

[Samuel M. Smith]

I have been playing around with a subclass of dict wrt a recipe for
setting dict items using attribute syntax.
The dict class has some read only attributes that generate an
exception if I try to assign a value to them.
I wanted to trap for this exception in a subclass using super but it
doesn't happen.
I have read Guido's tutorial on new style classes and Shalabh's
tuturial on new style attributes and methods, and thought
I understood what super was doing. But there is no discussion on read
only attributes and their associated magic.

It seems to me that there is some undocumented magic that does not
make sense to me.

for example

d = dict()
d.__iter__ returns
<method-wrapper object at 0x260a10>

If I try to assign a value to d.__iter__
d.__iter__ = False
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'dict' object attribute '__iter__' is read-only

If I use the setattr method, I get the exception also as expected

d.__setattr__('__iter__', False)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'dict' object attribute '__iter__' is read-only



but if I subclass and use super

class SD(dict):
pass

s = SD()
s.__iter__ returns
<method-wrapper object at 0x260a10>
so the object s has this attribute
hasattr(s,'__iter__') also returns
True

but s.__dict__ is empty at this stage so s has inherited this
attribute as a method



so far so good.
If I assign a value
s.__iter__ = False

it lets me but adds the attribute to s.__dict__ thereby shadowing the
method, no surprises yet
s.__dict__ returns
{'__iter__': False}

but I want to know if the attribute is a read only attribute of the
super class so that I don't shadow it
in the subclass. So I start over and try

s = SD()
super(SD,s).__setattr__('__iter__', True)

Expecting to get the ReadOnly exception but I don't get the exception.
Instead the attribute is added to s.__dict__.
s.__dict__ returns
{'__iter__': True}

Shouldn't the super __setattr__ call be using the same code as the
direct call to dict's __setattr__
and therefore produce an exception?

 

[Steven Bethard]

The dict class has some read only attributes that generate an exception
if I try to assign a value to them.
I wanted to trap for this exception in a subclass using super but it
doesn't happen.

class SD(dict):
pass
[snip]
s = SD()
super(SD,s).__setattr__('__iter__', True)

Expecting to get the ReadOnly exception but I don't get the exception.

Note that __iter__ is on the dict *type* not dict instances. Try this:

py> class SD(dict):
.... pass
....
py> super(SD, SD).__init__ = False
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
AttributeError: 'super' object attribute '__init__' is read-only

You can always shadow class-level attributes in the instance dict.
(That's what you were doing.) If you want to (try to) replace an
attribute in the class dict, you need to use the class object, not an
instance object.

HTH,

STeVe

 

[Samuel M. Smith]

The dict class has some read only attributes that generate an
exception
if I try to assign a value to them.
I wanted to trap for this exception in a subclass using super but it
doesn't happen.

class SD(dict):
pass
[snip]
s = SD()
super(SD,s).__setattr__('__iter__', True)

Expecting to get the ReadOnly exception but I don't get the
exception.

Note that __iter__ is on the dict *type* not dict instances. Try
this:

py> class SD(dict):
... pass
...
py> super(SD, SD).__init__ = False
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
AttributeError: 'super' object attribute '__init__' is read-only

You can always shadow class-level attributes in the instance dict.
(That's what you were doing.) If you want to (try to) replace an
attribute in the class dict, you need to use the class object, not an
instance object.

I guess that's where my understanding breaks down. I thought the only
way to access class attributes was by
calling the class directly as your example indicates but __iter__ is
a class attribute that I can access from the instance
at least to read it. So what determines which class attributes get
copied to the instance and which ones don't?

 

[Steven Bethard]

I guess that's where my understanding breaks down. I thought the only
way to access class attributes was by
calling the class directly as your example indicates but __iter__ is a
class attribute that I can access from the instance
at least to read it. So what determines which class attributes get
copied to the instance and which ones don't?

When "reading" an attribute, Python looks through the namespaces in the
order (instance, type). So if the attribute exists in the instance, the
instance-level value is returned. If the attribute does not exist in
the instance, but does exist in the class, then the class-level value is
returned:
.... x = 1
........ x = 1
.... def __init__(self):
.... self.x = 2
....1

When "writing" an attribute (i.e. using the assignment statement),
Python does not try to do any namespace searching. Thus if you use the
instance in an assignment statement, then it is the instance's
attributes that get modified, and if you use the class in an assignment
statement, then it is the class's attributes that get modififed:
.... pass
....Traceback (most recent call last):
.... pass
....1

HTH,

STeVe

P.S. Note that there is an additional complication resulting from the
fact that functions are descriptors:
.... pass
....<method-wrapper object at 0x00E74A10>

Even though the C instance is accessing the __iter__ function on the
class, it gets back a different value because descriptors return
different values depending on whether they are accessed from a class or
an instance. I don't think you need to understand this to solve your
problem though, so I won't go into any more details unless you think it
would be helpful.

 

[Samuel M. Smith]

P.S. Note that there is an additional complication resulting from the
fact that functions are descriptors:

... pass
...
<method-wrapper object at 0x00E74A10>

Even though the C instance is accessing the __iter__ function on the
class, it gets back a different value because descriptors return
different values depending on whether they are accessed from a
class or
an instance. I don't think you need to understand this to solve your
problem though, so I won't go into any more details unless you
think it
would be helpful.

I found your explanation very helpful. After reading it I went back
and read
my Nutshell book and realized that the explanation was in there but I
didn't "get" it until now.
Although I did find an exception to the rule for attribute writes.
(See !Whoops below)

If you would care to elaborate on the how the lookup differs with
method descriptor
it would be most appreciated. Mostly because it seems that having
slots defined
changes the method lookup as opposed to the variable lookup and
apparently some of the type class
variables are special enough that they have their own rules.

This might help explain why it is that when I define __slots__, the
behavior when writing an attribute is different for
attributes that exist in the class versus attributes that exist in
__slots__ versus attributes that
do not exist at all. It is also different if the class attribute is a
method vesus a variable.

For example
.... __slots__ = ['a','b']
....Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: a

So slot defined but not assigned gets error
5

OK here
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'C' object has no attribute 'c'

Surprise error gives no clue that slots is the reason for the error
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'C' object has no attribute 'c'

ditto


Now the behavior is different for class variables and methods when
slots defined
versus when slots is not defined.
Traceback (most recent call last):
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'super' object has only read-only attributes (assign
to .__iter__)<class '__main__.C'>

it let me assign it! But not shadowedTraceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'C' object has no attribute '__dict__'

!Whoops now I am confused again. Didn't you say

When "writing" an attribute (i.e. using the assignment statement),
Python does not try to do any namespace searching. Thus if you use
the
instance in an assignment statement, then it is the instance's
attributes that get modified, and if you use the class in an
assignment
statement, then it is the class's attributes that get modififed:

Then why wasn't __class__ added to c.__dict__ ? Looks like namespace
searching to me.

So to cross check if slots is not defined
.... pass
....{'__iter__': 1}

try again a different way

class B(C):
.... pass
....{'__iter__': 4}

OK but maybe __class__ is magic, so I tried again
.... a = 0
....{'a': 4}

OK __class__ is special

now with slots defined
.... __slots__ = ['b']
.... a = 0
....Traceback (most recent call last):

So the rule is that when __slots__ is defined class variables become
read only.

What if the class variable is included in __slots__
.... __slots__ = ['b']
.... b = 1
....Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'C' object attribute 'b' is read-only

So even though b is in slots I still can't create an instance
variable by that name
and shadow the class variable.

It feels like the implementation of slots is half baked.

Finally
Since the "way" of python is that if an object does not have an
attribute
but you can assign it one then it creates one dynamically (in the
same 'way' that if a variable does not
exist is creates one upon assignment).
Because slots break this paradigm then at the very least the error
messages should point out that this object
is using slots "so beware".

For example I would prefer something like the following

c.a
AttributeError: Slot 'a' not yet assigned

c.c
AttributeError: No slot named 'c' on instance

c.c = 4
AttributeError: No slot named 'c' on instance

if change rule to not access class from instance when slots define
c.__iter__ = 4
AttributeError: No slot named '__iter__' on instance

or with current behavior
AttributeError: No slot name '__iter__' class 'C' object attribute
'__iter__' is read-only


super(C,c).__iter__ = 4
TypeError: 'super' object has only read-only attributes (assign
to .__iter__)

 

[Steven Bethard]

If you would care to elaborate on the how the lookup differs with
method descriptor it would be most appreciated.

For the more authoritative guide, see:
http://users.rcn.com/python/download/Descriptor.htm

The basic idea is that a descriptor is an object that sits at the class
level, and redefines how some attribute accesses work. Consider a
simple example:
.... def __get__(self, obj, objtype=None):
.... if obj is None:
.... return 'called from class %r' % objtype
.... else:
.... return 'called from instance %r' % obj
........ d = D()
....'called from instance <__main__.C object at 0x00E73A30>'

As you can see, instances of the D class, when used as class attributes,
can tell whether they're being called by the class or the instance.
This means that descriptors with a __get__ method defined can do just
about anything on an attribute access.

Note that all functions in Python are descriptors, and they use the
__get__ method to return either an unbound method or a bound method,
depending on whether they were called from the type or the instance:
.... return x*2
........ func = f
....

This might help explain why it is that when I define __slots__, the
behavior when writing an attribute is different

Yes. Defining __slots__ basically tells the class to create descriptors
for each name in the list. So, for example:

... __slots__ = ['a','b']
...

Creates two descriptors that are attributes of class C: one named "a"
and one named "b".

Now the behavior is different for class variables and methods when
slots defined versus when slots is not defined.

Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'C' object attribute '__iter__' is read-only

Here, Python is trying to set the "__iter__" attribute of the object.
Since you defined __slots__, it tells you that it can't. So it never
even looks at the type.

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'super' object has only read-only attributes (assign to
.__iter__)

In this case, you explicitly request the superclass, so you get the same
error as before because you bypass the __slots__, which are defined for
instances of C, not for instances of the superclass, dict.

Then why wasn't __class__ added to c.__dict__ ? Looks like namespace
searching to me.

No, as you conclude later, __class__ is special, so you can still assign
to __class__ even when __slots__ is defined because it's not considered
a normal attribute. But note that __class__ is an *instance* attribute,
not a class attribute, so "c.__class__ = C" changes the class of that
single instance, and makes no change to the type:
.... pass
........ pass
....

>>> c1.__class__ = D
>>> C, c1, c2

[/QUOTE]
(<class '__main__.C'>, <__main__.D object at 0x00E73A30>, <__main__.C
object at 0x00E73210>)

So no, even with __class__, you're only assigning to the instance, and
so Python's not searching any additional namespaces.

now with slots defined
... __slots__ = ['b']
... a = 0
...Traceback (most recent call last):
5

So the rule is that when __slots__ is defined class variables become
read only.

That's not quite right. As you show above, class variables are still
modifiable from the class object. But yes, defining __slots__ means
that, from an instance, you can only modify the attributes defined in
__slots__.

What if the class variable is included in __slots__
... __slots__ = ['b']
... b = 1
...Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'C' object attribute 'b' is read-only

So even though b is in slots I still can't create an instance variable
by that name and shadow the class variable.

Yes, this behavior is documented:
http://docs.python.org/ref/slots.html

"""
__slots__ are implemented at the class level by creating descriptors
(3.3.2) for each variable name. As a result, class attributes cannot be
used to set default values for instance variables defined by __slots__;
otherwise, the class attribute would overwrite the descriptor assignment.
"""

The documentation isn't great, I'll agree, but the result is basically
that if you combine __slots__ with class attributes of the same name,
you're asking for trouble because your 'b' class attribute and your 'b'
slot both reside at the class level. IMO, the manual should probably
explicitly say that the behavior of such a class is undefined.

It feels like the implementation of slots is half baked.

I wouldn't go that far. Once you understand that __slots__ reside as
descriptors at the class level, you can see why most of the problems
arise. That said, I've never had the need to use __slots__ in any real
world code.

Because slots break this paradigm then at the very least the error
messages should point out that this object
is using slots "so beware".

For example I would prefer something like the following

c.a
AttributeError: Slot 'a' not yet assigned

c.c
AttributeError: No slot named 'c' on instance

c.c = 4
AttributeError: No slot named 'c' on instance

if change rule to not access class from instance when slots define
c.__iter__ = 4
AttributeError: No slot named '__iter__' on instance

or with current behavior
AttributeError: No slot name '__iter__' class 'C' object attribute
'__iter__' is read-only

super(C,c).__iter__ = 4
TypeError: 'super' object has only read-only attributes (assign to
.__iter__)

These seem like pretty reasonable requests. I would suggest adding them
as a feature request:
http://sourceforge.net/tracker/?group_id=5470&atid=355470
The one thing I don't know is how hard it is to produce these messages
-- I don't know the C-level code for __slots__ at all. Hopefully
someone else will!

STeVe

 

[Samuel M. Smith]

No, as you conclude later, __class__ is special, so you can still
assign
to __class__ even when __slots__ is defined because it's not
considered
a normal attribute. But note that __class__ is an *instance*
attribute,
not a class attribute, so "c.__class__ = C" changes the class of that
single instance, and makes no change to the type:

So no, even with __class__, you're only assigning to the instance, and
so Python's not searching any additional namespaces.

Thank you so much, very helpful comments. I hope it helps others as
well.
At the very least I now understand a little better about some of the
special
rules for special attributes.