Bypassing __getattribute__ for attribute access

[Adam Donahue]

As an exercise I'm attempting to write a metaclass that causes an
exception to be thrown whenever a user tries to access
'attributes' (in the traditional sense) via a direct reference.

Consider:

class X( object ):
y = 'private value'
def get_y( self ): return self.y

Normally one can access y here via:

X().y

or

X().get_y()

I want the former case, however, to throw an exception.

I figured the way to do this would be to introduce a metaclass that
overrides the default __getattrribute__ call and throws an exception.
So my first attempt was something like:

class XType( type ):
def __my_getattribute__( self, name ):
raise AttributeError()
def __init__( klass, name, bases, dict ):
super( XType, klass ).__init__( name, bases, dict )
setattr( klass, '__getattribute__',
klass.__my_getattribute__ )

But whereas the X().y attribute behaves as I intend, the X().get_y()
returns raises that exception as well:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in __my_getattribute__
AttributeError

So it looks as if 'attribute' here means any key in self.__dict__,
whether referenced via self.var, self.__dict__['var'] (because this
references __dict__), or getattr( self, 'var' ) (which is the same as
a direct self.var access, I believe).

So I tried:

class XType( type ):
def __my_getattribute__( self, name ):
if name != '__dict__':
raise AttributeError()
return super( self.__class__,
self ).__getattribute__( name )
def __init__( klass, name, bases, dict ):
super( XType, klass ).__init__( name, bases, dict )
setattr( klass, '__getattribute__',
klass.__my_getattribute__ )

This allows me to access X().__dict__ directly (and then
X().__dict__['y']), but it still limits caller access to the get_y()
method.

It sounds then like the "solution" will be to check whether the name
referenced is called __dict__ or is a method or function type,
otherwise throw the exception, and to ensure all internal calls are
handled via self.__dict__[name] not self.name.

Something like:

import types
class XType( type ):
def __my_getattribute__( self, name ):
if name != '__dict__' and not
isinstance( self.__dict__[name], types.FunctionType ):
raise AttributeError()
return super( self.__class__,
self ).__getattribute__( name )
def __init__( klass, name, bases, dict ):
super( XType, klass ).__init__( name, bases, dict )
setattr( klass, '__getattribute__',
klass.__my_getattribute__ )

Of course this is imperfect as a user can simply bypass the
__getattribute__ call too and access __dict__ directly, but it's
closer to what I was thinking. The problem is the return value for
functions is not bound - how do I bind these to the associated
instance?

(Caveat - I am not sure whether using __get__ itself in lieu of
__getattribute__ would be a better solution; but I would like to see
how binding would be done here for general knowledge.)

Thanks.

Adam

 

[Bruno Desthuilliers]

Adam Donahue a écrit :

As an exercise I'm attempting to write a metaclass that causes an
exception to be thrown whenever a user tries to access
'attributes' (in the traditional sense) via a direct reference.

I guess you're new to Python, and coming from either C++ or Java. Am I
wrong ?-)

And without even reading further, I can tell you're doing something
pretty complicated that just don't work.

(Ok, I cheated - I did read further !-)

Consider:

class X( object ):
y = 'private value'
def get_y( self ): return self.y

Normally one can access y here via:

X().y

or

X().get_y()

I want the former case, however, to throw an exception.

So called "private" or "protected" attributes (IOW, implementation stuff
the client code should not mess with) are denoted by a single leading
underscore. IOW, 'y' should be '_y'. It won't of course prevent anyone
to access the attribute, but then it's not your responsability anymore.

I know this sound surprising to C++/Java programmers, but experience
prove that it just work.


Now if all you want is to reproduce the Java systematic-getter-setter
dance - that is, use getters/setters 'just in case' you'd want to
refactor (which, FWIW, is the only rationale behind accessors), you just
don't need this with Python. We do have computed attributes here, so the
simplest thing is to start with a plain attribute, then refactor it into
a computed one if and when the need arises. This is *totally*
transparent to client code.

I figured the way to do this would be to introduce a metaclass that
overrides the default __getattrribute__ call and throws an exception.
So my first attempt was something like:

class XType( type ):
def __my_getattribute__( self, name ):
raise AttributeError()
def __init__( klass, name, bases, dict ):
super( XType, klass ).__init__( name, bases, dict )
setattr( klass, '__getattribute__',
klass.__my_getattribute__ )

But whereas the X().y attribute behaves as I intend, the X().get_y()
returns raises that exception as well:

Indeed. __getattribute__ is invoked for *each and every* attribute
lookup - including methods, since methods are attributes too. FWIW,
__getattribute__ it's something you should not mess with unless you know
what you're doing and why you're doing it.

So it looks as if 'attribute' here means any key in self.__dict__,

The '.' is the lookup operator. As soon as you have obj.anyname, you do
an attribute lookup (wether it fails or succeeds is another question).
And __getattribute__ is the implementation for this operator. So given
how you wrote your custom __getattribute__, you just made attribute
lookup impossible.

And FWIW, attribute lookup is much more complex than just looking up the
instance's __dict__ - it also looks up the class __dict__, then the
parent's classes __dict__, then look for a custom __getattr__ method
(which is used when the attribute has not been found so far). And if the
attribute found is a class attribute that implements the descriptor
protocol, then __getattribute__ is responsible for invoking this
protocol. IOW, __getattribute__ is one of the most critical magic methods.

whether referenced via self.var, self.__dict__['var'] (because this
references __dict__), or getattr( self, 'var' ) (which is the same as
a direct self.var access, I believe).

Practically, yes.

So I tried:

class XType( type ):
def __my_getattribute__( self, name ):
if name != '__dict__':
raise AttributeError()
return super( self.__class__,
self ).__getattribute__( name )
def __init__( klass, name, bases, dict ):
super( XType, klass ).__init__( name, bases, dict )
setattr( klass, '__getattribute__',
klass.__my_getattribute__ )

This allows me to access X().__dict__ directly (and then
X().__dict__['y']), but it still limits caller access to the get_y()
method.

cf above.

It sounds then like the "solution" will be to check whether the name
referenced is called __dict__ or is a method or function type,
otherwise throw the exception, and to ensure all internal calls are
handled via self.__dict__[name] not self.name.

My my my. Trouble ahead...

Something like:

import types
class XType( type ):
def __my_getattribute__( self, name ):
if name != '__dict__' and not
isinstance( self.__dict__[name], types.FunctionType ):
raise AttributeError()
return super( self.__class__,

*never* use self.__class__ (or type(self) etc) when calling super(). You
*really* want to pass the exact class here - else you'll have *very*
strange results.

self ).__getattribute__( name )
def __init__( klass, name, bases, dict ):
super( XType, klass ).__init__( name, bases, dict )
setattr( klass, '__getattribute__',
klass.__my_getattribute__ )

My my my...

Of course this is imperfect as a user can simply bypass the
__getattribute__ call too and access __dict__ directly,

Indeed. The fact is that there's just no way to prevent client code to
access your implementation. Period. So relax, stop fighting against the
langage, and learn to use it how it is.

but it's
closer to what I was thinking. The problem is the return value for
functions is not bound - how do I bind these to the associated
instance?

func.__get__(obj, obj.__class__)

But that should not be done when the function is an instance attribute -
only when it's a class one. And any class attribute implementing the
descriptor protocol should be treated that way.

(Caveat - I am not sure whether using __get__ itself in lieu of
__getattribute__ would be a better solution; but I would like to see
how binding would be done here for general knowledge.)

(simplified) In the normal case, when the attribute looked up happens to
be a class attribute and implements the descriptor protocol,
__getattribute__ returns attr.__get__(obj, type(obj). What attr.__get__
returns is up to whoever implemented type(attr). In the case of
functions, anyway, __get__ returns a method object, which is a callable
object wrapping the function, the target object and the class. When
called, this method object insert the target object (or class if it's a
classmethod) in front of the args list, and invoke the function with
this new args list. Which is why you need to declare self (or cls) as
first argument of a 'method' but not to explicitely pass it at call time.

Anyway : forget about "real" privacy in Python (FWIW, neither Java nor
C++ have "real" privacy - there are ways to bypass access restrictors in
both languages), just use the single leading underscore convention and
you'll be fine. And don't write explicit accessors - in fact, don't
write accessors at all until you need them, and when you need them, use
a property or a custom descriptor object, so it's transparant to client
code.

HTH

 

[Adam Donahue]

Bruno,

I appreciate your attempt to answer my questions below, although I
think my main point was lost amongst all your commentary and
assumptions. :^) I'm not inexperienced, but I take the blame for
the rambling initial post, though, which probably lead to the
confusion.

So let me be more direct:

From reading it seems that, indeed, __getattribute__ handles calling

attribute.__get__( obj, type(obj ), which does the binding of the
attribute to the object.

That is:

<thingy>.<attribute>

in turn is a call to

<thingy>.__getattribute__( '<attribute>' ) <1>

The default body of __getattribute__ in turn fetches the associated
value (by traversing the class hierarchy if necessary) and examines
its type in order to determine what to return.

Let's assume the ultimate value associated with the attribute (key) is
v.

If type(v) is a function, __getattribute__ returns
v.__get__( <thingy>, type( <thingy> )
If type(v) is, say, an integer, __getattribute__ returns v unmolested.

And so forth.

So:.... a = 1
.... class b( object ): pass
.... def c( self ): pass
....<bound method X.c of <__main__.X object at 0x81b2b4c>>

If my interpretation is correct, the X.c's __getattribute__ call knows
the attribute reference is via a class, and thus returns an unbound
method (though it does convert the value to a method). Likewise,
x.c's __getattribute__ returns the value as a method bound to the x
instance.

How does __getattribute__ knows the calling context. Its first
argument is the attribute name from what I can tell, not the object
calling it.

Is this correct so far?

Moving on to __get__. Assume:

class X( object ):
def foo(self):
print `self`
x = X()

Then:

x.foo()

Is similar (perhaps the same as) to:

X.foo.__get__( x, X )()

(__getattribute__ performs the transformation automatically when one
references via the . operator.)

And so one can do:
.... x = 1
....5

The logical next question then is how does one best add a new method
to this class so that future references to x.set_x() and X.set_x will
properly resolve? It seems the answer would be to somehow add to
X.__dict__ a new value, with key 'set_x', value the function set_x.

From there on the . operator I assume would perform the binding to X

or x as needed on-the-fly.

Adam

 

[Steven Bethard]

... def c( self ): pass
...
<bound method X.c of <__main__.X object at 0x81b2b4c>>

If my interpretation is correct, the X.c's __getattribute__ call knows
the attribute reference is via a class, and thus returns an unbound
method (though it does convert the value to a method).

No, that's wrong. It's not the __getattribute__ of "X.c", it's the
__getattribute__ of "X" itself::
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
AttributeError: 'function' object has no attribute 'c'

In the former case, we use the type's __getattribute__, and we get the
unbound method as expected. In the latter case, we ask the unbound
method object for an attribute which it doesn't have

Basically, classes define a __getattribute__ that looks something like::

def __getattribute__(self, name):
value = object.__getattribute__(self, name)
if hasattr(value , '__get__'):
return value .__get__(None, self)
return value

For more information, read:

http://users.rcn.com/python/download/Descriptor.htm

STeVe

 

[Bruno Desthuilliers]

Adam Donahue a écrit :

Bruno,

I appreciate your attempt to answer my questions below, although I
think my main point was lost amongst all your commentary and
assumptions. :^)

Possibly. I sometimes tend to get a bit verbose !-)

I'm not inexperienced,

Obviously not.

but I take the blame for
the rambling initial post, though, which probably lead to the
confusion.

So let me be more direct:
>
attribute.__get__( obj, type(obj ), which does the binding of the
attribute to the object.
Yes.

That is:

<thingy>.<attribute>

in turn is a call to

<thingy>.__getattribute__( '<attribute>' ) <1>
Yes.

The default body of __getattribute__ in turn fetches the associated
value (by traversing the class hierarchy if necessary) and examines
its type in order to determine what to return.
Yes.

Let's assume the ultimate value associated with the attribute (key) is
v.

If type(v) is a function,

If v supports the descriptor protocol (which is the case for function
objects and property objects) AND v is a class attribute

__getattribute__ returns
v.__get__( <thingy>, type( <thingy> )
Yes.

If type(v) is, say, an integer,

If v doesn't support the descriptor protocol

__getattribute__ returns v unmolested.
Yes.

And so forth.

So:


... a = 1

You understand that 'a' is a class attribute, don't you ?

... class b( object ): pass
... def c( self ): pass
...


1

Returns type(x).a

<class '__main__.b'>
idem


<bound method X.c of <__main__.X object at 0x81b2b4c>>
Yes.

If my interpretation is correct, the X.c's __getattribute__ call knows
the attribute reference is via a class,
Yes

and thus returns an unbound
method

It returns c.__get__(None, X). Which is implemented to return an unbound
method if the first arg is None.

(though it does convert the value to a method).

__getattribute__ doesn't convert anything here - and FWIW, doesn't care
if c is a function or property or whatnot. The only thing it looks for
is if c has a __get__ method.

Likewise,
x.c's __getattribute__ returns the value as a method bound to the x
instance.

Yes, because then __getattribute__ returns x.c.__get__(x, X), which,
since c is a function, returns a bound method.

How does __getattribute__ knows the calling context. Its first
argument is the attribute name from what I can tell, not the object
calling it.

Really ?-)

__getattribute__ is itself a function and an attribute of a class (and
FWIW, it's signature is __getattribute__(self, name)). So when itself
looked up, it returns a method object, which when called passes the
'target' object as first arg.

You already know that Python's classes are objects (instances of type
'type'). So you should by now understand how __getattribute__ knows it's
target. Got it ? Yes, right: when looking up attributes on a class, it's
the class's class (IOW: the type) __getattribute__ method that is
invoked !-)

Is this correct so far?

cf above.

Moving on to __get__. Assume:

class X( object ):
def foo(self):
print `self`
x = X()

Then:

x.foo()

Is similar (perhaps the same as) to:

X.foo.__get__( x, X )()

Almost but not quite. It's:

x.foo() <=> X.foo(x) <=> X.foo.im_func.__get__(x, X)()

It's a bit less confusing when the function is defined outside the class:

def bar(self):
print self

X.bar = bar

x = X()

Now you have:

(__getattribute__ performs the transformation automatically when one
references via the . operator.)

And so one can do:


... x = 1

Here, x is a class attribute.

Now this creates an instance attribute that will shadow the class attribute.

...


5

So far, so good.

The logical next question then is how does one best add a new method
to this class so that future references to x.set_x() and X.set_x will
properly resolve? It seems the answer would be to somehow add to
X.__dict__ a new value, with key 'set_x', value the function set_x.

Yes. Which is very simply done by just binding set_x to X.set_x. Just
like I did above. Dynamically adding methods to classes is pretty
straightforward, the tricky point is to dynamically add methods to
instances, since the descriptor protocol is only triggered for class
attributes. But you obviously found how to do it using func.__get__(obj,
type(obj)) !-)

or x as needed on-the-fly.

Yes.

NB: please some guru around correct me if I said something wrong (Alex ?
Tim ? Fredrick ? If you hear me ?)

 

[Chris Mellon]

On Thu, 2007-10-25 at 23:13 +0200, Bruno Desthuilliers wrote:

Yes. Which is very simply done by just binding set_x to X.set_x. Just
like I did above. Dynamically adding methods to classes is pretty
straightforward, the tricky point is to dynamically add methods to
instances, since the descriptor protocol is only triggered for class
attributes. But you obviously found how to do it using func.__get__(obj,
type(obj)) !-)

This is the greasy, getting your hands dirty way. I vastly prefer (and
reccomend) using the new module:
.... pass
.... .... print self
....

 

[Bruno Desthuilliers]

Chris Mellon a écrit :

On Thu, 2007-10-25 at 23:13 +0200, Bruno Desthuilliers wrote:
<snip excellent breakdown>


This is the greasy, getting your hands dirty way. I vastly prefer (and
reccomend) using the new module:

Indeed.

 

[Adam Donahue]

Thank you all for the detailed replies, I appreciate it. I only read
up on this yesterday morning, but I feel I've gotten a lot of insight
in a short time thanks to your contributions to this thread. Useful
all around!

Adam

 

[Michele Simionato]

As an exercise I'm attempting to write a metaclass that causes an
exception to be thrown whenever a user tries to access
'attributes' (in the traditional sense) via a direct reference.

Well, now thanks to Bruno and the others you know everything about
attribute access in Python. Still, you may be interested in this
recipe, which involves __setattr__ and non __getattribute__:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252158

Michele Simionato