hasattr + __getattr__: I think this is Python bug

[dmitrey]

hi all,
I have a class (FuncDesigner oofun) that has no attribute "size", but
it is overloaded in __getattr__, so if someone invokes
"myObject.size", it is generated (as another oofun) and connected to
myObject as attribute.

So, when I invoke in other code part "hasattr(myObject, 'size')",
instead o returning True/False it goes to __getattr__ and starts
constructor for another one oofun, that wasn't intended behaviour.
Thus "hasattr(myObject, 'size')" always returns True. It prevents me
of some bonuses and new features to be done in FuncDesigner.
True

Could you fix it?

 

[Chris Rebert]

hi all,
I have a class (FuncDesigner oofun) that has no attribute "size", but
it is overloaded in __getattr__, so if someone invokes
"myObject.size", it is generated (as another oofun) and connected to
myObject as attribute.

So, when I invoke in other code part "hasattr(myObject, 'size')",
instead o returning True/False it goes to __getattr__ and starts
constructor for another one oofun, that wasn't intended behaviour.
Thus "hasattr(myObject, 'size')" always returns True. It prevents me
of some bonuses and new features to be done in FuncDesigner.

True

Could you fix it?

There's probably some hackery you could do to check whether hasattr()
is in the call stack, and then not dynamically create the attribute in
__getattr__ if that's the case, but that's obviously quite kludgey.

/Slightly/ less hackish: Replace hasattr() in the __builtin__ module
with your own implementation that treats instances of FuncDesigner
specially.

Least ugly suggestion: Just don't use hasattr(); use your `x in
dir(y)` trick instead.

Subject: [...] I think this is Python bug

Nope, that's just how hasattr() works. See
http://docs.python.org/library/functions.html#hasattr (emphasis mine):
"""
hasattr(object, name)
The arguments are an object and a string. The result is True if
the string is the name of one of the object’s attributes, False if
not. (***This is implemented by calling getattr(object, name)*** and
seeing whether it raises an exception or not.)
"""

I suppose you could argue for the addition of a __hasattr__ special
method, but this seems like a really rare use case to justify adding
such a method (not to mention the language change moratorium is still
currently in effect).

Cheers,
Chris

 

[dmitrey]

There's probably some hackery you could do to check whether hasattr()
is in the call stack, and then not dynamically create the attribute in
__getattr__ if that's the case, but that's obviously quite kludgey.

It's too unreliable solution, hasattr may or may not appear in stack
wrt different cases

/Slightly/ less hackish: Replace hasattr() in the __builtin__ module
with your own implementation that treats instances of FuncDesigner
specially.

It's too unreliable as well

Least ugly suggestion: Just don't use hasattr(); use your `x in
dir(y)` trick instead.

something in dir() consumes O(n) operations for lookup, while hasattr
or getattr() require O(log(n)). It matters for me, because it's inside
deeply nested mathematical computations FuncDesigner is made for.

Subject: [...] I think this is Python bug

Nope, that's just how hasattr() works. Seehttp://docs.python.org/library/functions.html#hasattr(emphasis mine):
"""
hasattr(object, name)
    The arguments are an object and a string. The result is True if
the string is the name of one of the object’s attributes, False if
not. (***This is implemented by calling getattr(object, name)*** and
seeing whether it raises an exception or not.)
"""

Thus I believe this is very ugly implementation. Some code implemented
via "__getattr__" can start to execute arbitrary Python code, that is
certainly not desired behaviour "hasattr" was designed for (to perform
a check only), and it's too hard to reveal the situations, that makes
a potential holes for viruses etc.

Moreover, the implementation via "try getattr(object, name)" slowers
code evaluation, I wonder why it is implemented so.

I suppose you could argue for the addition of a __hasattr__ special
method, but this seems like a really rare use case to justify adding
such a method (not to mention the language change moratorium is still
currently in effect).

I think much more easier solution would be to implement an additional
argument to hasattr, e.g. hasattr(obj, field,
onlyLookupInExistingFields = {True/False}). I think by default it's
better set to True, now it behaves like False.

D.

 

[Jean-Michel Pichavant]

hi all,
I have a class (FuncDesigner oofun) that has no attribute "size", but
it is overloaded in __getattr__, so if someone invokes
"myObject.size", it is generated (as another oofun) and connected to
myObject as attribute.

So, when I invoke in other code part "hasattr(myObject, 'size')",
instead o returning True/False it goes to __getattr__ and starts
constructor for another one oofun, that wasn't intended behaviour.
Thus "hasattr(myObject, 'size')" always returns True. It prevents me
of some bonuses and new features to be done in FuncDesigner.


True

Could you fix it?

Quite simple, when calling b.size, return the computed value but do not
set it as attribute, the value will be computed on each b.size call, and
hasattr w. If you don't want to compute it each time, cause there no
reason for it to change, use a private dummy attribute to record the
value instead of size (__size for instance) and return the value of
__size when getting the value of size.


class Foo(object):
def __init__(self):
self.__size = None

def __getattr__(self, name):
if name == "size":
if self.__size is None:
self.__size = 5 # compute the value
return self.__size
raise AttributeError(name)

b = Foo()
print 'size' in dir(b)
print hasattr(b, 'size')
print 'size' in dir(b)

False
True
False

JM

 

[dmitrey]

Quite simple, when calling b.size, return the computed value but do not
set it as attribute, the value will be computed on each b.size call, and
hasattr w. If you don't want to compute it each time, cause there no
reason for it to change, use a private dummy attribute to record the
value instead of size (__size for instance) and return the value of
__size when getting the value of size.

class Foo(object):
    def __init__(self):
        self.__size = None

    def __getattr__(self, name):
        if name == "size":
            if self.__size is None:
                self.__size = 5 # compute the value
            return self.__size
        raise AttributeError(name)

b = Foo()
print 'size' in dir(b)
print hasattr(b, 'size')
print 'size' in dir(b)

False
True
False

JM

This doesn't stack with the following issue: sometimes user can write
in code "myObject.size = (some integer value)" and then it will be
involved in future calculations as ordinary fixed value; if user
doesn't supply it, but myObject.size is involved in calculations, then
the oofun is created to behave like similar numpy.array attribute.

 

[dmitrey]

e.g. one that just looks in the object's dictionary so as to avoid returning true for properties or other such fancy attributes.

So can anyone explain me how to look into object's dict? As I have
wrote, "something in dir(...)" requires O(numOfFields) while I would
like to use o(log(n))

How about using a property instead of the __getattr__() hook? A property is a computed attribute that (among other things) plays much nicer with hasattr.

Could anyone provide an example of it to be implemented, taking into
account that a user can manually set "myObject.size" to an integer
value?

 

[Neil Cerutti]

This doesn't stack with the following issue: sometimes user can
write in code "myObject.size = (some integer value)" and then
it will be involved in future calculations as ordinary fixed
value; if user doesn't supply it, but myObject.size is involved
in calculations, then the oofun is created to behave like
similar numpy.array attribute.

Telling them, "Don't do that," is a good solution in Python.

 

[dmitrey]

Telling them, "Don't do that," is a good solution in Python.

But this is already documented feature, and it works as intended, so
moving it into something like "myObject._size" will bring backward
incompatibility and break all FuncDesigner user API and style, where
no underlines are present, it will seem like a hack that it really
is.

Sometimes apriory knowing size value as fixed integer brings some code
speedup, also, if a user supplies the value, a check for computed
value wrt the provided size is performed each time.

 

[Jean-Michel Pichavant]

This doesn't stack with the following issue: sometimes user can write
in code "myObject.size = (some integer value)" and then it will be
involved in future calculations as ordinary fixed value; if user
doesn't supply it, but myObject.size is involved in calculations, then
the oofun is created to behave like similar numpy.array attribute.

Here are some solutions in my humble order of preference:
1/ ask the user to always fill the size field
2/ ask the user to never fill the size filed (you can override
__setattr__ to make sure...)
3/ override __setattr__ to set __size instead of size

JM

 

[Robert Kern]

something in dir() consumes O(n) operations for lookup, while hasattr
or getattr() require O(log(n)). It matters for me, because it's inside
deeply nested mathematical computations FuncDesigner is made for.

I'm not sure where you are getting log(n) from, but regardless, if you have so
many attributes that the O() matters more than the constant, you have more
problems than this.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Robert Kern]

So can anyone explain me how to look into object's dict? As I have
wrote, "something in dir(...)" requires O(numOfFields) while I would
like to use o(log(n))

('size' in obj.__dict__) is O(1) with a pretty low constant.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Bruno Desthuilliers]

dmitrey a écrit :
(snip)

This doesn't stack with the following issue: sometimes user can write
in code "myObject.size = (some integer value)" and then it will be
involved in future calculations as ordinary fixed value; if user
doesn't supply it, but myObject.size is involved in calculations, then
the oofun is created to behave like similar numpy.array attribute.

IOW, you want a default value for the size if it has not been specified
by the user, so you can safely use this attribute in computations. The
more straightforward solution is to define this attribute (with the
default value) in the initialiser, ie:


class MyClass(object):
def __init__(self, x, y):
self.x = x
self.y = y
self.size = Whatever()


If you don't want to create as many Whatever instances as MyClass
instances, you can create a single Whatever instance before defining
your class:

DEFAULT_WHATEVER = Whathever()

class MyClass(object):
def __init__(self, x, y):
self.x = x
self.y = y
self.size = DEFAULT_WHATEVER


HTH

 

[Bruno Desthuilliers]

Duncan Booth a écrit :

Or you could create the default as a class attribute

from the OP:
"""
I have a class (FuncDesigner oofun) that has no attribute "size", but
it is overloaded in __getattr__, so if someone invokes
"myObject.size", it is generated (as another oofun) and connected to
myObject as attribute.
"""

so this solution won't obviously work in this case !-)

Also and FWIW, I wouldn't advocate this solution if the "default" class
attribute is of a mutable type.

 

[Ethan Furman]

Duncan Booth a écrit :

from the OP:
"""
I have a class (FuncDesigner oofun) that has no attribute "size", but
it is overloaded in __getattr__, so if someone invokes
"myObject.size", it is generated (as another oofun) and connected to
myObject as attribute.
"""

so this solution won't obviously work in this case !-)

Also and FWIW, I wouldn't advocate this solution if the "default" class
attribute is of a mutable type.

Well, it is Monday, so I may be missing something obvious, but what is
the effective difference between these two solutions?

~Ethan~

 

[Bruno Desthuilliers]

Ethan Furman a écrit :

(snip)


Well, it is Monday, so I may be missing something obvious, but what is
the effective difference between these two solutions?

Now it's Tuesday !-)


Ok, let's see:

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information..... whatever = Foo()
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>

 

[Bruno Desthuilliers]

Bruno Desthuilliers a écrit :

Ethan Furman a écrit :

If you meant "what is the difference between creating the "whatever"
attribute with a default value in the initializer and creating it on
demand in the __getattr__ hook", the main difference is that in the
first case, the instance is garanteed to have this attribute, so you get
rid of "hasattr" checks (and the unwanted side effects) or, worse,
direct check of the instance __dict__. Except for a couple of corner
case, client code shouldn't have to worry about this kind of things -
this breaks encapsulation.

 

[Ethan Furman]

Bruno Desthuilliers a écrit :

If you meant "what is the difference between creating the "whatever"
attribute with a default value in the initializer and creating it on
demand in the __getattr__ hook", the main difference is that in the
first case, the instance is garanteed to have this attribute, so you get
rid of "hasattr" checks (and the unwanted side effects) or, worse,
direct check of the instance __dict__. Except for a couple of corner
case, client code shouldn't have to worry about this kind of things -
this breaks encapsulation.

Yay Tuesday!

What I meant was what is the difference between:

[Bruno Desthuilliers]

> DEFAULT_WHATEVER = Whathever()
> class MyClass(object):
> def __init__(self, x, y):
> self.size = DEFAULT_WHATEVER

and

[Duncan Booth]

> class MyClass(object):
> size = Whatever()
> def __init__(self, x, y):
> ...

in both cases the object ends up with a size attribute and no further
need of __getattr__. Of course, the warning about having a mutable
object as a class attribute stands.

To phrase it another way, why does your solution (Bruno) work, but
Duncan's "obviously won't"?

~Ethan~

 

[Bruno Desthuilliers]

Ethan Furman a écrit :

Bruno Desthuilliers a écrit :

If you meant "what is the difference between creating the "whatever"
attribute with a default value in the initializer and creating it on
demand in the __getattr__ hook", the main difference is that in the
first case, the instance is garanteed to have this attribute, so you
get rid of "hasattr" checks (and the unwanted side effects) or, worse,
direct check of the instance __dict__. Except for a couple of corner
case, client code shouldn't have to worry about this kind of things -
this breaks encapsulation.

Yay Tuesday!

What I meant was what is the difference between:

[Bruno Desthuilliers]

DEFAULT_WHATEVER = Whathever()
class MyClass(object):
def __init__(self, x, y):
self.size = DEFAULT_WHATEVER

and

[Duncan Booth]

class MyClass(object):
size = Whatever()
def __init__(self, x, y):
...

in both cases the object ends up with a size attribute and no further
need of __getattr__. Of course, the warning about having a mutable
object as a class attribute stands.
Indeed.

To phrase it another way, why does your solution (Bruno) work, but
Duncan's "obviously won't"?

As it is written (and assuming the name "Whatever" is bound to a
callable !-)), Duncan's code would work, even if it might not be the
safest solution in the case of a mutable type.

The problem here is that the OP stated that the "size" attribute was to
be of the same type as the host class, so the code would look something
like:

class MyClass(object):
size = MyClass()

which will raise a NameError, since MyClass is not yet defined when
"size=MyClass()" is executed.

 

[Ethan Furman]

Ethan Furman a écrit :

Bruno Desthuilliers a écrit :
Ethan Furman a écrit :
Bruno Desthuilliers wrote:
Duncan Booth a écrit :
(snip)

Or you could create the default as a class attribute

from the OP:
"""
I have a class (FuncDesigner oofun) that has no attribute "size", but
it is overloaded in __getattr__, so if someone invokes
"myObject.size", it is generated (as another oofun) and connected to
myObject as attribute.
"""

so this solution won't obviously work in this case !-)

Also and FWIW, I wouldn't advocate this solution if the "default"
class attribute is of a mutable type.

Well, it is Monday, so I may be missing something obvious, but what
is the effective difference between these two solutions?


If you meant "what is the difference between creating the "whatever"
attribute with a default value in the initializer and creating it on
demand in the __getattr__ hook", the main difference is that in the
first case, the instance is garanteed to have this attribute, so you
get rid of "hasattr" checks (and the unwanted side effects) or,
worse, direct check of the instance __dict__. Except for a couple of
corner case, client code shouldn't have to worry about this kind of
things - this breaks encapsulation.

Yay Tuesday!

What I meant was what is the difference between:

[Bruno Desthuilliers]

DEFAULT_WHATEVER = Whathever()
class MyClass(object):
def __init__(self, x, y):
self.size = DEFAULT_WHATEVER

and

[Duncan Booth]

class MyClass(object):
size = Whatever()
def __init__(self, x, y):
...

in both cases the object ends up with a size attribute and no further
need of __getattr__. Of course, the warning about having a mutable
object as a class attribute stands.
Indeed.

To phrase it another way, why does your solution (Bruno) work, but
Duncan's "obviously won't"?

As it is written (and assuming the name "Whatever" is bound to a
callable !-)), Duncan's code would work, even if it might not be the
safest solution in the case of a mutable type.

The problem here is that the OP stated that the "size" attribute was to
be of the same type as the host class, so the code would look something
like:

class MyClass(object):
size = MyClass()

which will raise a NameError, since MyClass is not yet defined when
"size=MyClass()" is executed.

Ah ha! I *knew* I was missing something. Thank you for the
explanation. So the correct methods, then, would be to either include
the size attribute in the __init__, or add it to the class /after/ the
class was defined.

I feel much better now... slightly silly, but better.

~Ethan~