Computing class variable on demand?

[fortepianissimo]

We all know that using __getattr__() we can compute an instance
variable on demand, for example:

class Foo:
def __getattr__ (self, name):
if name == 'bar':
self.bar = 'apple'
return self.bar
else:
raise AttributeError()

Then we can

f = Foo()
s1 = f.bar
s2 = f.bar # this uses the "cached" result

My question is can we do the same with class variables? I can't find a
class-equivalent __getattr__ for this purpose. The key is to allow the
use of the following statement

Foo.bar # bar is a class variable of Foo computed on demand
without unnecessary fudge. Thanks.

 

[Steven Bethard]

We all know that using __getattr__() we can compute an instance
variable on demand, for example:

class Foo:
def __getattr__ (self, name):
if name == 'bar':
self.bar = 'apple'
return self.bar
else:
raise AttributeError()

Then we can

f = Foo()
s1 = f.bar
s2 = f.bar # this uses the "cached" result

My question is can we do the same with class variables?

You can do this using a metaclass, e.g.:

py> class FooType(type):
.... def __getattr__(self, name):
.... if name == 'bar':
.... self.bar = 'apple'
.... return self.bar
.... else:
.... raise AttributeError('no attribute named %r' % name)
....
py> class Foo(object):
.... __metaclass__ = FooType
....
py> Foo.bar
'apple'

However, you probably don't want to. What's your use case?

Steve

 

[fortepianissimo]

This seems to be what I need. My use case is to do lengthy
intialization as late as possible. In this case this is to initialize
class variables. Does this make sense?

Thank you.

 

[Steven Bethard]

This seems to be what I need. My use case is to do lengthy
intialization as late as possible. In this case this is to initialize
class variables. Does this make sense?

Yup. If they definitely have to be class variables, then yeah, you
probably need to use a metaclass. If you're looking for late
initialization, Scott David Daniels has a nice descriptor recipe for this:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/363602

Probably this is a better call than using __getattr__. With it, your
code might look like[1]:

py> class LazyAttribute(object):
.... def __init__(self, calculate_function):
.... self._calculate = calculate_function
.... def __get__(self, obj, _=None):
.... if obj is None:
.... return self
.... value = self._calculate(obj)
.... setattr(obj, self._calculate.func_name, value)
.... return value
....
py> class Foo(object):
.... class __metaclass__(type):
.... @LazyAttribute
.... def bar(self):
.... print 'slow initialization'
.... return 'apple'
....
py> Foo.bar
slow initialization
'apple'
py> Foo.bar
'apple'

That way, you define all the attributes that need to be lazily
initialized in the __metaclass__ class. If all your classes will have
the same lazily initialized attributes, you could also write this like:

py> class BarType(type):
.... @LazyAttribute
.... def bar(self):
.... print 'slow initialization'
.... return 'apple'
....
py> class Foo(object):
.... __metaclass__ = BarType
....
py> class Baz(object):
.... __metaclass__ = BarType
....
py> Foo.bar
slow initialization
'apple'
py> Foo.bar
'apple'
py> Baz.bar
slow initialization
'apple'
py> Baz.bar
'apple'

But I suspect the former is what you want...

Steve

[1] If you're using Python < 2.4, you'll have to make the decorator call
manually, e.g.

def bar(self):
...
bar = LazyAttribute(bar)

 

[fortepianissimo]

Thank you so much about this useful tip! I learned the new decorator
feature of 2.4 simply because of your post.

Unfortunately I don't have luxury right now to run Python 2.4 (for what
I'm doing anyways). You mentioned the way to do decorator in 2.3. Still
I have a question here. Here is Scott David Daniels's code for lazy
initialization:

class Lazy (object):
def __init__ (self, calculate_function):
self._calculate = calculate_function

def __get__ (self, obj, _=None):
if obj is None:
return self
value = self._calculate(obj)
setattr(obj, self._calculate.func_name, value)
return value

The problem I run into using this for *instance* variables is: the
setattr() call won't work with a class with __slots__ defined - it
simply produces error that the attribute we want to modify is
read-only. Is there a workaround of this problem?

By the way this implementation is indeed better than the __getattr__
since in the latter I need to do a series of if... elif... else to
decide what to do with different attribute names. In the solution
above, I imagine the attribute names are hashed so it should have
better performance (the first time anyways).

 

[Steven Bethard]

Thank you so much about this useful tip! I learned the new decorator
feature of 2.4 simply because of your post.

Unfortunately I don't have luxury right now to run Python 2.4 (for what
I'm doing anyways). You mentioned the way to do decorator in 2.3. Still
I have a question here. Here is Scott David Daniels's code for lazy
initialization:

class Lazy (object):
def __init__ (self, calculate_function):
self._calculate = calculate_function

def __get__ (self, obj, _=None):
if obj is None:
return self
value = self._calculate(obj)
setattr(obj, self._calculate.func_name, value)
return value

The problem I run into using this for *instance* variables is: the
setattr() call won't work with a class with __slots__ defined - it
simply produces error that the attribute we want to modify is
read-only. Is there a workaround of this problem?

Sounds like you're declaring the class variables in your __slots__ too.
Is this true? I don't think that's necessary -- __slots__ is only for
used for instances. So, for example, this code should work okay:

py> class Foo(object):
.... __slots__ = ['baz']
.... class __metaclass__(type):
.... def bar(self):
.... print 'slow initialization'
.... return 'apple'
.... bar = LazyAttribute(bar)
.... def __init__(self, baz):
.... self.baz = baz
....
py> f = Foo(1)
py> f.baz
1
py> f.bar
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
AttributeError: 'Foo' object has no attribute 'bar'
py> f.__class__.bar
slow initialization
'apple'
py> Foo.bar
'apple'

Note that if you want to reference the class variable, you have to
specficially go through the class, instead of counting on the instance
lookup as classes without __slots__ can. But as long as you don't
declare 'bar' as a slot, you should still be able to access Foo.bar.

Note that you probably don't want to have 'bar' as both a class variable
and an instance variable -- then the instance variable will just hide
the class variable...

HTH,

Steve

 

[fortepianissimo]

Thanks Steve - actually my question was simpler than that. I just
wanted to use Daniels' recipe of lazy initialization on objects with
__slots__:

class Lazy(object):
def __init__(self, calculate_function):
self._calculate = calculate_function

def __get__(self, obj, _=None):
if obj is None:
return self
value = self._calculate(obj)
setattr(obj, self._calculate.func_name, value)
return value


class SomeClass(object):

__slots__ = 'someprop'

@Lazy
def someprop(self):
print 'Actually calculating value'
return 13


o = SomeClass()
print o.someprop
print o.someprop



Running the code above will produce:

Actually calculating value
Traceback (most recent call last):
File "Lazy.py", line 26, in ?
print o.someprop
File "Lazy.py", line 11, in __get__
setattr(obj, self._calculate.func_name, value)
AttributeError: 'SomeClass' object attribute 'someprop' is read-only


Removing the __slots__ statement, everything would run normally.

Is there any workaround?

 

[Steven Bethard]

Thanks Steve - actually my question was simpler than that. I just
wanted to use Daniels' recipe of lazy initialization on objects with
__slots__:

class Lazy(object):
def __init__(self, calculate_function):
self._calculate = calculate_function

def __get__(self, obj, _=None):
if obj is None:
return self
value = self._calculate(obj)
setattr(obj, self._calculate.func_name, value)
return value


class SomeClass(object):

__slots__ = 'someprop'

@Lazy
def someprop(self):
print 'Actually calculating value'
return 13


o = SomeClass()
print o.someprop
print o.someprop



Running the code above will produce:

Actually calculating value
Traceback (most recent call last):
File "Lazy.py", line 26, in ?
print o.someprop
File "Lazy.py", line 11, in __get__
setattr(obj, self._calculate.func_name, value)
AttributeError: 'SomeClass' object attribute 'someprop' is read-only


Removing the __slots__ statement, everything would run normally.

Is there any workaround?

Hmm... Well, there's a thread on a similar topic:

http://mail.python.org/pipermail/python-dev/2003-May/035575.html

The summary is basically that '__slots__' creates descriptors for all
the names in the __slots__ iterable. If you define a function (or a
Lazy object) at the class level with the same name as a slot, it
replaces the descriptor for that slot name. So then when setattr gets
called, it can't find the descriptor anymore, so it can't write the value...

One workaround would be to have Lazy store the values instead of setting
the attribute on the instance -- something like:

py> class Lazy(object):
.... def __init__(self, calculate_function):
.... self._calculate = calculate_function
.... self._values = {}
.... def __get__(self, obj, _=None):
.... if obj is None:
.... return self
.... obj_id = id(obj)
.... if obj_id not in self._values:
.... self._values[obj_id] = self._calculate(obj)
.... return self._values[obj_id]
....
py>
py> class SomeClass(object):
.... __slots__ = []
.... @Lazy
.... def someprop(self):
.... print 'Actually calculating value'
.... return 13
....
py> o = SomeClass()
py> o.someprop
Actually calculating value
13
py> o.someprop
13

Basically Lazy just stores a mapping from object ids to calculated
values. Not as pretty as the original solution, but if you have to have
__slots__ defined[1], I guess this might work.

Note that I don't need to declare any slots since the someprop
descriptor is a class-level attribute, not an instance-level one...

Steve

[1] Are you sure you have to use __slots__? Unless you're creating a
very large number of these objects, you probably won't notice the
difference in memory consumption...