How to memoize/cache property access?

[thebjorn]

I seem to be writing the following boilerplate/pattern quite
frequently to avoid hitting the database until absolutely necessary,
and to only do it at most once:

class Foo(object):
@property
def expensive(self):
if not hasattr(self, '_expensiv'):
self._expensive = <insert expensive db call here>
return self._expensive

it's a bit verbose, and it violates the DRY (Don't Repeat Yourself)
principle -- and it has on at least one occasion resulted in really
slow code that produces the correct results (did you spot the typo
above?).

It would have been nice to be able to write

class Foo(object):
@property
def expensive(self):
self.expensive = <insert expensive db call here>
return self.expensive

but apparently I "can't set [that] attribute" :-(

I'm contemplating using a decorator to hide the first pattern:

def memprop(fn):
def _fn(self): # properties only take self
memname = '_' + fn.__name__
if not hasattr(self, memname):
setattr(self, memname, fn(self))
return getattr(self, memname)
return property(fget=_fn, doc=fn.__doc__)

which means I can very simply write

class Foo(object):
@memprop
def expensive(self):
return <insert expensive db call here>

I'm a bit hesitant to start planting home-grown memprop-s all over the
code-base though, so I'm wondering... does this seem like a reasonable
thing to do? Am I re-inventing the wheel? Is there a better way to
approach this problem?

-- bjorn

 

[Duncan Booth]

It would have been nice to be able to write

class Foo(object):
@property
def expensive(self):
self.expensive = <insert expensive db call here>
return self.expensive

but apparently I "can't set [that] attribute" :-(

You can set and access it directly in __dict__. How about something along
these lines:

def cachedproperty(f):
name = f.__name__
def getter(self):
try:
return self.__dict__[name]
except KeyError:
res = self.__dict__[name] = f(self)
return res
return property(getter)

class Foo(object):
@cachedproperty
def expensive(self):
print "expensive called"
return 42
expensive called
4242

It is still calling the getter every time though, so not as fast as a plain
attribute lookup.

 

[Michele Simionato]

I seem to be writing the following boilerplate/pattern quite
frequently to avoid hitting the database until absolutely necessary ...

I use the following module:

$ cat cache.py
class cached(property):
'Convert a method into a cached attribute'
def __init__(self, method):
private = '_' + method.__name__
def fget(s):
try:
return getattr(s, private)
except AttributeError:
value = method(s)
setattr(s, private, value)
return value
def fdel(s):
del s.__dict__[private]
super(cached, self).__init__(fget, fdel=fdel)
@staticmethod
def reset(self):
cls = self.__class__
for name in dir(cls):
attr = getattr(cls, name)
if isinstance(attr, cached):
delattr(self, name)

if __name__ == '__main__': # a simple test
import itertools
counter = itertools.count()
class Test(object):
@cached
def foo(self):
return counter.next()
reset = cached.reset

p = Test()
print p.foo
print p.foo
p.reset()
print p.foo
print p.foo
p.reset()
print p.foo

Michele Simionato

 

[thebjorn]

I use the following module:

[...]

I love it! much better name too ;-)

I changed your testcase to include a second cached property (the
naming is in honor of my late professor in Optimization of Functional
Languages class: "...any implementation that calls bomb_moscow is per
definition wrong, even if the program produces the correct result..."
-- it was a while ago ;-)

if __name__ == '__main__': # a simple test
import itertools
counter = itertools.count()
class Test(object):
@cached
def foo(self):
return counter.next()

@cached
def bomb_moscow(self):
print 'fire missiles'
return counter.next()

reset = cached.reset

it didn't start WWIII, but I had to protect attribute deletion to get
it to run:

def fdel(s):
if private in s.__dict__:
del s.__dict__[private]

I'm a bit ambivalent about the reset functionality. While it's a
wonderful demonstration of a staticmethod, the very few times I've
felt the need to "freshen-up" the object, I've always felt it was best
to create it again from scratch. Do you have many uses of it in your
code?

-- bjorn

 

[Michele Simionato]

On Dec 20, 6:40 pm, thebjorn

I'm a bit ambivalent about the reset functionality. While it's a
wonderful demonstration of a staticmethod, the very few times I've
felt the need to "freshen-up" the object, I've always felt it was best
to create it again from scratch. Do you have many uses of it in your
code?

My use case is for web applications where the configuration
parameters are stored in a database. If you change them,
you can re-read them by resetting the configuration object
(you may have a reset button in the administrator Web user
interface) without restarting the application.

Michele Simionato