Easier way to save result of a function?

[James Mitchelhill]

Sorry for the clunky subject line - I have a feeling that not knowing
the proper terms for this is part of my problem.

I'm trying to write a class that analyses some data. I only want it to
do as much work as necessary, so it saves method results to a
dictionary, like so:


class MyClass:
def __init__(self):
self.data = {}

def doSomething(self):
return self.data.setdefault("someresult", self._doSomething())

def _doSomething(self):
# heavy processing here
result = 1
return result

def doMore(self):
return self.data.setdefault("moreresult", self._doMore())

def _doMore(self):
# more heavy processing here
return self.doSomething() + 1


This also seems kind of clunky. Is there a better way to acheive this?
Or is this the usual idiom for this kind of thing? I've looked at
decorators as they seem like they could be used for this, but I've no
idea how exactly, or even if they're the appropriate tool.

Thanks.

 

[Mike Kent]

Sorry for the clunky subject line - I have a feeling that not knowing
the proper terms for this is part of my problem.

I'm trying to write a class that analyses some data. I only want it to
do as much work as necessary, so it saves method results to a
dictionary, like so:

The term for this is 'memoization'. You can find several recipes for
this in the online Python Cookbook; for example, see
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/466320

 

[Alex Martelli]

Sorry for the clunky subject line - I have a feeling that not knowing
the proper terms for this is part of my problem.

I'm trying to write a class that analyses some data. I only want it to
do as much work as necessary, so it saves method results to a
dictionary, like so:


class MyClass:
def __init__(self):
self.data = {}

def doSomething(self):
return self.data.setdefault("someresult", self._doSomething())

def _doSomething(self):
# heavy processing here
result = 1
return result

def doMore(self):
return self.data.setdefault("moreresult", self._doMore())

def _doMore(self):
# more heavy processing here
return self.doSomething() + 1


This also seems kind of clunky. Is there a better way to acheive this?
Or is this the usual idiom for this kind of thing? I've looked at
decorators as they seem like they could be used for this, but I've no
idea how exactly, or even if they're the appropriate tool.

No, this code is calling the heavy-processing functions
*unconditionally* -- Python's semantics is to compute every argument of
a function or method before starting to execute the code of that
function or method (almost all languages behave that way, save a very
few "pure" functional languages!), and that applies to the setdefault
methods of dictionaries too.

What you want to do is called "memoization" (sometimes "caching", but
that's a very generic term used in a bazillion different situations and
you may get overwhelmed by search results if you try searching for
it!-), and you can find plenty of recipes for it in the Python Cookbook
and elsewhere on the net.

For your needs, I would suggest first defining your class "normally"
(writing the methods without memoization) and then looping on all
methods of interest decorating them with automatic memoization; I just
suggested a similar kind of looping for decoration purposes in my very
latest post on another thread;-). You won't use the _syntax_ of
decorators, but, hey, who cares?-)

Assuming (for simplicity but with no real loss of conceptual generality)
that all of your heavy-processing methods have names starting with 'do',
and that each of them takes only 'self' as its argument, you could for
example do something like:

def memoizeMethod(cls, n, m):
def decorated(self):
if n in self._memo: return self._memo[n]
result = self._memo[n] = m(self)
return result
decorated.__name__ = n
setattr(cls, n, decorated)

def memoizeClass(cls):
cls._memo = {}
for n,m in inspect.getmembers(cls, inspect.ismethoddescriptors):
if not n.startswith('do'): continue
memoizeMethod(cls, n, m)

Now just write your MyClass with the doThis doThat methods you want, and
right after the class statement add

memoizeClass(MyClass)


Voila, that's it (untested details, but the general idea is sound;-).


Alex

 

[James Mitchelhill]

James Mitchelhill <[email protected]> wrote:

For your needs, I would suggest first defining your class "normally"
(writing the methods without memoization) and then looping on all
methods of interest decorating them with automatic memoization; I just
suggested a similar kind of looping for decoration purposes in my very
latest post on another thread;-). You won't use the _syntax_ of
decorators, but, hey, who cares?-)

Assuming (for simplicity but with no real loss of conceptual generality)
that all of your heavy-processing methods have names starting with 'do',
and that each of them takes only 'self' as its argument, you could for
example do something like:

Thanks, that's awesome! Definitely not something I'd have ever been able
to work out myself - I think I need to learn more about nested functions
and introspection.

def memoizeMethod(cls, n, m):
def decorated(self):
if n in self._memo: return self._memo[n]
result = self._memo[n] = m(self)
return result
decorated.__name__ = n
setattr(cls, n, decorated)

def memoizeClass(cls):
cls._memo = {}
for n,m in inspect.getmembers(cls, inspect.ismethod):
if not n.startswith('do'): continue
memoizeMethod(cls, n, m)

(I've changed inspect.ismethoddescriptors to inspect.ismethod and
unindented the last line.)

Now just write your MyClass with the doThis doThat methods you want, and
right after the class statement add

memoizeClass(MyClass)


Voila, that's it (untested details, but the general idea is sound;-).

Thanks again.

 

[Ant]

Thanks, that's awesome! Definitely not something I'd have ever been able
to work out myself - I think I need to learn more about nested functions
and introspection.

I've recently found nested functions incredibly useful in many places
in my code, particularly as a way of producing functions that are
pre-set with some initialization data. I've written a bit about it
here: http://antroy.blogspot.com/ (the entry about Partial Functions)

def memoizeMethod(cls, n, m):
def decorated(self):
if n in self._memo: return self._memo[n]
result = self._memo[n] = m(self)
return result
decorated.__name__ = n
setattr(cls, n, decorated)

Couldn't this be more simply written as:

def memoizeMethod(cls, n, m):
def decorated(self):
if not n in self._memo:
self._memo[n] = m(self)
return self._memo[n]
decorated.__name__ = n
setattr(cls, n, decorated)

I've not seen the use of chained = statements before. Presumably it
sets all variables to the value of the last one?

 

[Alex Martelli]

Thanks, that's awesome! Definitely not something I'd have ever been able
to work out myself - I think I need to learn more about nested functions
and introspection.

I've recently found nested functions incredibly useful in many places
in my code, particularly as a way of producing functions that are
pre-set with some initialization data. I've written a bit about it
here: http://antroy.blogspot.com/ (the entry about Partial Functions)

def memoizeMethod(cls, n, m):
def decorated(self):
if n in self._memo: return self._memo[n]
result = self._memo[n] = m(self)
return result
decorated.__name__ = n
setattr(cls, n, decorated)

Couldn't this be more simply written as:

def memoizeMethod(cls, n, m):
def decorated(self):
if not n in self._memo:
self._memo[n] = m(self)
return self._memo[n]
decorated.__name__ = n
setattr(cls, n, decorated)

Yes, at a tiny performance cost (probably hard to measure), though I
would spell the guard "if n not in self._memo:" .

I've not seen the use of chained = statements before. Presumably it
sets all variables to the value of the last one?

It sets all ``variables'' ("left-hand sides", LHS) to the right-hand
side (RHS) value; here, it lets you avoid an extra indexing in order to
obtain the value to return (a tiny cost, to be sure).


Alex