Using arguments in a decorator

[Rotwang]

Hi all, here's a problem I don't know how to solve. I'm using Python 2.7.2.

I'm doing some stuff in Python which means I have cause to call
functions that take a while to return. Since I often want to call such a
function more than once with the same arguments, I've written a
decorator to eliminate repeated calls by storing a dictionary whose
items are arguments and their results:

def memo(func):
def memofunc(*args, **kwargs):
twargs = tuple(kwargs.items())
if (args, twargs) in memofunc.d:
return copy(memofunc.d[(args, twargs)])
memofunc.d[(args, twargs)] = func(*args, **kwargs)
return copy(memofunc.d[(args, twargs)])
memofunc.__name__ = func.__name__
memofunc.d = {}
return memofunc


If a function f is decorated by memo, whenever f is called with
positional arguments args and keyword arguments kwargs, the decorated
function defines twargs as a hashable representation of kwargs, checks
whether the tuple (args, twargs) is in f's dictionary d, and if so
returns the previously calculated value; otherwise it calculates the
value and adds it to the dictionary (copy() is a function that returns
an object that compares equal to its argument, but whose identity is
different - this is useful if the return value is mutable).

As far as I know, the decorated function will always return the same
value as the original function. The problem is that the dictionary key
stored depends on how the function was called, even if two calls should
be equivalent; hence the original function gets called more often than
necessary. For example, there's this:
def f(x, y = None, *a, **k):
return x, y, a, k
{((1, 2), ()): (1, 2, (), {}), ((), (('y', 2), ('x', 1))): (1, 2, (), {})}


What I'd like to be able to do is something like this:

def memo(func):
def memofunc(*args, **kwargs):
#
# define a tuple consisting of values for all named positional
# arguments occurring in the definition of func, including
# default arguments if values are not given by the call, call
# it named
#
# define another tuple consisting of any positional arguments
# that do not correspond to named arguments in the definition
# of func, call it anon
#
# define a third tuple consisting of pairs of names and values
# for those items in kwargs whose keys are not named in the
# definition of func, call it key
#
if (named, anon, key) in memofunc.d:
return copy(memofunc.d[(named, anon, key)])
memofunc.d[(named, anon, key)] = func(*args, **kwargs)
return copy(memofunc.d[(named, anon, key)])
memofunc.__name__ = func.__name__
memofunc.d = {}
return memofunc


But I don't know how. I know that I can see the default arguments of the
original function using func.__defaults__, but without knowing the
number and names of func's positional arguments (which I don't know how
to find out) this doesn't help me. Any suggestions?

 

[Jon Clements]

Hi all, here's a problem I don't know how to solve. I'm using Python 2.7.2.

I'm doing some stuff in Python which means I have cause to call
functions that take a while to return. Since I often want to call such a
function more than once with the same arguments, I've written a
decorator to eliminate repeated calls by storing a dictionary whose
items are arguments and their results:

def memo(func):
def memofunc(*args, **kwargs):
twargs = tuple(kwargs.items())
if (args, twargs) in memofunc.d:
return copy(memofunc.d[(args, twargs)])
memofunc.d[(args, twargs)] = func(*args, **kwargs)
return copy(memofunc.d[(args, twargs)])
memofunc.__name__ = func.__name__
memofunc.d = {}
return memofunc


If a function f is decorated by memo, whenever f is called with
positional arguments args and keyword arguments kwargs, the decorated
function defines twargs as a hashable representation of kwargs, checks
whether the tuple (args, twargs) is in f's dictionary d, and if so
returns the previously calculated value; otherwise it calculates the
value and adds it to the dictionary (copy() is a function that returns
an object that compares equal to its argument, but whose identity is
different - this is useful if the return value is mutable).

As far as I know, the decorated function will always return the same
value as the original function. The problem is that the dictionary key
stored depends on how the function was called, even if two calls should
be equivalent; hence the original function gets called more often than
necessary. For example, there's this:
def f(x, y = None, *a, **k):
return x, y, a, k
{((1, 2), ()): (1, 2, (), {}), ((), (('y', 2), ('x', 1))): (1, 2, (), {})}


What I'd like to be able to do is something like this:

def memo(func):
def memofunc(*args, **kwargs):
#
# define a tuple consisting of values for all named positional
# arguments occurring in the definition of func, including
# default arguments if values are not given by the call, call
# it named
#
# define another tuple consisting of any positional arguments
# that do not correspond to named arguments in the definition
# of func, call it anon
#
# define a third tuple consisting of pairs of names and values
# for those items in kwargs whose keys are not named in the
# definition of func, call it key
#
if (named, anon, key) in memofunc.d:
return copy(memofunc.d[(named, anon, key)])
memofunc.d[(named, anon, key)] = func(*args, **kwargs)
return copy(memofunc.d[(named, anon, key)])
memofunc.__name__ = func.__name__
memofunc.d = {}
return memofunc


But I don't know how. I know that I can see the default arguments of the
original function using func.__defaults__, but without knowing the
number and names of func's positional arguments (which I don't know how
to find out) this doesn't help me. Any suggestions?

Possibly take a look at functools.lru_cache (which is Python 3.2+), and use the code from that (at it's part of the stdlib, someone must have done design and testing on it!). http://hg.python.org/cpython/file/default/Lib/functools.py

Jon

 

[Rotwang]

Hi all, here's a problem I don't know how to solve. I'm using Python 2.7.2.

I'm doing some stuff in Python which means I have cause to call
functions that take a while to return. Since I often want to call such a
function more than once with the same arguments, I've written a
decorator to eliminate repeated calls by storing a dictionary whose
items are arguments and their results:

def memo(func):
def memofunc(*args, **kwargs):
twargs = tuple(kwargs.items())
if (args, twargs) in memofunc.d:
return copy(memofunc.d[(args, twargs)])
memofunc.d[(args, twargs)] = func(*args, **kwargs)
return copy(memofunc.d[(args, twargs)])
memofunc.__name__ = func.__name__
memofunc.d = {}
return memofunc


If a function f is decorated by memo, whenever f is called with
positional arguments args and keyword arguments kwargs, the decorated
function defines twargs as a hashable representation of kwargs, checks
whether the tuple (args, twargs) is in f's dictionary d, and if so
returns the previously calculated value; otherwise it calculates the
value and adds it to the dictionary (copy() is a function that returns
an object that compares equal to its argument, but whose identity is
different - this is useful if the return value is mutable).

As far as I know, the decorated function will always return the same
value as the original function. The problem is that the dictionary key
stored depends on how the function was called, even if two calls should
be equivalent; hence the original function gets called more often than
necessary. For example, there's this:

@memo

def f(x, y = None, *a, **k):
return x, y, a, k

f(1, 2) (1, 2, (), {})
f.d {((1, 2), ()): (1, 2, (), {})}
f(y = 2, x = 1) (1, 2, (), {})
f.d

{((1, 2), ()): (1, 2, (), {}), ((), (('y', 2), ('x', 1))): (1, 2, (), {})}


What I'd like to be able to do is something like this:

def memo(func):
def memofunc(*args, **kwargs):
#
# define a tuple consisting of values for all named positional
# arguments occurring in the definition of func, including
# default arguments if values are not given by the call, call
# it named
#
# define another tuple consisting of any positional arguments
# that do not correspond to named arguments in the definition
# of func, call it anon
#
# define a third tuple consisting of pairs of names and values
# for those items in kwargs whose keys are not named in the
# definition of func, call it key
#
if (named, anon, key) in memofunc.d:
return copy(memofunc.d[(named, anon, key)])
memofunc.d[(named, anon, key)] = func(*args, **kwargs)
return copy(memofunc.d[(named, anon, key)])
memofunc.__name__ = func.__name__
memofunc.d = {}
return memofunc


But I don't know how. I know that I can see the default arguments of the
original function using func.__defaults__, but without knowing the
number and names of func's positional arguments (which I don't know how
to find out) this doesn't help me. Any suggestions?

Possibly take a look at functools.lru_cache (which is Python 3.2+), and use the code from that (at it's part of the stdlib, someone must have done design and testing on it!). http://hg.python.org/cpython/file/default/Lib/functools.py

Thanks for your reply, but this decorator doesn't seem to attempt to get
around the problem I mentioned. This is pasted from IDLE:
def f(x, y = None, *a, **k):
return x, y, a, k
CacheInfo(hits=0, misses=2, maxsize=None, currsize=2)

 

[Steven D'Aprano]

Possibly take a look at functools.lru_cache (which is Python 3.2+), and
use the code from that (at it's part of the stdlib, someone must have
done design and testing on it!).

With respect Jon, did you read the Original Poster's question closely?
Using a LRU cache doesn't even come close to fixing his problem, which
occurs *before* you do the lookup in the cache.

Rotwang's problem is that if you have a function with default arguments:

def func(spam=42):
return result_of_time_consuming_calculation()

then these three function calls are identical and should (but don't)
share a single cache entry:

func()
func(42)
func(spam=42)

The OP would like all three to share a single cache entry without needing
two redundant calculations, which take a long time.

The problem is that the three calls give three different patterns of args
and kwargs:

(), {}
(42,) {}
(), {'spam': 42}

hence three different cache entries, two of which are unnecessary.

 

[Steven D'Aprano]

def memo(func):
def memofunc(*args, **kwargs):
twargs = tuple(kwargs.items())
if (args, twargs) in memofunc.d:
return copy(memofunc.d[(args, twargs)])
memofunc.d[(args, twargs)] = func(*args, **kwargs) return
copy(memofunc.d[(args, twargs)])
memofunc.__name__ = func.__name__
memofunc.d = {}
return memofunc

Note that this decorator is underpowered and arguable buggy: it
suppresses the decorated function's doc string. You should use
functools.wraps to copy the name, doc string and anything else necessary.

Here is how I would write the above.


import functools

def memoise(func):
"""Decorator to memoise a function."""
cache = {}
@functools.wraps(func)
def inner(*args, **kwargs):
# Make sure keyword args are always looked up in a consistent
# order by sorting. One minor subtlety: since kwargs cannot
# have duplicate keys, sorted() will never try to break ties
# by comparing values. So this will work even for unsortable
# values like complex numbers.
kw = tuple(sorted(kwargs.items()))
try:
# Cache hit?
result = cache[(args, kw)]
except KeyError:
# Cache miss.
result = func(*args, **kwargs)
cache[(args, kw)] = result
except TypeError:
# Cache failure; at least one argument is uncacheable.
result = func(*args, **kwargs)
# Is the cache too big?
if len(cache) > 1000:
# Dump an arbitrary item. A LRU (least recently used) cache
# would be better, but this will do.
cache.popitem()
return result
# Expose the cache to the outside world.
inner._cache = cache
return inner

[...]

But I don't know how. I know that I can see the default arguments of the
original function using func.__defaults__, but without knowing the
number and names of func's positional arguments (which I don't know how
to find out) this doesn't help me. Any suggestions?

http://docs.python.org/release/3.1.5/library/inspect.html?#inspect.getfullargspec

.... x = 42
.... return (a,b,c,d,args,kwargs)
....FullArgSpec(args=['a', 'b'], varargs='args', varkw='kwargs',
defaults=(1,), kwonlyargs=['c', 'd'], kwonlydefaults={'d': 2},
annotations={})

 

[Jon Clements]

With respect Jon, did you read the Original Poster's question closely?
Using a LRU cache doesn't even come close to fixing his problem, which
occurs *before* you do the lookup in the cache.

I did indeed Steven - what I was suggesting was that functools.lru_cache would be a good starting point. Although I will completely admit that I didn't read the code for functools.lru_cache thoroughly enough to realise it wouldn't be suitable for the OP (ie, it sounded right, looked okay at a glance, and I figured it wasn't a totally unreasonable assumption of a suggestion- so guess I fell into the old 'assume' trap! [not the first time, and won't be the last for sure!])

Rotwang's problem is that if you have a function with default arguments:

def func(spam=42):
return result_of_time_consuming_calculation()

then these three function calls are identical and should (but don't)
share a single cache entry:

func()
func(42)
func(spam=42)

The OP would like all three to share a single cache entry without needing
two redundant calculations, which take a long time.

The problem is that the three calls give three different patterns of args
and kwargs:

(), {}
(42,) {}
(), {'spam': 42}

hence three different cache entries, two of which are unnecessary.

I'm wondering if it wouldn't be unreasonable for lru_cache to handle this.

Cheers, Jon.

 

[Rotwang]

[...]

Here is how I would write the above.


import functools

def memoise(func):
"""Decorator to memoise a function."""
cache = {}
@functools.wraps(func)
def inner(*args, **kwargs):
# Make sure keyword args are always looked up in a consistent
# order by sorting. One minor subtlety: since kwargs cannot
# have duplicate keys, sorted() will never try to break ties
# by comparing values. So this will work even for unsortable
# values like complex numbers.
kw = tuple(sorted(kwargs.items()))
try:
# Cache hit?
result = cache[(args, kw)]
except KeyError:
# Cache miss.
result = func(*args, **kwargs)
cache[(args, kw)] = result
except TypeError:
# Cache failure; at least one argument is uncacheable.
result = func(*args, **kwargs)
# Is the cache too big?
if len(cache)> 1000:
# Dump an arbitrary item. A LRU (least recently used) cache
# would be better, but this will do.
cache.popitem()
return result
# Expose the cache to the outside world.
inner._cache = cache
return inner

Thanks. Incidentally, would there be any way to look at the value of
cache if it weren't for the statement inner._cache = cache? For that
matter, I don't understand why 'cache' isn't in inner.__globals__.

[...]

But I don't know how. I know that I can see the default arguments of the
original function using func.__defaults__, but without knowing the
number and names of func's positional arguments (which I don't know how
to find out) this doesn't help me. Any suggestions?

http://docs.python.org/release/3.1.5/library/inspect.html?#inspect.getfullargspec

... x = 42
... return (a,b,c,d,args,kwargs)
...FullArgSpec(args=['a', 'b'], varargs='args', varkw='kwargs',
defaults=(1,), kwonlyargs=['c', 'd'], kwonlydefaults={'d': 2},
annotations={})

Cool; that's not available for Python 2 but it looks like
inspect.getargspec() is and does pretty much the same thing. Also
inspect.getcallargs() does exactly what I need for my decorator. Thanks
very much for your help.

 

[Rotwang]

[...]

But I don't know how. I know that I can see the default arguments of the
original function using func.__defaults__, but without knowing the
number and names of func's positional arguments (which I don't know how
to find out)

Although the existence of inspect.getcallargs() has rendered this latter
question moot for my purposes, in case anyone cares I've just noticed
that the above information can be found by using
func.__code__.co_argcount and func.__code__.co_varnames.