lambda closure question

[Ted Lilley]

What I want to do is pre-load functions with arguments by iterating
through a list like so:

class myclass: .... pass
def func(self, arg): .... print arg
mylist = ["my", "sample", "list"]
for item in mylist:

.... setattr(myclass, item, lamdba self: func(self, item))

This attaches a list of functions to the class, making them bound
methods when called from a class instance. The functions are all
similar except that they know their name through the item argument.
The lambda takes the underlying function and preloads it with its name,
and since lambdas support closures, it remembers the item variable that
was used to create it when it is actually called like so:
list

That's great, since I can write a generic function that just needs to
know its name to make a slight modification to its behavior, then
attach it a bunch of times to a class under different names.

Unfortunately, it doesn't work. It seems the closure keeps track of
the variable fed to it dynamically - if the variable changes after the
lambda is created, the lambda still references the _variable_ not the
original _value_ and so gets the new value like so:
list

At least, that's the explanation I'm deducing from this behavior.
Assuming that's the way Guido intended it to be (about as dynamic as it
can get), I'm at a loss to do what _I_ want it to do. In fact, I don't
think there's any way to generate the lambdas properly without coding
in the name as a literal string, since any attempt to use a variable
reference will always get modified as the loop iterates.

Am I missing something? Help.

 

[Carl Banks]

What I want to do is pre-load functions with arguments by iterating
through a list like so:

class myclass: ... pass
def func(self, arg): ... print arg
mylist = ["my", "sample", "list"]
for item in mylist:

... setattr(myclass, item, lamdba self: func(self, item))
[snip]

Unfortunately, it doesn't work. It seems the closure keeps track of
the variable fed to it dynamically - if the variable changes after the
lambda is created, the lambda still references the _variable_ not the
original _value_ and so gets the new value like so:
list

At least, that's the explanation I'm deducing from this behavior.

And that's the correct explanation, chief.

Assuming that's the way Guido intended it to be (about as dynamic as it
can get), I'm at a loss to do what _I_ want it to do. In fact, I don't
think there's any way to generate the lambdas properly without coding
in the name as a literal string, since any attempt to use a variable
reference will always get modified as the loop iterates.

It is intended that way. As an example of why that is: consider a
nested function called "printvars()" that you could insert in various
places within a function to print out the value of some local variable.
If you did that, you wouldn't want printvars to print the values at
the time it was bound, would you?

Anyways, this can be worked around, but I'd suppose it's not as
convenient as you'd like. (You could define crystalize outside the
loop, but I have it in the loop to keep it near the place where you
need it.)

.. mylist = ["my","sample","list"]
.. for item in mylist:
.. def crystalize(x):
.. return lambda self: func(self,x)
.. setattr(myclass,item,crystalize(item))

If you're hellbent on only using lambda, that can be done too:

.. setattr(myclass,item,(lambda xlambda self:
func(self,x)))(item))

You may not be aware of it, but what you're trying to do is called
"currying"; you might want to search the Python Cookbook for recipes on
it.

 

[Carl Banks]

Ted Lilley wrote:

Unfortunately, it doesn't work. It seems the closure keeps track of
the variable fed to it dynamically - if the variable changes after [...]

At least, that's the explanation I'm deducing from this behavior.

And that's the correct explanation, chief.

It is intended that way. As an example of why that is: consider a
nested function called "printvars()" that you could insert in various
places within a function to print out the value of some local variable.
If you did that, you wouldn't want printvars to print the values at
the time it was bound, would you?

Allow me to disagree (and start a new "confused with closures" thread

We know that python does not have references to variables. To some
newcomers this may seem annoying but eventually they understand the
pythonic way and they do without them. But in the case of closures
python supports references!

That's right. It's called Practicality beats purity.

The question is what's more important: to be able to use a nested
function to reference the outer scope, or to be able to use a nested
function as a closure inside the function it was defined in? (Both
approaches effectively allow using a nested function as a closure if
it's used after the function returns.)

The Gods decided, for practical reasons, the former was more important.
My experience bears this out: I find that about half the nested
functions I use are to reference an outer scope, half I return as a
closure. Only once or twice did I try to use a nested function inside
the defining function. I would guess this is more or less typical of
how nested functions are used. If so, it was the right decision.

 

[Kent Johnson]

What I want to do is pre-load functions with arguments by iterating
through a list like so:


... pass

... print arg

mylist = ["my", "sample", "list"]
for item in mylist:

... setattr(myclass, item, lamdba self: func(self, item))

This attaches a list of functions to the class, making them bound
methods when called from a class instance.


list
Unfortunately, it doesn't work. It seems the closure keeps track of
the variable fed to it dynamically - if the variable changes after the
lambda is created, the lambda still references the _variable_ not the
original _value_ and so gets the new value like so:

list

list

The closure isn't bound until the scope containing it is exited. A simple workaround is to bind item
as a default argument to the lambda:
for item in mylist:
setattr(myclass, item, lambda self, item=item: func(self, item))

You can also make a helper function that returns the closure, the closure will be bound each time
the helper returns:

def make_f(item):
def f(self): func(self, item)
return f

mylist = ["my", "sample", "list"]
for item in mylist:
setattr(myclass, item, make_f(item))

Kent

 

[Steven Bethard]

You may not be aware of it, but what you're trying to do is called
"currying"; you might want to search the Python Cookbook for recipes on
it.

Or look for "partial function application" which has been argued to be
the correct term for this use... Also see PEP 309:

http://www.python.org/peps/pep-0309.html

With an appropriately defined "partial" (I've taken the one from the PEP
and added the descriptor machinery necessary to make it work as an
instancemethod):

-------------------- functional.py --------------------
class partial(object):
def __init__(*args, **kwargs):
self = args[0]
try:
self.fn = args[1]
except IndexError:
raise TypeError('expected 2 or more arguments, got ' %
len(args))
self.args = args[2:]
self.kwargs = kwargs
def __call__(self, *args, **kwargs):
if kwargs and self.kwargs:
d = self.kwargs.copy()
d.update(kwargs)
else:
d = kwargs or self.kwargs
return self.fn(*(self.args + args), **d)
def __get__(self, obj, type=None):
if obj is None:
return self
return partial(self, obj)

-------------------------------------------------------

py> class C(object):
.... pass
....
py> def func(self, arg):
.... return arg
....
py> lst = ['a', 'b', 'c']
py> for item in lst:
.... setattr(C, item, functional.partial(func, arg=item))
....
py> c = C()
py> c.a(), c.b(), c.c()
('a', 'b', 'c')

STeVe

 

[jfj]

Ted Lilley wrote:

Unfortunately, it doesn't work. It seems the closure keeps track of
the variable fed to it dynamically - if the variable changes after [...]

At least, that's the explanation I'm deducing from this behavior.

And that's the correct explanation, chief.

It is intended that way. As an example of why that is: consider a
nested function called "printvars()" that you could insert in various
places within a function to print out the value of some local variable.
If you did that, you wouldn't want printvars to print the values at
the time it was bound, would you?

Allow me to disagree (and start a new "confused with closures" thread

We know that python does not have references to variables. To some
newcomers this may seem annoying but eventually they understand the
pythonic way and they do without them. But in the case of closures
python supports references!

Nested functions, seem to do two independent things:
1) reference variables of an outer local scoope
2) are functions bound dynamically to constants

These two are independent because in:
##############
def foo():
def nested():
print x
f = nested
x = 'sassad'
f()
x = 'aafdss'
f()
return f
##################

once foo() returns there is no way to modify 'x'!
It becomes a kind of constant.

So IMVHO, one will never need both features. You either want to
return a function "bound to a local value" *xor* "use a local
function which has access to locals as if they were globals".

Supposing the default behaviour was that python does what Carl
suggests (closures don't keep track of variables), would there
be an elegant pythonic way to implement "nested functions that
reference variables from their outer scope"? (although if find
this little useful and un-pythonic)

Personally, i think that this is a problem with the way lisp
and other languages define the term "closure". But python is
different IMO, and the ability to reference variables is not
useful.


cheers,

jfj

 

[jfj]

Ted Lilley wrote:

Unfortunately, it doesn't work. It seems the closure keeps track

of

the variable fed to it dynamically - if the variable changes after

[...]

At least, that's the explanation I'm deducing from this behavior.


And that's the correct explanation, chief.

It is intended that way. As an example of why that is: consider a
nested function called "printvars()" that you could insert i

various

places within a function to print out the value of some local

variable.

If you did that, you wouldn't want printvars to print the values

at

the time it was bound, would you?

Allow me to disagree (and start a new "confused with closures" thread

We know that python does not have references to variables. To some
newcomers this may seem annoying but eventually they understand the
pythonic way and they do without them. But in the case of closures
python supports references!

That's right. It's called Practicality beats purity.

Yes, but according to the python philosophy one could pass locals()
to the nested function and grab the values from there. Seems practical
for the rareness of this...

My experience bears this out: I find that about half the nested
functions I use are to reference an outer scope, half I return as a
closure. Only once or twice did I try to use a nested function inside
the defining function. I would guess this is more or less typical of
how nested functions are used. If so, it was the right decision.

The question is how many of the nested functions you use take advantage
of the fact that the deref'd variable is a *reference to a variable* and
not a *constant* (different constant for each definition of the nested
function of course).
Or, IOW, it seems very reasonable that if somebody wants to write the
"printvars()" function, he could simply pass locals().

I would accept this being the right decision if there was a statement
like "global var", called "deref var" with which nested funcs could
modify those variables. Then we could say:

#############
def foo():
def f1(y):
deref x
x = y
def f2():
print x
x=11
return f1, f2
####################
But one should go with OOP in that case instead.

Ogligatory Python 3000 suggestion:
I hope in python 3000, we'll get rid of CellObjects and insert
freevars at the consts of the function object at MAKE_CLOSURE.


jfj

 

[Ted Lilley]

Wow, a lot of great discussion. Almost a bit too much for me to
grasp...I do see two or more nuggets that really address my issue.

As a side note, I'm familiar with the term currying from a friend who
learned ML and Scheme quite some time ago. Not sure if that's the true
origin, but it was a sufficiently different context from Python (or at
least I thought) that I didn't want to rely on its meaning. I was also
sufficiently unsure of it's _exact_ meaning, since we're talking about
two slightly different models, that I didn't want to use the term for
that reason as well. It's gratifying to know that it was a relevant
concept afterall, the partial function application discussion
notwithstanding. It's also gratifying to see such a strong community
versed in functional programming styles. Although I've only started
working with FP in Python, it's been a useful tool for making my
programming simpler.

The first useful nugget is the crystalization method. I spent some
time thinking about the problem after I posted it and came up with the
same workaround, in fact, the exact same syntax. It's a bit inelegant,
but it gets the job done for what I want to do. I won't even attempt
to take a position in the debate about whether Python should work by
default the way it does or the way I want to use it. Seems we have
well-spoken advocates for both sides.

The other nifty nugget is Kent's suggestion for using old-style Python
default arguments to capture the variable value. Since it looks
slightly more elegant I'm going to give it a shot.

I have to say, I was happily surprised by the volume and quality of
response to my little issue. Thanks everyone!

Ted

 

[Ted Lilley]

I replied a few minutes ago thanking everyone for their pointers. Very
interesting reading. I don't see my post yet, so I do hope it comes
through. I'm using a new newsreading service and don't yet have
confidence in it.

In any case, the addition of the default-setting argument in the lambda
works famously and is only a slight modification to my code. While I
don't think it's particularly intuitive (I would hate to be a
hypothetical programmer trying to puzzle out what my code does), it
gets the job done and is a heck of a lot easier than attaching the list
of attributes manually as I would've done without Python's FP
capabilities.

Thanks again,

Ted

 

[Carl Banks]

Yes, but according to the python philosophy one could pass locals()
to the nested function and grab the values from there. Seems practical
for the rareness of this...

First of all, I really don't agree that it's a rare need. I use it
pretty often.

Second of all, the whole point of referencing the outer scope is so you
don't *have* to pass stuff around. Saying that you can accomplish the
same thing by passing locals() isn't right, because the thing you were
accomplishing was not having to pass anything around.

Say you have a suite of functions, all of which are called by some main
function and each other, and all of which need to access a lot of the
same data. The best, most straightforward way to do it is to have the
common data be a local variable of the main function, and nest the
suite inside it. The other way to do it, passing around a common data
structure, add complexity and hurts performance.

And it's a pretty common situation; a lot of recursive algorithms can
make very good use of this.

 

[Carl Banks]

Say you have a suite of functions, all of which are called by some main
function and each other, and all of which need to access a lot of the
same data. The best, most straightforward way to do it is to have the
common data be a local variable of the main function, and nest the
suite inside it. The other way to do it, passing around a common data
structure, add complexity and hurts performance.

This could use an example. It'll be a little mathematical; bear with
me.

Say you want to calculate a list of points of an iterated fractal.
Here's the idea: you have a set of linear transformations. You take
the origin (0,0) as the first point, and then apply each transformation
in turn to get a new point. You recursively apply each transformation
at each additional point, up to a certain depth.

What's the best way to write such a function? Using a nested function,
it's easy as pie. (Details may be a little wrong....)

.. def ifs(transformations, maxdepth):
.. def add_point(x,y,angle,depth):
.. ifspoints.append((x,y,angle))
.. if depth < maxdepth:
.. for t in transformations:
.. nx = x + t.x*cos(angle) - t.y*sin(angle)
.. ny = y + t.x*sin(angle) * t.y*cos(angle)
.. nangle = angle + t.angle
.. add_point(nx,ny,nangle,depth+1)
.. ifspoints = []
.. add_point(0.0, 0.0, 0.0, 0)
.. return ifspoints

If you didn't have the nested function, you'd have to pass not only
depth, but also maxdepth, transformations, and ifspoints around.

If you had something a little more complex than this, with more than
one nested function and more complex pieces of data you'd need to pass
around, the savings you get from not having to pass a data structure
around and referncing all that data though the structure can be quite a
lot.

 

[Dirk Thierbach]

As a side note, I'm familiar with the term currying from a friend who
learned ML and Scheme quite some time ago. Not sure if that's the true
origin, but it was a sufficiently different context from Python (or at
least I thought) that I didn't want to rely on its meaning. I was also
sufficiently unsure of it's _exact_ meaning,

The exact meaning is reasonably well explained, together with the
origins, on Wikipedia:

http://en.wikipedia.org/wiki/Currying

That meaning shouldn't change if applied to different computer languages.
I can understand the temptation to confuse partial application with
currying (for languages like Python which take tuples of arguments
by default, currying is one way to make partial application possible;
supplying default arguments is another), but nonetheless they are different
things.

- Dirk

 

[jfj]

Say you want to calculate a list of points of an iterated fractal.
Here's the idea: you have a set of linear transformations. You take
the origin (0,0) as the first point, and then apply each transformation
in turn to get a new point. You recursively apply each transformation
at each additional point, up to a certain depth.

I like these kind of things. Ahhh, the best use out of one's computer.

What's the best way to write such a function? Using a nested function,
it's easy as pie. (Details may be a little wrong....)

. def ifs(transformations, maxdepth):
. def add_point(x,y,angle,depth):
. ifspoints.append((x,y,angle))
. if depth < maxdepth:
. for t in transformations:
. nx = x + t.x*cos(angle) - t.y*sin(angle)
. ny = y + t.x*sin(angle) * t.y*cos(angle)
. nangle = angle + t.angle
. add_point(nx,ny,nangle,depth+1)
. ifspoints = []
. add_point(0.0, 0.0, 0.0, 0)
. return ifspoints

If you didn't have the nested function, you'd have to pass not only
depth, but also maxdepth, transformations, and ifspoints around.

I see. Some people complained that you have to use "self." all the
time in methods If that was lifted the above would be done with
a class then?


The costly extra feature is this:
###############
def foo():
def f():
print x
x=1
f()
x=2
f()
return f
foo()()
#############
which prints '1 2 2'

The fractal code runs a little _slower_ because of this ability.
Although the specific program does not take advantage of it!

If you had something a little more complex than this, with more than
one nested function and more complex pieces of data you'd need to pass
around, the savings you get from not having to pass a data structure
around and referncing all that data though the structure can be quite a
lot.

Generally, I believe OOP is a superset of nested functions (as a
feature). Nested functions make sense to me as primary python
def statements which ``create new code objects with arbitary constants''

When I see :
###########
def foo(x):
def bar():
print x
return bar
###############
and the call:
xx = foo(1)

I imagine that python runs:
def bar():
print 1
xx=bar

But alas, it doesn't
Nooooooooo. It carries around a silly cell object for the
rest of its lifetime. see xx.func_closure.
At least, I think this is what happens...


jfj

 

[Mike Meyer]

Say you have a suite of functions, all of which are called by some main
function and each other, and all of which need to access a lot of the
same data. The best, most straightforward way to do it is to have the
common data be a local variable of the main function, and nest the
suite inside it. The other way to do it, passing around a common data
structure, add complexity and hurts performance.

The third way to do it is to make the thing a class. Make all the
functions methods of the class, and all the data instance
variables. You may well want to make the main function the __call__
method of the class. In an OO environment, this may be a better, more
straightforward way than nesting fnctions.

See <URL:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52549 > for an
example of this.

<mike

 

[Carl Banks]

The third way to do it is to make the thing a class.

Which is kind of the same thing as passing a structure around, only
with a somewhat more convenient syntax. But yes.

Make all the
functions methods of the class, and all the data instance
variables. You may well want to make the main function the __call__
method of the class. In an OO environment, this may be a better, more
straightforward way than nesting fnctions.

There are good reasons for sometimes using a class, but
straightforwardness is, IMHO, not one of them.

You're writing a function (in the abstract sense: you are creating an
entity that you call with input to get output, and that has local
storage that doesn't need to exist afterwards). Is it more
straightforward to implement an (abstract) function with a function
object (i.e., the object that was designed exactly for that purpose),
or with a class (designed for a different purpose: to create new
types)? I'd definitely go with the former. Implementing an (abstract)
function as a class obscures its purpose.

I agree that there are many times when a class is a more appropriate
implementation of an (abstract) function, such as if your suite of
subfunctions has to rebind common variables, or if your (abstract)
function has so many return values it's better to define it as a class
and have the caller refer to the attributes. However, I'd say this
comes at the expense of straightforwardness (or, rather, it exchanges
one sort of striaghtforwardness for another sort judged more
important).


As for implementing an (abstract) function as a class just because
you're in an OO environment: that's just thinking backwards.

"OOP is the shizzle so I'm going to implement everything as a class,"
is simply the wrong way to think. There's a reason OOP is the shizzle:
because it's very often a good representation of the problem. But it
isn't always. And when it isn't, and when a better way exists (as in
this case), you should use the better way, even in the highly OO
environment. OOP is a means, not an end.

 

[Carl Banks]

The costly extra feature is this:
###############
def foo():
def f():
print x
x=1
f()
x=2
f()
return f
foo()()
#############
which prints '1 2 2'

The fractal code runs a little _slower_ because of this ability.
Although the specific program does not take advantage of it!

True, in this case. (Although it's not that much slower; I think it's
only one extra dereference per access plus a little overhead.) But try
to extend your mind a bit and imagine that the recursive nested
function is called as part of a larger routine. Say, for example,
something that seeks a certain shape for the fractal.

.. def ifs(transformations,maxdepth):
.. def add_point(...):
.. ...
.. def test_shape():
.. ...
.. while True:
.. add_point(0,0,0,0)
.. x = test_shape()
.. if x < THRESHOLD:
.. break
.. transformations = adjust(transformation)

transformations gets rebound, so you'd need a reference to it.

I realize that this is getting more and more situationally specific (I
wouldn't yet say contrived, though; I have referenced changing bindings
a few times). Maybe you have a point.

 

[Antoon Pardon]

Ted Lilley wrote:

Unfortunately, it doesn't work. It seems the closure keeps track of
the variable fed to it dynamically - if the variable changes after [...]

At least, that's the explanation I'm deducing from this behavior.

And that's the correct explanation, chief.

It is intended that way. As an example of why that is: consider a
nested function called "printvars()" that you could insert in various
places within a function to print out the value of some local variable.
If you did that, you wouldn't want printvars to print the values at
the time it was bound, would you?

Allow me to disagree (and start a new "confused with closures" thread

We know that python does not have references to variables. To some
newcomers this may seem annoying but eventually they understand the
pythonic way and they do without them. But in the case of closures
python supports references!

Nested functions, seem to do two independent things:
1) reference variables of an outer local scoope
2) are functions bound dynamically to constants

These two are independent because in:
##############
def foo():
def nested():
print x
f = nested
x = 'sassad'
f()
x = 'aafdss'
f()
return f
##################

once foo() returns there is no way to modify 'x'!
It becomes a kind of constant.

In this particular case yes. But not in general, what about
this:
.... l = []
.... def pop():
.... return l.pop()
.... def push(e):
.... l.append(e)
.... return pop, push
.... 2

 

[Antoon Pardon]

once foo() returns there is no way to modify 'x'!
It becomes a kind of constant.

In this particular case yes. But not in general, what about
this:

def F():

... l = []
... def pop():
... return l.pop()
... def push(e):
... l.append(e)
... return pop, push
...

Others will point this out, but if I'm fast enough...

This does not change the object referenced by l.
It calls methods of it and because it is mutable the containts
of the list are modified.
'l' is a list at address, 0xXXXX and that can never change once
F() has returned.

I'm sorry but in my understanding of english mutating an object
is changing or modifying it. Yes it is stil the same list but
that list was not unmodified.

But I'll get back at what seems you actually wanted to say:
That there is no way to rebind 'x' or in my case 'l' and
with that I have to agree although I personnaly find that
a lack in python

 

[Diez B. Roggisch]

But I'll get back at what seems you actually wanted to say:

That there is no way to rebind 'x' or in my case 'l' and
with that I have to agree although I personnaly find that
a lack in python

It's not only that way in python, but in java too. So it seems that there is
a fundamental principle behind it: In a language that allows sideeffects,
these will actually happen.

If you're after side-effect free programming, I recommend using a functional
programming language like ocaml or haskell. In these, you can't mutate
objects at all, only recreate them and rebind them to new variables. This
prevents whole classes of errors - but of course it also introduces all
kinds of other constraints on your programming style, e.g. using monads and
so on.

 

[Duncan Booth]

But I'll get back at what seems you actually wanted to say:
That there is no way to rebind 'x' or in my case 'l' and
with that I have to agree although I personnaly find that
a lack in python

'no way' is a bit strong. You can use hacks such as the one I posted a
couple of weeks back:
l = 'hello there'
def get():
return l
return hack.mksetter(lambda: l), get
'testing'

'no official, supported, or portable way' would be more accurate.

I've never actually felt the need to set variables from inside a nested
function, and I wouldn't use a hack like this in production code unless
there was a *very* good reason for it, but since I figured out how to
actually do it I'm suprised just how many things could actually be
simplified if there was an official way to do this.