scoping with lambda in loops

[Ian McMeans]

I was bitten by a bug today that depended on how lambda works. It took
me quite a while to realize what was going on.

First, I made multiple lambda functions inside a loop, each of which
depended on the current loop variable.

a = []
for index in range(5):

a.append(lambda: index)


Now, see if you can guess what the output was for each of the
functions in the list a:

a[0](), a[1](), a[2](), a[3](), a[4]()

I had expected it to be (0, 1, 2, 3, 4), but actually, it's:

(4, 4, 4, 4, 4)

This really surprised me. I guess what is happening is that each
lambda knows what the context of execution is where it was defined,
and doesn't actually evaluate until the function is called, and when
it does evaluate, it uses the current value of the variable. Is this
related to static scoping? A similar thing would happen if you defined
a nested function that used a variable declared in the outer function,
then changed that variable, and called the nested function.

Can someone recommend a way to code around this gotcha? I'm having
trouble. I want the functions created inside the loop to execute with
the value of the loop index at the moment when the function is made.

 

[Christos TZOTZIOY Georgiou]

First, I made multiple lambda functions inside a loop, each of which
depended on the current loop variable.

a = []
for index in range(5):

a.append(lambda: index)

Now, see if you can guess what the output was for each of the
functions in the list a:

a[0](), a[1](), a[2](), a[3](), a[4]()

I had expected it to be (0, 1, 2, 3, 4), but actually, it's:

(4, 4, 4, 4, 4)

This really surprised me. I guess what is happening is that each
lambda knows what the context of execution is where it was defined,
and doesn't actually evaluate until the function is called, and when
it does evaluate, it uses the current value of the variable. Is this
related to static scoping? A similar thing would happen if you defined
a nested function that used a variable declared in the outer function,
then changed that variable, and called the nested function.

Can someone recommend a way to code around this gotcha? I'm having
trouble. I want the functions created inside the loop to execute with
the value of the loop index at the moment when the function is made.

I think this is a FAQ (perhaps it was FAQ 6.10?), and you can find many
threads on the subject if you do a search on groups.google.com.

The typical way to deal with this, IIRC, is to change your lambda
declaration into:

a.append(lambda index=index: index)

so that index gets evaluated at definition time.

 

[Joost Kremers]

a = []
for index in range(5):

a.append(lambda: index) [...]
Can someone recommend a way to code around this gotcha? I'm having
trouble. I want the functions created inside the loop to execute with
the value of the loop index at the moment when the function is made.

intuitively, i would think this should do the trick:

a = []
for index in range(5):

.... a.append(lambda x=index: x)

and testing shows that it does indeed.

but the reason why is rather vague to me, (i'm still rather new at this...)
so perhaps i should think a little more before trying to explain. (and i'm
sure someone else will come and do it better than i ever could.)

 

[Andrew Koenig]

Ian> First, I made multiple lambda functions inside a loop, each of which
Ian> depended on the current loop variable.

a = []
for index in range(5):
a.append(lambda: index)

Ian> Now, see if you can guess what the output was for each of the
Ian> functions in the list a:

a[0](), a[1](), a[2](), a[3](), a[4]()

Ian> I had expected it to be (0, 1, 2, 3, 4), but actually, it's:

Ian> (4, 4, 4, 4, 4)

Ian> This really surprised me.

Suppose you did it this way:

a = []
for index in range(5):
def foo():
return index
a.append(foo)

What result would you expect now, and why?

 

[David Eppstein]

First, I made multiple lambda functions inside a loop, each of which
depended on the current loop variable.

a = []
for index in range(5):

a.append(lambda: index)


Now, see if you can guess what the output was for each of the
functions in the list a:

a[0](), a[1](), a[2](), a[3](), a[4]()

I had expected it to be (0, 1, 2, 3, 4), but actually, it's:

(4, 4, 4, 4, 4)

This really surprised me. I guess what is happening is that each
lambda knows what the context of execution is where it was defined,
and doesn't actually evaluate until the function is called, and when
it does evaluate, it uses the current value of the variable. Is this
related to static scoping?

It's related to closures. If you're using lambda, you're probably a
lisp programmer, and should know all about closures. Creating a
function object with def or lambda, within an outer function scope,
creates a closure for that outer function call. The inner function's
accesses to variables from the outer function will return the
most-recently-updated binding from the closure. If you call the outer
function again, you will get a different unrelated closure.

If you the inner function to have its own local variable that stores
some expression value as it existed at the creation time of the inner
function, rather than re-evaluating the expression whenever the inner
function is called, the standard way is to use a defaulted keyword
parameter:

a = []
for index in range(5):
a.append(lambda index=index: index)

or maybe more concisely

a = [lambda index=index: index for index in range(5)]

 

[martin z]

a = []

for index in range(5):
a.append(lambda index=index: index)

or maybe more concisely

a = [lambda index=index: index for index in range(5)]

You know how Python is supposed to be executable pseudocode? Well that
stuff is farking ugly. If I handed pseudocode like that into any TA in one
of my classes, I'd be toast. Is there any way to do that in a legible
manner?

 

[Dave Benjamin]

a = []
for index in range(5):
a.append(lambda index=index: index)

or maybe more concisely

a = [lambda index=index: index for index in range(5)]

You know how Python is supposed to be executable pseudocode? Well that
stuff is farking ugly. If I handed pseudocode like that into any TA in one
of my classes, I'd be toast. Is there any way to do that in a legible
manner?

The following reads pretty well to me:

produce_value = lambda value: lambda: value
a = [produce_value(index) for index in range(5)]
a[3]()

3

Dave

 

[David Eppstein]

a = []
for index in range(5):
a.append(lambda index=index: index)

or maybe more concisely

a = [lambda index=index: index for index in range(5)]

You know how Python is supposed to be executable pseudocode? Well that
stuff is farking ugly. If I handed pseudocode like that into any TA in one
of my classes, I'd be toast. Is there any way to do that in a legible
manner?[/QUOTE]

How about this:

def makefunction(x):
def thefunction():
return x
return thefunction

a = map(makefunction, range(5))

The identifiers are still a little uninformative, but it's hard to do
better without more information from the original poster...

 

[Jacek Generowicz]

a = []
for index in range(5):

a.append(lambda: index)


Now, see if you can guess what the output was for each of the
functions in the list a:

a[0](), a[1](), a[2](), a[3](), a[4]()

I had expected it to be (0, 1, 2, 3, 4), but actually, it's:

(4, 4, 4, 4, 4)

This really surprised me. I guess what is happening is that each
lambda knows what the context of execution is where it was defined,
and doesn't actually evaluate until the function is called, and when
it does evaluate, it uses the current value of the variable. Is this
related to static scoping?

It's related to _lexical_ scoping. It is called a lexical closure.

As of the time when nested scopes were introduced into Python,
whenever a name is referenced, it is first sought in the local lexical
scope (ie, the bit of text in the local function body); if it is not
found there, it is sought in the closest enclosing lexical scope (the
text of any enclosing functions), and so on; when you run out of
enclosing functions you try global, then builtin scope.

There is no "index" variable in your lambda's local scope, so it has
to resort to using the one in the global scope ... which, by the time
you get around to calling your functions, has been set to 4.

A similar thing would happen if you defined a nested function that
used a variable declared in the outer function, then changed that
variable, and called the nested function.
Yup.

Can someone recommend a way to code around this gotcha?

Make your own local binding. Function call parameters make local
bindings. So, within a lambda, you can achieve this by using a keyword
argument.

lambda index=index:index (or lambda i=index:i)

Now there is a local index and a global index, so the lambda uses the
local one. Because the default value is evaluated at the time the
lambda expression is evaluated, each local index has a value
corresponding to whatever the global index had at the time the lambda
was evaluated.