Confused about closures and scoping rules

[Fernando Perez]

Hi all,

consider the following small example:

"""
Small test to try to understand a strange subtlety with closures
"""

def outer(nmax):

aa = []
for n in range(nmax):

def a(y):
return (y,n)
print 'Closure and cell id:',id(a.func_closure),\
id(a.func_closure[0])
aa.append(a)

return aa

print 'Closure creation.'
nmax = 3
aa = outer(nmax)

print
print 'Closure use.'
for n in range(nmax):
print '%s:%s' % (n,aa[n]('hello'))

################## EOF #################


If I run this, I get:

planck[test]> python debug_closures.py
Closure creation.
Closure and cell id: 1075998828 1075618940
Closure and cell id: 1075999052 1075618940
Closure and cell id: 1075999084 1075618940

Closure use.
0'hello', 2)
1'hello', 2)
2'hello', 2)


My confusion arises from the printout after 'closure use'. I was expecting that
each new function 'a' created inside the loop in 'outer' would capture the
value of n, therefore my expectation was to see a printout like:

0'hello', 0)
1'hello', 1)... etc.

However, what happens is a bit different. As can be seen from the printouts
of 'Closure and cell id', in each pass of the loop a new closure is created,
but it reuses the *same* cell object every time. For this reason, all the
closures end up sharing the scope with the values determined by the *last*
iteration of the loop.

This struck me as counterintuitive, but I couldn't find anything in the
official docs indicating what the expected behavior should be. Any
feedback/enlightenment would be welcome. This problem appeared deep inside a
complicated code and it took me almost two days to track down what was going
on...

Cheers,

f

 

[Diez B. Roggisch]

Hi all,

consider the following small example:

"""
Small test to try to understand a strange subtlety with closures
"""

def outer(nmax):

aa = []
for n in range(nmax):

def a(y):
return (y,n)
print 'Closure and cell id:',id(a.func_closure),\
id(a.func_closure[0])
aa.append(a)

return aa

print 'Closure creation.'
nmax = 3
aa = outer(nmax)

print
print 'Closure use.'
for n in range(nmax):
print '%s:%s' % (n,aa[n]('hello'))

################## EOF #################


If I run this, I get:

planck[test]> python debug_closures.py
Closure creation.
Closure and cell id: 1075998828 1075618940
Closure and cell id: 1075999052 1075618940
Closure and cell id: 1075999084 1075618940

Closure use.
0'hello', 2)
1'hello', 2)
2'hello', 2)


My confusion arises from the printout after 'closure use'. I was expecting that
each new function 'a' created inside the loop in 'outer' would capture the
value of n, therefore my expectation was to see a printout like:

0'hello', 0)
1'hello', 1)... etc.

However, what happens is a bit different. As can be seen from the printouts
of 'Closure and cell id', in each pass of the loop a new closure is created,
but it reuses the *same* cell object every time. For this reason, all the
closures end up sharing the scope with the values determined by the *last*
iteration of the loop.

This struck me as counterintuitive, but I couldn't find anything in the
official docs indicating what the expected behavior should be. Any
feedback/enlightenment would be welcome. This problem appeared deep inside a
complicated code and it took me almost two days to track down what was going
on...

It's a FAQ. The reason is that the created closures don't capture the
_value_, but the _name_. Plus of course the locals()-dictionary outside
the function a to perform the lookup of that name. Which has the value
bound to it in the last iteration.

Common cure for this is to create an a-local name that shadows the outer
variable and is simultaneously bound to the desired value:

def outer(nmax):

aa = []
for n in range(nmax):
foo = 'bar'
def a(y,n=n):
bar = foo
return (y,n)
print 'Closure and cell id:',id(a.func_closure),\
id(a.func_closure[0])
aa.append(a)

return aa

print 'Closure creation.'
nmax = 3
aa = outer(nmax)

print
print 'Closure use.'
for n in range(nmax):
print '%s:%s' % (n,aa[n]('hello'))


Notice the foo/bar - that was necessary to actually create a closure at
all (to keep your printing working), as python statically checks if
there needs one to be.



Diez

 

[Fernando Perez]

Diez B. Roggisch wrote:

It's a FAQ. The reason is that the created closures don't capture the
_value_, but the _name_. Plus of course the locals()-dictionary outside
the function a to perform the lookup of that name. Which has the value
bound to it in the last iteration.

Common cure for this is to create an a-local name that shadows the outer
variable and is simultaneously bound to the desired value:

Many thanks (also to JP) for the clear explanation. Greatly appreciated.

Cheers,

f