Misunderstanding about closures

[Alexander May]

When I define a function in the body of a loop, why doesn't the function
"close" on the loop vairable? See example below.

Thanks,
Alex

C:\Documents and Settings\Alexander May>python
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

l=[]
for x in xrange(10):

.... def f():
.... return x
.... l.append(f)
....[<function f at 0x008E66F0>, <function f at 0x008E6AB0>, <function f at
0x008EC130>, <function f at 0x008EC170>, <functi
on f at 0x008EC1B0>, <function f at 0x008EC1F0>, <function f at 0x008EC230>,
.... f()
....
9
9
9
9
9
9
9
9
9
9

On the other hand, the following works as I expected.

l=[]
def makef(x):

.... def f():
.... return x
.... return f
........ l.append(makef(x))
....[<function f at 0x008E6AB0>, <function f at 0x008EC330>, <function f at
0x008EC1F0>, <function f at 0x008EC230>, <functi
on f at 0x008EC270>, <function f at 0x008EC2B0>, <function f at 0x008EC0F0>,
.... f()
....
0
1
2
3
4
5
6
7
8
9

 

[Michael Geary]

When I define a function in the body of a loop, why doesn't the
function "close" on the loop vairable? See example below.

Hi Alex,

Your message title is correct: you're misunderstanding how closures work.


A closure doesn't save the current value that's bound to a name, it saves a
reference to the name itself. If you bind a different value to the name
later, the closure will see that new value.

In your first example, there is a single local name x that each instance of
the f closure refers to, so they all see the same value.

In the second example, each time you call the makef function, you create a
new local name x that belongs to that instance of makef. So when you create
the closures inside makef, each one sees its own value that is unrelated to
the others.

BTW, you'll see the same effect in JavaScript or any other language that
supports closures.

-Mike

C:\Documents and Settings\Alexander May>python
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

l=[]
for x in xrange(10):

... def f():
... return x
... l.append(f)
...[<function f at 0x008E66F0>, <function f at 0x008E6AB0>, <function f at
0x008EC130>, <function f at 0x008EC170>, <functi
on f at 0x008EC1B0>, <function f at 0x008EC1F0>, <function f at 0x008EC230>,
... f()
...
9
9
9
9
9
9
9
9
9
9

On the other hand, the following works as I expected.

l=[]
def makef(x):

... def f():
... return x
... return f
...... l.append(makef(x))
...[<function f at 0x008E6AB0>, <function f at 0x008EC330>, <function f at
0x008EC1F0>, <function f at 0x008EC230>, <functi
on f at 0x008EC270>, <function f at 0x008EC2B0>, <function f at 0x008EC0F0>,
... f()
...
0
1
2
3
4
5
6
7
8
9

 

[fishboy]

When I define a function in the body of a loop, why doesn't the function
"close" on the loop vairable? See example below.

Hmmm, I'm having a hard time phrasing the answer to this in a form
besides, "Because the function doesn't 'close' on a loop variable"

Hrmmm
......
Ok, how about this

x = 'a'
for x in range(10):
foo(x)
bar(x)

What value do you expect 'x' to be in bar(x)?

Because it sounds like you were expecting it to be bar('a') instead of
bar(9).

Hrmm, I just remembered. That's a C thing, isn't it? The loop
variable being only defined inside the loop.

God that makes to happy. Forgetting stuff like that. Maybe in
another 20 years I'll forget everything I ever knew about static typed
languages and pass into hacker Nirvana.

 

[Alexander Schmolck]

Hi Alex,

Your message title is correct: you're misunderstanding how closures work.

Not necessarily. See below.

[snipped]

BTW, you'll see the same effect in JavaScript or any other language that
supports closures.

Not quite. In fact in the language that more or less started it all, scheme,
the standard iteration construct 'do' does indeed introduce a *new binding* on
each iteration.


;; Note: this is is a literate translation and *not* idiomatic scheme -- no
;; schemer would write it like that
(define l '()) ; an empty list

(do ((x 0 (+ 1 x))) ; start x with 0, then add 1 at each step
((= x 10)) ; stop when x is 10
(set! l (cons (lambda () x) l))) ; add a new function closing over x to
; the *front* of l

(set! l (reverse l)) ; since we added to the front we
; have to reverse l

((list-ref l 3)) ; get list element 3 and execute it

3 ;ANSWER

Common Lisp OTOH doesn't -- like python:

;; This is similarly non-idiomatic (and won't work in emacs lisp, which hasn't
;; got lexical scope)
(defvar l '()) ; an empty list

(do ((x 0 (+ 1 x))) ; start x with 0, then add 1 at each step
((= x 10)) ; when x is 10 return the reversed list
(setf l (cons (lambda () x) l))) ; add a new function closing over x to
; the *front* of l

(setq l (reverse l)) ; since we added to the front we
; have to reverse l

(funcall (nth 3 l)) ; get list element 3 and execute it

 

[Hung Jung Lu]

Your message title is correct: you're misunderstanding how closures work.

So seems to be your case.

In your first example, there is a single local name x that each instance of
the f closure refers to, so they all see the same value.

There is a name, period. This name is neither local or global. This
name is looked up first in the locals() dictionary, then in the
globals() dictionary. In this particular example, because the locals()
dictionary is empty, this name is actually pulled from globals().

In the second example, each time you call the makef function, you create a
new local name x that belongs to that instance of makef. So when you create
the closures inside makef, each one sees its own value that is unrelated to
the others.

And your explanation is supposed to enlighten a beginner?

A more complete explanation. The key is in the argument list. Because
'x' appears in the argument list of makef(x), this name is inserted
into the locals() dictionary of makef()'s scope. That is, the name 'x'
inside makef()'s scope is pulled from the locals() dictionary of that
scope. Now, due to the magic of nested scope (which was not always the
case in older versions of Python), this 'x' is also inserted into the
locals() dictionary of the nested f() function, and this 'x' is bound
to the value at that moment, because during the constructions of f(),
it is found that 'x' is used in expressions AND it exists in the
containing scope's locals() dictionary at the moment of construction.
In particular, it will not work correctly if you replace the statement
"return x" with "return eval('x')". Everything becomes more clear when
you insert statements to print out the locals() and globals()
dictionaries.

regards,

Hung Jung

 

[Michael Geary]

Not quite. In fact in the language that more or less started it all,
scheme, the standard iteration construct 'do' does indeed introduce
a *new binding* on each iteration.

Yeah, there are two separate issues here. It wasn't clear which one Alex M.
was misunderstanding, and I made an assumption about it (always a bad
idea!).

One issue, which I assumed was the problem, has nothing to do with loops,
but with the very nature of closures: What does a closure save, the current
value that a name or variable is bound to, or a reference to that variable?

The other issue is what you're talking about: Does a loop introduce new
bindings on each iteration or not?

To illustrate, here's a variation on Alex's example without the loop:

g = []

def outer():
x = 1
def f():
return x
g.append( f )
x = 2
g.append( f )

outer()
print g[0](), g[1]()

This prints:

2 2

If a closure saved the current value of a variable, it would print:

1 2

Now I am guessing that if you translated this code into any language that
supports closures, including Scheme, you would get the "2 2" result, is that
right? After all, this is pretty much the definition of a closure, that it
saves a reference, not the current value.

If the point of confusion was whether a loop creates new bindings or not,
then my reply was irrelevant--but maybe this discussion will help someone
else understand closures better.

Alex M., now you know the rest of the story...

-Mike

 

[Michael Geary]

And your explanation is supposed to enlighten a beginner?

A more complete explanation. The key is in the argument list.
Because 'x' appears in the argument list of makef(x), this name
is inserted into the locals() dictionary of makef()'s scope. That
is, the name 'x' inside makef()'s scope is pulled from the locals()
dictionary of that scope. Now, due to the magic of nested scope
(which was not always the case in older versions of Python),
this 'x' is also inserted into the locals() dictionary of the nested
f() function, and this 'x' is bound to the value at that moment,
because during the constructions of f(), it is found that 'x' is
used in expressions AND it exists in the containing scope's
locals() dictionary at the moment of construction. In particular,
it will not work correctly if you replace the statement "return x"
with "return eval('x')". Everything becomes more clear when
you insert statements to print out the locals() and globals()
dictionaries.

Thank you for the more complete and accurate explanation, Hung Jung. I was
thinking in two languages at once, JavaScript and Python, and I started to
write in terms of how closures work in JavaScript. Then I thought I'd better
make it more relevant to Python but didn't do a very good job switching
over.

-Mike

 

[Alexander Schmolck]

g = []
def outer():
x = 1
def f():
return x
g.append( f )
x = 2
g.append( f )

outer()
print g[0](), g[1]()

This prints:

2 2

Now I am guessing that if you translated this code into any language that
supports closures, including Scheme, you would get the "2 2" result, is that
right?

Yep -- here's the scheme version.

(define g '())
(define (outer)
(let* ((x 1)
(f (lambda () x)))
(set! g (cons f g))
(set! x 2)
(set! g (cons f g))))
(outer)
(list ((car g)) ((cadr g)))
=> (2 2)

'as

 

[Michele Simionato]

Not quite. In fact in the language that more or less started it all, scheme,
the standard iteration construct 'do' does indeed introduce a *new binding* on
each iteration.

Common Lisp OTOH doesn't -- like python:

<snip>

A thoughtful discussion of Python/Scheme/Lisp closures and for loops
was made by
Jacek Generowicz in this thread:

http://groups.google.it/groups?hl=i...da&btnG=Cerca&meta=group%3Dcomp.lang.python.*

 

[Alexander May]

Thanks everybody. I now have a complete understanding of all the issues.
Alex

Alexander Schmolck <[email protected]> wrote in message

<snip>

A thoughtful discussion of Python/Scheme/Lisp closures and for loops
was made by
Jacek Generowicz in this thread:

http://groups.google.it/groups?hl=i...cern.ch&rnum=1&prev=/groups?hl=it&lr=&ie=UTF-
8%26oe%3DUTF-8%26q%3Dsimionato%2Blambda%26btnG%3DCerca%26meta%3Dgroup%253Dco
mp.lang.python.*

 

[Jacek Generowicz]

Not quite. In fact in the language that more or less started it all,
scheme, the standard iteration construct 'do' does indeed introduce
a *new binding* on each iteration.

Yes, but this is a consequence of Scheme faking iteration with
recursion.

Common Lisp OTOH doesn't -- like python:

If you used recursion-based iteration constructs in either of these
languages, the same would happen.

Future Googlers please refer to:

http://www.google.com/[email protected]

 

[Alexander Schmolck]

Yes, but this is a consequence of Scheme faking iteration with
recursion.

Why "faking"? How is this CL-style do below inferior to the real thing (apart
from the fact that it's possibly inelgant or broken because it's quickly
hacked together by someone of limited competence as a scheme programmer?

;; cl-style do (with mandatory step clause, to make the code a bit more
;; compact)
(define-syntax cl-style-do
(syntax-rules ()
;; this...
[(cl-style-do ((var init step) ...) (end-test result ...) body ...)
;; ... gets rewritten as
(let ((var init) ...) ; normal let; similar to var = init; ...
(let loop () ; named let: like def loop(): ...; loop()
(if end-test
(begin
(if #f #f) ; trick to get void return value if no result form
result ...) ; is given
(begin body ...
(set! var step) ...
(loop)))))]
))

;; for comparison, this is a (again slightly simplified) scheme style do
(define-syntax scheme-style-do
(syntax-rules ()
;; this...
[(scheme-style-do ((var init step) ...) (end-test result ...) body ...)
;; ... gets rewritten as
(let loop ((var init) ...) ; like def loop(var...): ...; loop(init...)
(if end-test
(begin
(if #f #f)
result ...)
(begin body ...
(loop step ...))))]
))


Recycling the contorted pythonesque scheme example I posted earlier:

(define l '()) ; an empty list

(cl-style-do ((x 0 (+ 1 x))) ; start x with 0, then add 1 at each step
((= x 10)) ; stop when x is 10
(set! l (cons (lambda () x) l))) ; add a new function closing over x to
; the *front* of l

(set! l (reverse l)) ; since we added to the front we
; have to reverse l

((list-ref l 3)) ; get list element 3 and execute it

=> 10 (same as in CL; not 3 as with standard do)

If you used recursion-based iteration constructs in either of these
languages, the same would happen.

Future Googlers please refer to:

http://www.google.com/[email protected]

Again you seem to imply that somehow scheme does something inappropriate
and/or is somehow limiting compared to CL/python style iteration -- whereas
AFAICT the exact opposite applies.

I can't see how the new bindings that scheme's do introduces on each iteration
step matter unless you create a closure in the body of the do loop and in that
case chances are scheme's behavior is *precisely* what you want. And if, for
some (obscure ?) reason, CL style do is what you need then I think you can
trivially implement it as above.

OTOH, if you want scheme style tail recursion (which makes it feasible to
express some problems significantly simpler and quite a few more lucidly than
if you had to use plain iteration) in (standard) CL or python you're pretty
screwed -- you can to some extent fake it in CL, but even that requires a lot
of work. For python you'd have to write a compiler to standard python (or
bytecode).

'as

 

[Jacek Generowicz]

you seem to imply that somehow scheme does something inappropriate

No, that's what you inferred; I never intended to imply any such thing.

I merely pointed out the facts:

- Scheme iteration is recursion based

- Scheme lexical scoping is not significantly different from Python's;
the apparent differences can be explained by the above.

Please don't falsely attribute any value judgements on the subject, to
me.

 

[Alexander Schmolck]

No, that's what you inferred;

Hence "seem to".

I never intended to imply any such thing.

Great, but I thought that your post might give people the wrong idea and
that's why I wanted to clarify things a bit.

I merely pointed out the facts:

- Scheme iteration is recursion based

- Scheme lexical scoping is not significantly different from Python's;
the apparent differences can be explained by the above.

Since I neither claimed that scheme iteration is not recursion based nor that
lexical scoping from scheme is in any way different from python's you might in
fact have followed up to my post for a similar reason.

This dredging up of former quotes is generally a bad thing to do, but I'll do
it anyway in the hope to lay the whole thing to rest.

[in] scheme, the standard iteration construct 'do' does indeed introduce
a *new binding* on each iteration.

Yes, but this is a consequence of Scheme faking iteration with
recursion.

Not really. Also, faking X with Y is usually taken to imply that X is inferior
to Y (in general or at least in some specific capacity).

This is not the case here, in fact the behavior you can achieve with scheme's
recursion is a strict superset of what can be achieved with CL's and python's
iteration. Also, notably, the converse is not the case.

Unfortunately, I don't think that it's so easy (or even possible?) to
demonstrate it the other way around (ie show a Scheme snippet in which
add0(0) == add1(0) == 1), because of Scheme's insistence on using recursion
to implement iteration.

It think it is both possible and easy. Also, citing "X's insistence on Y" as a
cause for (some desirable) A being not easily (or at all) achievable, might
give the impression of a deficiency born out of mere stubborness.

If you can find a way of making a genuine loop in Scheme, you should
observe similar behaviour to that of the original Python example.

I hope the code I posted has shown that it is just about as easy to write a
version of DO that is to all extents and purposes identical to CL's eponymous
looping construct as it is to write the one with rebinding semantics that
comes bundled with scheme.

Please don't falsely attribute any value judgements on the subject, to
me.

I don't think I have.

By using the formulation "seem to imply" I have made it plain that this is
merely a possible *interpretation* of what you have said. I also hope to have
made it comprehensible why your post might be read in such a way and why
readers of it might profit from additional clarifying information.

'as

 

[Jacek Generowicz]

This dredging up of former quotes is generally a bad thing to do, but I'll do
it anyway in the hope to lay the whole thing to rest.

[in] scheme, the standard iteration construct 'do' does indeed introduce
a *new binding* on each iteration.

Yes, but this is a consequence of Scheme faking iteration with
recursion.

Not really.

What do you mean? (Forget the use of the word "faking".)
Iteration-based recursion introduces a new binding because each
iteration is a new function invocation with its concommittant binding
of local variables. _The same happens in Python_ if you make use of a
recursion-based iteration construct. That was the point of the post
under discussion; to show that the perceived differences beteween the
ways that closures work in Python and Scheme are a result of
differences in the recursion constructs used, rather than differences
in the way closures work.

I also hope to have made it comprehensible why your post might be
read in such a way and why readers of it might profit from
additional clarifying information.

I agree that some people might profit from additional clarifying
infromation.

I also think that most of the information you provided is OT in this
group ... and OT to the topic, which was about how _closures_ work in
_Python_, and not about the merits (or demertis) of Scheme iteration.

So, as you suggested, let's let it rest.


PS Re-reading my previous reply I see that I probably sounded to
agressive, defensive, whatever ... sorry.