Misunderstanding about closures

Discussion in 'Python' started by Alexander May, Jun 7, 2004.

  1. When I define a function in the body of a loop, why doesn't the function
    "close" on the loop vairable? See example below.

    Thanks,
    Alex

    C:\Documents and Settings\Alexander May>python
    Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
    win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> l=[]
    >>> for x in xrange(10):

    .... def f():
    .... return x
    .... l.append(f)
    ....
    >>> l

    [<function f at 0x008E66F0>, <function f at 0x008E6AB0>, <function f at
    0x008EC130>, <function f at 0x008EC170>, <functi
    on f at 0x008EC1B0>, <function f at 0x008EC1F0>, <function f at 0x008EC230>,
    <function f at 0x008EC270>, <function f at
    0x008EC2B0>, <function f at 0x008EC2F0>]
    >>> for f in l:

    .... f()
    ....
    9
    9
    9
    9
    9
    9
    9
    9
    9
    9

    On the other hand, the following works as I expected.

    >>> l=[]
    >>> def makef(x):

    .... def f():
    .... return x
    .... return f
    ....
    >>> for x in xrange(10):

    .... l.append(makef(x))
    ....
    >>> l

    [<function f at 0x008E6AB0>, <function f at 0x008EC330>, <function f at
    0x008EC1F0>, <function f at 0x008EC230>, <functi
    on f at 0x008EC270>, <function f at 0x008EC2B0>, <function f at 0x008EC0F0>,
    <function f at 0x008EC370>, <function f at
    0x008EC3B0>, <function f at 0x008EC3F0>]
    >>> for f in l:

    .... f()
    ....
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    >>>
     
    Alexander May, Jun 7, 2004
    #1
    1. Advertising

  2. Alexander May wrote:
    > When I define a function in the body of a loop, why doesn't the
    > function "close" on the loop vairable? See example below.


    Hi Alex,

    Your message title is correct: you're misunderstanding how closures work.
    :)

    A closure doesn't save the current value that's bound to a name, it saves a
    reference to the name itself. If you bind a different value to the name
    later, the closure will see that new value.

    In your first example, there is a single local name x that each instance of
    the f closure refers to, so they all see the same value.

    In the second example, each time you call the makef function, you create a
    new local name x that belongs to that instance of makef. So when you create
    the closures inside makef, each one sees its own value that is unrelated to
    the others.

    BTW, you'll see the same effect in JavaScript or any other language that
    supports closures.

    -Mike

    > C:\Documents and Settings\Alexander May>python
    > Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
    > win32
    > Type "help", "copyright", "credits" or "license" for more information.
    > >>> l=[]
    > >>> for x in xrange(10):

    > ... def f():
    > ... return x
    > ... l.append(f)
    > ...
    > >>> l

    > [<function f at 0x008E66F0>, <function f at 0x008E6AB0>, <function f at
    > 0x008EC130>, <function f at 0x008EC170>, <functi
    > on f at 0x008EC1B0>, <function f at 0x008EC1F0>, <function f at

    0x008EC230>,
    > <function f at 0x008EC270>, <function f at
    > 0x008EC2B0>, <function f at 0x008EC2F0>]
    > >>> for f in l:

    > ... f()
    > ...
    > 9
    > 9
    > 9
    > 9
    > 9
    > 9
    > 9
    > 9
    > 9
    > 9
    >
    > On the other hand, the following works as I expected.
    >
    > >>> l=[]
    > >>> def makef(x):

    > ... def f():
    > ... return x
    > ... return f
    > ...
    > >>> for x in xrange(10):

    > ... l.append(makef(x))
    > ...
    > >>> l

    > [<function f at 0x008E6AB0>, <function f at 0x008EC330>, <function f at
    > 0x008EC1F0>, <function f at 0x008EC230>, <functi
    > on f at 0x008EC270>, <function f at 0x008EC2B0>, <function f at

    0x008EC0F0>,
    > <function f at 0x008EC370>, <function f at
    > 0x008EC3B0>, <function f at 0x008EC3F0>]
    > >>> for f in l:

    > ... f()
    > ...
    > 0
    > 1
    > 2
    > 3
    > 4
    > 5
    > 6
    > 7
    > 8
    > 9
    > >>>

    >
    >
     
    Michael Geary, Jun 7, 2004
    #2
    1. Advertising

  3. Alexander May

    fishboy Guest

    On Mon, 07 Jun 2004 03:27:22 GMT, "Alexander May"
    <> wrote:

    >When I define a function in the body of a loop, why doesn't the function
    >"close" on the loop vairable? See example below.
    >

    Hmmm, I'm having a hard time phrasing the answer to this in a form
    besides, "Because the function doesn't 'close' on a loop variable"

    Hrmmm
    ......
    Ok, how about this

    x = 'a'
    for x in range(10):
    foo(x)
    bar(x)

    What value do you expect 'x' to be in bar(x)?

    Because it sounds like you were expecting it to be bar('a') instead of
    bar(9).

    Hrmm, I just remembered. That's a C thing, isn't it? The loop
    variable being only defined inside the loop.

    God that makes to happy. Forgetting stuff like that. Maybe in
    another 20 years I'll forget everything I ever knew about static typed
    languages and pass into hacker Nirvana.

    ><{{{*>
     
    fishboy, Jun 7, 2004
    #3
  4. "Michael Geary" <> writes:

    > Alexander May wrote:
    >> When I define a function in the body of a loop, why doesn't the
    >> function "close" on the loop vairable? See example below.

    >
    > Hi Alex,
    >
    > Your message title is correct: you're misunderstanding how closures work.
    > :)


    Not necessarily. See below.

    [snipped]

    > BTW, you'll see the same effect in JavaScript or any other language that
    > supports closures.


    Not quite. In fact in the language that more or less started it all, scheme,
    the standard iteration construct 'do' does indeed introduce a *new binding* on
    each iteration.


    ;; Note: this is is a literate translation and *not* idiomatic scheme -- no
    ;; schemer would write it like that
    (define l '()) ; an empty list

    (do ((x 0 (+ 1 x))) ; start x with 0, then add 1 at each step
    ((= x 10)) ; stop when x is 10
    (set! l (cons (lambda () x) l))) ; add a new function closing over x to
    ; the *front* of l

    (set! l (reverse l)) ; since we added to the front we
    ; have to reverse l

    ((list-ref l 3)) ; get list element 3 and execute it
    > 3 ;ANSWER



    Common Lisp OTOH doesn't -- like python:

    ;; This is similarly non-idiomatic (and won't work in emacs lisp, which hasn't
    ;; got lexical scope)
    (defvar l '()) ; an empty list

    (do ((x 0 (+ 1 x))) ; start x with 0, then add 1 at each step
    ((= x 10)) ; when x is 10 return the reversed list
    (setf l (cons (lambda () x) l))) ; add a new function closing over x to
    ; the *front* of l

    (setq l (reverse l)) ; since we added to the front we
    ; have to reverse l

    (funcall (nth 3 l)) ; get list element 3 and execute it
    > 10
     
    Alexander Schmolck, Jun 7, 2004
    #4
  5. Alexander May

    Hung Jung Lu Guest

    "Michael Geary" <> wrote:
    >
    > Your message title is correct: you're misunderstanding how closures work.
    > :)


    So seems to be your case. :)

    > In your first example, there is a single local name x that each instance of
    > the f closure refers to, so they all see the same value.


    There is a name, period. This name is neither local or global. This
    name is looked up first in the locals() dictionary, then in the
    globals() dictionary. In this particular example, because the locals()
    dictionary is empty, this name is actually pulled from globals().

    > In the second example, each time you call the makef function, you create a
    > new local name x that belongs to that instance of makef. So when you create
    > the closures inside makef, each one sees its own value that is unrelated to
    > the others.


    And your explanation is supposed to enlighten a beginner?

    A more complete explanation. The key is in the argument list. Because
    'x' appears in the argument list of makef(x), this name is inserted
    into the locals() dictionary of makef()'s scope. That is, the name 'x'
    inside makef()'s scope is pulled from the locals() dictionary of that
    scope. Now, due to the magic of nested scope (which was not always the
    case in older versions of Python), this 'x' is also inserted into the
    locals() dictionary of the nested f() function, and this 'x' is bound
    to the value at that moment, because during the constructions of f(),
    it is found that 'x' is used in expressions AND it exists in the
    containing scope's locals() dictionary at the moment of construction.
    In particular, it will not work correctly if you replace the statement
    "return x" with "return eval('x')". Everything becomes more clear when
    you insert statements to print out the locals() and globals()
    dictionaries.

    regards,

    Hung Jung
     
    Hung Jung Lu, Jun 7, 2004
    #5
  6. > > Alexander May wrote:
    > >> When I define a function in the body of a loop, why doesn't the
    > >> function "close" on the loop vairable? See example below.


    > Michael Geary wrote:
    > > Your message title is correct: you're misunderstanding how
    > > closures work. :)
    > >
    > > BTW, you'll see the same effect in JavaScript or any other
    > > language that supports closures.


    Alexander Schmolck wrote:
    > Not quite. In fact in the language that more or less started it all,
    > scheme, the standard iteration construct 'do' does indeed introduce
    > a *new binding* on each iteration.


    Yeah, there are two separate issues here. It wasn't clear which one Alex M.
    was misunderstanding, and I made an assumption about it (always a bad
    idea!).

    One issue, which I assumed was the problem, has nothing to do with loops,
    but with the very nature of closures: What does a closure save, the current
    value that a name or variable is bound to, or a reference to that variable?

    The other issue is what you're talking about: Does a loop introduce new
    bindings on each iteration or not?

    To illustrate, here's a variation on Alex's example without the loop:

    g = []

    def outer():
    x = 1
    def f():
    return x
    g.append( f )
    x = 2
    g.append( f )

    outer()
    print g[0](), g[1]()

    This prints:

    2 2

    If a closure saved the current value of a variable, it would print:

    1 2

    Now I am guessing that if you translated this code into any language that
    supports closures, including Scheme, you would get the "2 2" result, is that
    right? After all, this is pretty much the definition of a closure, that it
    saves a reference, not the current value.

    If the point of confusion was whether a loop creates new bindings or not,
    then my reply was irrelevant--but maybe this discussion will help someone
    else understand closures better.

    Alex M., now you know the rest of the story... :)

    -Mike
     
    Michael Geary, Jun 7, 2004
    #6
  7. Hung Jung Lu wrote:
    > And your explanation is supposed to enlighten a beginner?
    >
    > A more complete explanation. The key is in the argument list.
    > Because 'x' appears in the argument list of makef(x), this name
    > is inserted into the locals() dictionary of makef()'s scope. That
    > is, the name 'x' inside makef()'s scope is pulled from the locals()
    > dictionary of that scope. Now, due to the magic of nested scope
    > (which was not always the case in older versions of Python),
    > this 'x' is also inserted into the locals() dictionary of the nested
    > f() function, and this 'x' is bound to the value at that moment,
    > because during the constructions of f(), it is found that 'x' is
    > used in expressions AND it exists in the containing scope's
    > locals() dictionary at the moment of construction. In particular,
    > it will not work correctly if you replace the statement "return x"
    > with "return eval('x')". Everything becomes more clear when
    > you insert statements to print out the locals() and globals()
    > dictionaries.


    Thank you for the more complete and accurate explanation, Hung Jung. I was
    thinking in two languages at once, JavaScript and Python, and I started to
    write in terms of how closures work in JavaScript. Then I thought I'd better
    make it more relevant to Python but didn't do a very good job switching
    over. :)

    -Mike
     
    Michael Geary, Jun 7, 2004
    #7
  8. "Michael Geary" <> writes:
    > g = []
    > def outer():
    > x = 1
    > def f():
    > return x
    > g.append( f )
    > x = 2
    > g.append( f )
    >
    > outer()
    > print g[0](), g[1]()
    >
    > This prints:
    >
    > 2 2


    > Now I am guessing that if you translated this code into any language that
    > supports closures, including Scheme, you would get the "2 2" result, is that
    > right?


    Yep -- here's the scheme version.

    (define g '())
    (define (outer)
    (let* ((x 1)
    (f (lambda () x)))
    (set! g (cons f g))
    (set! x 2)
    (set! g (cons f g))))
    (outer)
    (list ((car g)) ((cadr g)))
    => (2 2)

    'as
     
    Alexander Schmolck, Jun 7, 2004
    #8
  9. Alexander Schmolck <> wrote in message news:<>...
    > Not quite. In fact in the language that more or less started it all, scheme,
    > the standard iteration construct 'do' does indeed introduce a *new binding* on
    > each iteration.

    <snip>
    > Common Lisp OTOH doesn't -- like python:
    >

    <snip>

    A thoughtful discussion of Python/Scheme/Lisp closures and for loops
    was made by
    Jacek Generowicz in this thread:

    http://groups.google.it/groups?hl=i...da&btnG=Cerca&meta=group%3Dcomp.lang.python.*
     
    Michele Simionato, Jun 8, 2004
    #9
  10. Thanks everybody. I now have a complete understanding of all the issues.
    Alex

    "Michele Simionato" <> wrote in message
    news:...
    > Alexander Schmolck <> wrote in message

    news:<>...
    > > Not quite. In fact in the language that more or less started it all,

    scheme,
    > > the standard iteration construct 'do' does indeed introduce a *new

    binding* on
    > > each iteration.

    > <snip>
    > > Common Lisp OTOH doesn't -- like python:
    > >

    > <snip>
    >
    > A thoughtful discussion of Python/Scheme/Lisp closures and for loops
    > was made by
    > Jacek Generowicz in this thread:
    >
    >

    http://groups.google.it/groups?hl=i...cern.ch&rnum=1&prev=/groups?hl=it&lr=&ie=UTF-
    8%26oe%3DUTF-8%26q%3Dsimionato%2Blambda%26btnG%3DCerca%26meta%3Dgroup%253Dco
    mp.lang.python.*
     
    Alexander May, Jun 8, 2004
    #10
  11. Alexander Schmolck <> writes:

    > Not quite. In fact in the language that more or less started it all,
    > scheme, the standard iteration construct 'do' does indeed introduce
    > a *new binding* on each iteration.


    Yes, but this is a consequence of Scheme faking iteration with
    recursion.

    > Common Lisp OTOH doesn't -- like python:


    If you used recursion-based iteration constructs in either of these
    languages, the same would happen.

    Future Googlers please refer to:

    http://www.google.com/groups?as_umsgid=
     
    Jacek Generowicz, Jun 9, 2004
    #11
  12. Jacek Generowicz <> writes:

    > Alexander Schmolck <> writes:
    >
    >> Not quite. In fact in the language that more or less started it all,
    >> scheme, the standard iteration construct 'do' does indeed introduce
    >> a *new binding* on each iteration.

    >
    > Yes, but this is a consequence of Scheme faking iteration with
    > recursion.


    Why "faking"? How is this CL-style do below inferior to the real thing (apart
    from the fact that it's possibly inelgant or broken because it's quickly
    hacked together by someone of limited competence as a scheme programmer:)?

    ;; cl-style do (with mandatory step clause, to make the code a bit more
    ;; compact)
    (define-syntax cl-style-do
    (syntax-rules ()
    ;; this...
    [(cl-style-do ((var init step) ...) (end-test result ...) body ...)
    ;; ... gets rewritten as
    (let ((var init) ...) ; normal let; similar to var = init; ...
    (let loop () ; named let: like def loop(): ...; loop()
    (if end-test
    (begin
    (if #f #f) ; trick to get void return value if no result form
    result ...) ; is given
    (begin body ...
    (set! var step) ...
    (loop)))))]
    ))

    ;; for comparison, this is a (again slightly simplified) scheme style do
    (define-syntax scheme-style-do
    (syntax-rules ()
    ;; this...
    [(scheme-style-do ((var init step) ...) (end-test result ...) body ...)
    ;; ... gets rewritten as
    (let loop ((var init) ...) ; like def loop(var...): ...; loop(init...)
    (if end-test
    (begin
    (if #f #f)
    result ...)
    (begin body ...
    (loop step ...))))]
    ))


    Recycling the contorted pythonesque scheme example I posted earlier:

    (define l '()) ; an empty list

    (cl-style-do ((x 0 (+ 1 x))) ; start x with 0, then add 1 at each step
    ((= x 10)) ; stop when x is 10
    (set! l (cons (lambda () x) l))) ; add a new function closing over x to
    ; the *front* of l

    (set! l (reverse l)) ; since we added to the front we
    ; have to reverse l

    ((list-ref l 3)) ; get list element 3 and execute it

    => 10 (same as in CL; not 3 as with standard do)


    >> Common Lisp OTOH doesn't -- like python:

    >
    > If you used recursion-based iteration constructs in either of these
    > languages, the same would happen.
    >
    > Future Googlers please refer to:
    >
    > http://www.google.com/groups?as_umsgid=


    Again you seem to imply that somehow scheme does something inappropriate
    and/or is somehow limiting compared to CL/python style iteration -- whereas
    AFAICT the exact opposite applies.

    I can't see how the new bindings that scheme's do introduces on each iteration
    step matter unless you create a closure in the body of the do loop and in that
    case chances are scheme's behavior is *precisely* what you want. And if, for
    some (obscure ?) reason, CL style do is what you need then I think you can
    trivially implement it as above.

    OTOH, if you want scheme style tail recursion (which makes it feasible to
    express some problems significantly simpler and quite a few more lucidly than
    if you had to use plain iteration) in (standard) CL or python you're pretty
    screwed -- you can to some extent fake it in CL, but even that requires a lot
    of work. For python you'd have to write a compiler to standard python (or
    bytecode).

    'as
     
    Alexander Schmolck, Jun 12, 2004
    #12
  13. Alexander Schmolck <> writes:

    > you seem to imply that somehow scheme does something inappropriate


    No, that's what you inferred; I never intended to imply any such thing.

    I merely pointed out the facts:

    - Scheme iteration is recursion based

    - Scheme lexical scoping is not significantly different from Python's;
    the apparent differences can be explained by the above.

    Please don't falsely attribute any value judgements on the subject, to
    me.
     
    Jacek Generowicz, Jun 16, 2004
    #13
  14. Jacek Generowicz <> writes:

    > Alexander Schmolck <> writes:
    >
    >> you seem to imply that somehow scheme does something inappropriate

    >
    > No, that's what you inferred;


    Hence "seem to".

    > I never intended to imply any such thing.


    Great, but I thought that your post might give people the wrong idea and
    that's why I wanted to clarify things a bit.

    > I merely pointed out the facts:
    >
    > - Scheme iteration is recursion based
    >
    > - Scheme lexical scoping is not significantly different from Python's;
    > the apparent differences can be explained by the above.


    Since I neither claimed that scheme iteration is not recursion based nor that
    lexical scoping from scheme is in any way different from python's you might in
    fact have followed up to my post for a similar reason.

    This dredging up of former quotes is generally a bad thing to do, but I'll do
    it anyway in the hope to lay the whole thing to rest.

    > Alexander Schmolck <> writes:
    >> [in] scheme, the standard iteration construct 'do' does indeed introduce
    >> a *new binding* on each iteration.


    > Yes, but this is a consequence of Scheme faking iteration with
    > recursion.


    Not really. Also, faking X with Y is usually taken to imply that X is inferior
    to Y (in general or at least in some specific capacity).

    This is not the case here, in fact the behavior you can achieve with scheme's
    recursion is a strict superset of what can be achieved with CL's and python's
    iteration. Also, notably, the converse is not the case.

    > Unfortunately, I don't think that it's so easy (or even possible?) to
    > demonstrate it the other way around (ie show a Scheme snippet in which
    > add0(0) == add1(0) == 1), because of Scheme's insistence on using recursion
    > to implement iteration.


    It think it is both possible and easy. Also, citing "X's insistence on Y" as a
    cause for (some desirable) A being not easily (or at all) achievable, might
    give the impression of a deficiency born out of mere stubborness.

    > If you can find a way of making a genuine loop in Scheme, you should
    > observe similar behaviour to that of the original Python example.


    I hope the code I posted has shown that it is just about as easy to write a
    version of DO that is to all extents and purposes identical to CL's eponymous
    looping construct as it is to write the one with rebinding semantics that
    comes bundled with scheme.

    > Please don't falsely attribute any value judgements on the subject, to
    > me.


    I don't think I have.

    By using the formulation "seem to imply" I have made it plain that this is
    merely a possible *interpretation* of what you have said. I also hope to have
    made it comprehensible why your post might be read in such a way and why
    readers of it might profit from additional clarifying information.

    'as
     
    Alexander Schmolck, Jun 16, 2004
    #14
  15. Alexander Schmolck <> writes:

    > This dredging up of former quotes is generally a bad thing to do, but I'll do
    > it anyway in the hope to lay the whole thing to rest.
    >
    > > Alexander Schmolck <> writes:
    > >> [in] scheme, the standard iteration construct 'do' does indeed introduce
    > >> a *new binding* on each iteration.

    >
    > > Yes, but this is a consequence of Scheme faking iteration with
    > > recursion.

    >
    > Not really.


    What do you mean? (Forget the use of the word "faking".)
    Iteration-based recursion introduces a new binding because each
    iteration is a new function invocation with its concommittant binding
    of local variables. _The same happens in Python_ if you make use of a
    recursion-based iteration construct. That was the point of the post
    under discussion; to show that the perceived differences beteween the
    ways that closures work in Python and Scheme are a result of
    differences in the recursion constructs used, rather than differences
    in the way closures work.

    > I also hope to have made it comprehensible why your post might be
    > read in such a way and why readers of it might profit from
    > additional clarifying information.


    I agree that some people might profit from additional clarifying
    infromation.

    I also think that most of the information you provided is OT in this
    group ... and OT to the topic, which was about how _closures_ work in
    _Python_, and not about the merits (or demertis) of Scheme iteration.

    So, as you suggested, let's let it rest.


    PS Re-reading my previous reply I see that I probably sounded to
    agressive, defensive, whatever ... sorry.
     
    Jacek Generowicz, Jun 17, 2004
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mark Hobley

    vec - a misunderstanding

    Mark Hobley, Mar 18, 2006, in forum: Perl
    Replies:
    1
    Views:
    2,621
    John Bokma
    Mar 18, 2006
  2. Kevin Spencer

    Re: Class property misunderstanding

    Kevin Spencer, Jun 26, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    840
    Kevin Spencer
    Jun 26, 2003
  3. Philip Tripp

    View State Misunderstanding

    Philip Tripp, Jul 24, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    546
    Philip Tripp
    Jul 26, 2004
  4. Shawn Wildermuth

    Misunderstanding ConnectionStrings?

    Shawn Wildermuth, Jan 9, 2006, in forum: ASP .Net
    Replies:
    6
    Views:
    1,930
    Steven Cheng[MSFT]
    Jan 11, 2006
  5. Robert Brewer

    RE: Misunderstanding about closures

    Robert Brewer, Jun 7, 2004, in forum: Python
    Replies:
    1
    Views:
    434
    Hung Jung Lu
    Jun 7, 2004
Loading...

Share This Page