closures and dynamic binding

Discussion in 'Python' started by Aaron \Castironpi\ Brady, Sep 28, 2008.

  1. Hello all,

    To me, this is a somewhat unintuitive behavior. I want to discuss the
    parts of it I don't understand.

    >>> f= [ None ]* 10
    >>> for n in range( 10 ):

    .... f[ n ]= lambda: n
    ....
    >>> f[0]()

    9
    >>> f[1]()

    9

    I guess I can accept this part so far, though it took a little getting
    used to. I'm writing some code and found the following workaround,
    but I don't think it should give different results. Maybe I'm not
    understanding some of the details of closures.

    >>> f= [ None ]* 10
    >>> for n in range( 10 ):

    .... f[ n ]= (lambda n: ( lambda: n ) )( n )
    ....
    >>> f[0]()

    0
    >>> f[1]()

    1

    Which is of course the desired effect. Why doesn't the second one
    just look up what 'n' is when I call f[0], and return 9?
    Aaron \Castironpi\ Brady, Sep 28, 2008
    #1
    1. Advertising

  2. On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:

    > To me, this is a somewhat unintuitive behavior. I want to discuss the
    > parts of it I don't understand.
    >
    >>>> f= [ None ]* 10
    >>>> for n in range( 10 ):

    > ... f[ n ]= lambda: n
    > ...
    >>>> f[0]()

    > 9
    >>>> f[1]()

    > 9


    `n` is looked up at the time ``f[0]`` is called. At that time it is
    bound to 9.

    >>>> f= [ None ]* 10
    >>>> for n in range( 10 ):

    > ... f[ n ]= (lambda n: ( lambda: n ) )( n ) ...
    >>>> f[0]()

    > 0
    >>>> f[1]()

    > 1
    >
    > Which is of course the desired effect. Why doesn't the second one just
    > look up what 'n' is when I call f[0], and return 9?


    It *does* look up `n` at the time you call ``f[0]`` but this time it's
    the local `n` of the outer ``lambda`` function and that is bound to
    whatever the function was called with. At the time it was called the
    global `n` was bound to 0. Maybe it get's more clear if you don't name
    it `n`:

    In [167]: f = [None] * 10

    In [168]: for n in xrange(10):
    .....: f[n] = (lambda x: lambda: x)(n)
    .....:

    In [169]: f[0]()
    Out[169]: 0

    Ciao,
    Marc 'BlackJack' Rintsch
    Marc 'BlackJack' Rintsch, Sep 28, 2008
    #2
    1. Advertising

  3. On Sep 28, 1:14 am, Marc 'BlackJack' Rintsch <> wrote:
    > On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:
    > > To me, this is a somewhat unintuitive behavior.  I want to discuss the
    > > parts of it I don't understand.

    >
    > >>>> f= [ None ]* 10
    > >>>> for n in range( 10 ):

    > > ...     f[ n ]= lambda: n
    > > ...
    > >>>> f[0]()

    > > 9
    > >>>> f[1]()

    > > 9

    >
    > `n` is looked up at the time ``f[0]`` is called.  At that time it is
    > bound to 9.
    >
    > >>>> f= [ None ]* 10
    > >>>> for n in range( 10 ):

    > > ...     f[ n ]= (lambda n: ( lambda: n ) )( n ) ...
    > >>>> f[0]()

    > > 0
    > >>>> f[1]()

    > > 1

    >
    > > Which is of course the desired effect.  Why doesn't the second one just
    > > look up what 'n' is when I call f[0], and return 9?

    >
    > It *does* look up `n` at the time you call ``f[0]`` but this time it's
    > the local `n` of the outer ``lambda`` function and that is bound to
    > whatever the function was called with.  At the time it was called the
    > global `n` was bound to 0.  Maybe it get's more clear if you don't name
    > it `n`:
    >
    > In [167]: f = [None] * 10
    >
    > In [168]: for n in xrange(10):
    >    .....:     f[n] = (lambda x: lambda: x)(n)
    >    .....:
    >
    > In [169]: f[0]()
    > Out[169]: 0
    >
    > Ciao,
    >         Marc 'BlackJack' Rintsch


    Hi Marc,

    It's my understanding that 'n' gets a new value every time through the
    loop. n= 0 on the first pass, n= 1 on the second pass, and so on. n=
    9 by the end, and that's why `lambda: n` always returns 9. It queries
    the variable 'n', and finds 9. (This got lengthy. I started thinking
    aloud.)

    In your version of the indirect example, it queries the variable 'x',
    which it finds in a new distinct scope in each f element. In f[0], x=
    0. In f[1], x= 1. There are 10 different 'x' variables throughout
    the contents of f. The direct example does not do this allocation of
    ten different 'x's.

    It's sort of helping. I think I feel like the following is more like
    what I'm looking for:

    (Non-standard)
    >>> f= [ None ]* 10
    >>> for n in range( 10 ):

    .... f[ n ]= n
    ....
    >>> f[0]

    9
    >>> f[1]

    9

    because when you access f[0], it looks up the variable 'n'. Obviously
    not.

    (Unproduced)
    >>> f= [ None ]* 10
    >>> for n in range( 10 ):

    .... f[ n ]= late( n ) #late/lazy
    ....
    >>> f[0]

    9
    >>> f[1]

    9

    >>> f= [ None ]* 10
    >>> for n in range( 10 ):

    .... f[ n ]= early( n ) #early/eager
    ....
    >>> f[0]

    0
    >>> f[1]

    1

    For the functions, I want a function that returns 'n', either late or
    early.

    (Unproduced)
    >>> for n in range( 10 ):

    .... f[ n ]= lambda: late( n )
    >>> f[0]()

    9
    >>> for n in range( 10 ):

    .... f[ n ]= lambda: early( n )
    >>> f[0]()

    0

    I don't think you could pull this off. 'late' and 'early' succeed
    with quotes around n, early('n') and late('n'), in the direct
    assignments, but the functions aren't so lucky. 'n' has gone on to a
    better world by the time 'early' gets any control.

    This might have some success.

    (Unproduced)
    >>> for n in range( 10 ):

    .... f[ n ]= late( lambda: n )
    >>> f[0]()

    9
    >>> for n in range( 10 ):

    .... f[ n ]= early( lambda: n )
    >>> f[0]()

    0

    Though it's beyond my foresight to tell if it's feasible as stated, if
    you need quotes, how much introspection you would need, etc. Plus,
    'late' and 'early' were accepting quoted parameters earlier. How
    would they look inside a function? Could a simple decorator provide
    the service?

    On a tangent about mutables, it's not that numbers, strings, and
    tuples are 'immutable' per se, it's just that they don't have any
    methods which mutate them (or assignable properties). Lists and
    dictionaries do. It's up to the user whether a custom class does.
    Aaron \Castironpi\ Brady, Sep 28, 2008
    #3
  4. Aaron \Castironpi\ Brady

    Terry Reedy Guest

    Aaron "Castironpi" Brady wrote:
    > Hello all,
    >
    > To me, this is a somewhat unintuitive behavior. I want to discuss the
    > parts of it I don't understand.
    >
    >>>> f= [ None ]* 10
    >>>> for n in range( 10 ):

    > ... f[ n ]= lambda: n


    This is equivalent to

    for n in range(10):
    def g(): return n
    f[n] = g

    which is equivalent to

    def g(): return n
    f = [g]*10
    n = 9

    >>>> f[0]()

    > 9
    >>>> f[1]()

    > 9


    which make this not so surprising as the original lambda version is to
    some people.

    > I guess I can accept this part so far, though it took a little getting
    > used to. I'm writing some code and found the following workaround,
    > but I don't think it should give different results. Maybe I'm not
    > understanding some of the details of closures.
    >
    >>>> f= [ None ]* 10
    >>>> for n in range( 10 ):

    > ... f[ n ]= (lambda n: ( lambda: n ) )( n )


    This is equivalent to

    for n in range(10):
    def g(n):
    def h:
    return n
    return h
    f[n] = g(n)

    Now, to avoid the needless confusion of 'n's, g is equivalent to

    def g(x):
    def h:
    return x
    return h

    (One could do the same change in the lambdas, too, of course).
    so that g(n)() == n, with n stored in each closure h...

    > ...
    >>>> f[0]()

    > 0
    >>>> f[1]()

    > 1


    to be regurgitated when each is called.

    Terry Jan Reedy
    Terry Reedy, Sep 28, 2008
    #4
  5. On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:

    > Hello all,
    >
    > To me, this is a somewhat unintuitive behavior. I want to discuss the
    > parts of it I don't understand.
    >
    >>>> f= [ None ]* 10
    >>>> for n in range( 10 ):

    > ... f[ n ]= lambda: n
    > ...
    >>>> f[0]()

    > 9
    >>>> f[1]()

    > 9
    >
    > I guess I can accept this part so far, though it took a little getting
    > used to. I'm writing some code and found the following workaround, but
    > I don't think it should give different results. Maybe I'm not
    > understanding some of the details of closures.
    >
    >>>> f= [ None ]* 10
    >>>> for n in range( 10 ):

    > ... f[ n ]= (lambda n: ( lambda: n ) )( n )
    > ...
    >>>> f[0]()

    > 0
    >>>> f[1]()

    > 1
    >
    > Which is of course the desired effect. Why doesn't the second one just
    > look up what 'n' is when I call f[0], and return 9?


    That's an awfully complicated solution. A much easier way to get the
    result you are after is to give each function its own local copy of n:

    f[n] = lambda n=n: n

    As for why the complicated version works, it may be clearer if you expand
    it from a one-liner:

    # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

    inner = lambda: n
    outer = lambda n: inner
    f[n] = outer(n)

    outer(0) => inner with a local scope of n=0
    outer(1) => inner with a local scope of n=1 etc.

    Then, later, when you call inner() it grabs the local scope and returns
    the number you expected.


    --
    Steven
    Steven D'Aprano, Sep 28, 2008
    #5
  6. On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
    cybersource.com.au> wrote:
    > On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:
    > > Hello all,

    >
    > > To me, this is a somewhat unintuitive behavior.  I want to discuss the
    > > parts of it I don't understand.

    >
    > >>>> f= [ None ]* 10
    > >>>> for n in range( 10 ):

    > > ...     f[ n ]= lambda: n
    > > ...
    > >>>> f[0]()

    > > 9
    > >>>> f[1]()

    > > 9

    >
    > > I guess I can accept this part so far, though it took a little getting
    > > used to.  I'm writing some code and found the following workaround, but
    > > I don't think it should give different results.  Maybe I'm not
    > > understanding some of the details of closures.

    >
    > >>>> f= [ None ]* 10
    > >>>> for n in range( 10 ):

    > > ...     f[ n ]= (lambda n: ( lambda: n ) )( n )
    > > ...
    > >>>> f[0]()

    > > 0
    > >>>> f[1]()

    > > 1

    >
    > > Which is of course the desired effect.  Why doesn't the second one just
    > > look up what 'n' is when I call f[0], and return 9?

    >
    > That's an awfully complicated solution. A much easier way to get the
    > result you are after is to give each function its own local copy of n:
    >
    > f[n] = lambda n=n: n
    >
    > As for why the complicated version works, it may be clearer if you expand
    > it from a one-liner:
    >
    > # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
    >
    > inner = lambda: n
    > outer = lambda n: inner
    > f[n] = outer(n)
    >
    > outer(0) => inner with a local scope of n=0
    > outer(1) => inner with a local scope of n=1 etc.
    >
    > Then, later, when you call inner() it grabs the local scope and returns
    > the number you expected.
    >
    > --
    > Steven


    Steven,

    I must have misunderstood. Here's my run of your code:

    >>> inner = lambda: n
    >>> outer = lambda n: inner
    >>> outer(0)

    <function <lambda> at 0x00A01170>
    >>> a=outer(0)
    >>> b=outer(1)
    >>> a()

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 1, in <lambda>
    NameError: global name 'n' is not defined

    Why doesn't 'inner' know it's been used in two different scopes, and
    look up 'n' based on the one it's in?
    Aaron \Castironpi\ Brady, Sep 28, 2008
    #6
  7. Aaron \Castironpi\ Brady

    Terry Reedy Guest

    Aaron "Castironpi" Brady wrote:
    > On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-


    >> As for why the complicated version works, it may be clearer if you expand
    >> it from a one-liner:
    >>
    >> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
    >>
    >> inner = lambda: n
    >> outer = lambda n: inner
    >> f[n] = outer(n)
    >>
    >> outer(0) => inner with a local scope of n=0
    >> outer(1) => inner with a local scope of n=1 etc.


    For this to work, the 'expansion' has to be mental and not actual.
    Which is to say, inner must be a text macro to be substituted back into
    outer.

    >> Then, later, when you call inner() it grabs the local scope and returns
    >> the number you expected.


    >
    > I must have misunderstood. Here's my run of your code:


    I cannot speak to what Steven meant, but

    >>>> inner = lambda: n


    when inner is actually compiled outside of outer, it is no longer a
    closure over outer's 'n' and 'n' will be looked for in globals instead.

    >>>> outer = lambda n: inner
    >>>> outer(0)

    > <function <lambda> at 0x00A01170>
    >>>> a=outer(0)
    >>>> b=outer(1)
    >>>> a()

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > File "<stdin>", line 1, in <lambda>
    > NameError: global name 'n' is not defined
    >
    > Why doesn't 'inner' know it's been used in two different scopes, and
    > look up 'n' based on the one it's in?


    That would be dynamic rather than lexical scoping.
    Terry Reedy, Sep 28, 2008
    #7
  8. On Sun, 28 Sep 2008 17:47:44 -0400, Terry Reedy wrote:

    > Aaron "Castironpi" Brady wrote:
    >> On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

    >
    >>> As for why the complicated version works, it may be clearer if you
    >>> expand it from a one-liner:
    >>>
    >>> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
    >>>
    >>> inner = lambda: n
    >>> outer = lambda n: inner
    >>> f[n] = outer(n)
    >>>
    >>> outer(0) => inner with a local scope of n=0 outer(1) => inner with a
    >>> local scope of n=1 etc.

    >
    > For this to work, the 'expansion' has to be mental and not actual. Which
    > is to say, inner must be a text macro to be substituted back into outer.


    Er, yes, that's what I meant, sorry for not being more explicit. That's
    why it wasn't a copy and paste of actual running code.

    Or perhaps I just confused myself and was talking nonsense.



    --
    Steven
    Steven D'Aprano, Sep 29, 2008
    #8
  9. On Sep 28, 4:47 pm, Terry Reedy <> wrote:
    > Aaron "Castironpi" Brady wrote:
    > > On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
    > >> As for why the complicated version works, it may be clearer if you expand
    > >> it from a one-liner:

    >
    > >> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

    >
    > >> inner = lambda: n
    > >> outer = lambda n: inner
    > >> f[n] = outer(n)

    >
    > >> outer(0) => inner with a local scope of n=0
    > >> outer(1) => inner with a local scope of n=1 etc.

    >
    > For this to work, the 'expansion' has to be mental and not actual.
    > Which is to say, inner must be a text macro to be substituted back into
    > outer.
    >
    > >> Then, later, when you call inner() it grabs the local scope and returns
    > >> the number you expected.

    >
    > > I must have misunderstood.  Here's my run of your code:

    >
    > I cannot speak to what Steven meant, but
    >
    > >>>> inner = lambda: n

    >
    > when inner is actually compiled outside of outer, it is no longer a
    > closure over outer's 'n' and 'n' will be looked for in globals instead.
    >
    > >>>> outer = lambda n: inner
    > >>>> outer(0)

    > > <function <lambda> at 0x00A01170>
    > >>>> a=outer(0)
    > >>>> b=outer(1)
    > >>>> a()

    > > Traceback (most recent call last):
    > >   File "<stdin>", line 1, in <module>
    > >   File "<stdin>", line 1, in <lambda>
    > > NameError: global name 'n' is not defined

    >
    > > Why doesn't 'inner' know it's been used in two different scopes, and
    > > look up 'n' based on the one it's in?

    >
    > That would be dynamic rather than lexical scoping.


    I couldn't find how those apply on the wikipedia website. It says:
    "dynamic scoping can be dangerous and almost no modern languages use
    it", but it sounded like that was what closures use. Or maybe it was
    what 'inner' in Steven's example would use. I'm confused.

    Actually, I'll pick this apart a little bit. See above when I
    suggested 'late' and 'early' functions which control (or simulate)
    different bindings. I get the idea that 'late' bound functions would
    use a dangerous "dynamic scope", but I could be wrong; that's just my
    impression.

    > >> inner = lambda: n
    > >> outer = lambda n: inner
    > >> f[n] = outer(n)

    >
    > >> outer(0) => inner with a local scope of n=0
    > >> outer(1) => inner with a local scope of n=1 etc.


    If you defined these as:

    inner= late( lambda: n )
    outer= lambda n: inner

    You could get the right results. It's not even clear you need
    quotes. Perhaps 'late' could carry the definition of 'n' with it when
    it's returned from 'outer'.

    In my proposal, it makes a copy of the "localest" namespace, at least
    all the variables used below it, then returns its argument in an
    original closure.
    Aaron \Castironpi\ Brady, Sep 29, 2008
    #9
  10. Aaron \Castironpi\ Brady

    Terry Reedy Guest

    Aaron "Castironpi" Brady wrote:
    > On Sep 28, 4:47 pm, Terry Reedy <> wrote:
    >> Aaron "Castironpi" Brady wrote:


    >>>>>> inner = lambda: n

    >> when inner is actually compiled outside of outer, it is no longer a
    >> closure over outer's 'n' and 'n' will be looked for in globals instead.
    >>
    >>>>>> outer = lambda n: inner
    >>>>>> outer(0)
    >>> <function <lambda> at 0x00A01170>
    >>>>>> a=outer(0)
    >>>>>> b=outer(1)
    >>>>>> a()
    >>> Traceback (most recent call last):
    >>> File "<stdin>", line 1, in <module>
    >>> File "<stdin>", line 1, in <lambda>
    >>> NameError: global name 'n' is not defined
    >>> Why doesn't 'inner' know it's been used in two different scopes, and
    >>> look up 'n' based on the one it's in?

    >> That would be dynamic rather than lexical scoping.

    >
    > I couldn't find how those apply on the wikipedia website. It says:
    > "dynamic scoping can be dangerous and almost no modern languages use
    > it", but it sounded like that was what closures use. Or maybe it was
    > what 'inner' in Steven's example would use. I'm confused.


    As I understand it, partly from postings here years ago...

    Lexical: The namespace scope of 'n' in inner is determined by where
    inner is located in the code -- where is is compiled. This is Python
    (and nearly all modern languages). Even without closures, the global
    scope of a function is the module it is defined in.

    Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
    determined by where inner is called from. This is what you seemed to be
    suggesting -- look up 'n' based on the scope it is *used* in.

    Even without closures, dynamic scoping would be if the global scope of a
    function for each call were the module it is called in.

    tjr
    Terry Reedy, Sep 29, 2008
    #10
  11. In message
    <>,
    Aaron "Castironpi" Brady wrote:

    > [Wikipedia] says:
    > "dynamic scoping can be dangerous and almost no modern languages use
    > it", but it sounded like that was what closures use.


    Closures will use whatever the language says they use. LISP used dynamic
    binding, but this caused some interesting problems as mentioned above. Perl
    allows both, depending on whether the local is declared with "local"
    or "my". Python uses only lexical, though there's probably some way to get
    dynamic behaviour if you want. :)
    Lawrence D'Oliveiro, Sep 29, 2008
    #11
  12. Aaron \Castironpi\ Brady

    Paul Boddie Guest

    On 29 Sep, 05:56, Terry Reedy <> wrote:
    >
    > As I understand it, partly from postings here years ago...
    >
    > Lexical: The namespace scope of 'n' in inner is determined by where
    > inner is located in the code -- where is is compiled.  This is Python
    > (and nearly all modern languages).  Even without closures, the global
    > scope of a function is the module it is defined in.


    This is how I understand it, too. The confusing part involves the
    definition of any inner function and how any "external" names are
    bound or defined. As we've seen...

    def f(x):
    def g():
    return x
    x += 1 # added for illustration
    return g

    ....it might look at first glance like the function known as g (within
    f) should return the initial value of x (as known within f), since
    that was the value x had when g was defined. The following is an
    example execution trace based on that mental model:

    fn = f(1)
    -> def f(1):
    -> def g(): # g defined with x as 1
    -> return x # would be 1
    -> x += 1 # x becomes 2
    -> return g
    fn()
    -> def g():
    -> return x # would still be 1

    However, as we know, this isn't the case in real Python since g isn't
    initialised with the value of x at the time of its definition - it
    instead maintains access to the namespace providing x. We must
    therefore revise the example:

    fn = f(1)
    -> def f(1):
    -> def g(): # g refers to x within f(1)
    -> return x # would be the current value of x within f(1)
    -> x += 1 # x becomes 2
    -> return g
    fn()
    -> def g(): # g refers to x within f(1)
    -> return x # would be the current value of x within f(1), which is
    2

    This is the dynamic aspect of closures: values aren't used to
    initialise inner functions; names are looked up from their origin.

    > Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
    > determined by where inner is called from. This is what you seemed to be
    > suggesting -- look up 'n' based on the scope it is *used* in.


    Indeed. Dynamic scoping is confusing in that one has to set up an
    appropriate "environment" for the closure to access so that references
    to names can be meaningfully satisfied; obviously, this creates all
    sorts of encapsulation problems. The confusing aspect of lexical
    scoping, however, especially if non-local names can be changed, is
    that the effects of closures are potentially distant - one doesn't
    always invoke inner functions in the place where they were defined, as
    we see above - and such effects may thus happen within "abandoned
    namespaces" - that is, namespaces which can no longer be revisited and
    used in their original context. So, in the above, once f has been
    invoked, the namespace for that invocation effectively lives on, but
    that namespace is not a general one for f - if we invoke f again, we
    get another namespace as one should reasonably expect.

    A somewhat separate issue is illustrated by the modification of x
    within f. Although for most statements, we would expect the value of x
    to evolve following from a top-to-bottom traversal of the code within
    a unit, function definition statements do not behave like, say, "for",
    "if" or "while" statements. Now although this should be obvious at the
    module global level, I feel that it can be easy to overlook within a
    function where one normally expects to find plain old control-flow
    constructs, expressions, assignments and so on. It is pertinent to
    note, with respect to the original inquiry, that lambda functions are
    subject to the same caveats, and I believe that lexical scoping was
    introduced precisely to make lambda functions less cumbersome to
    employ, eliminating the need to explicitly initialise them using the
    "identity" default parameters trick, but obviously introducing the
    consequences and potential misunderstandings described above.

    Paul
    Paul Boddie, Sep 29, 2008
    #12
  13. On Sep 28, 6:43 am, "Aaron \"Castironpi\" Brady"
    <> wrote:
    > Hello all,
    >
    > To me, this is a somewhat unintuitive behavior.  I want to discuss the
    > parts of it I don't understand.
    >
    > >>> f= [ None ]* 10
    > >>> for n in range( 10 ):

    >
    > ...     f[ n ]= lambda: n
    > ...>>> f[0]()
    > 9
    > >>> f[1]()

    >
    > 9
    >


    You may want to read this old thread that goes in detail explaining
    scoping issues
    with closures in for loops:

    http://groups.google.com/group/comp...en&lnk=gst&q=simionato jacek#93503c5b9c66226e
    Michele Simionato, Sep 29, 2008
    #13
  14. Aaron \Castironpi\ Brady

    Terry Reedy Guest

    Paul Boddie wrote:
    > On 29 Sep, 05:56, Terry Reedy <> wrote:

    ....
    >> Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
    >> determined by where inner is called from. This is what you seemed to be
    >> suggesting -- look up 'n' based on the scope it is *used* in.

    ....

    > A somewhat separate issue is illustrated by the modification of x
    > within f. Although for most statements, we would expect the value of x
    > to evolve following from a top-to-bottom traversal of the code within
    > a unit, function definition statements do not behave like, say, "for",
    > "if" or "while" statements. Now although this should be obvious at the
    > module global level, I feel that it can be easy to overlook within a
    > function where one normally expects to find plain old control-flow
    > constructs, expressions, assignments and so on. It is pertinent to
    > note, with respect to the original inquiry, that lambda functions are
    > subject to the same caveats,


    Please: Python does not have 'lambda functions'. Def statements and
    lambda expressions both define instances of the function class. So this
    amounts to saying "functions are subject to the same caveats as functions."

    > and I believe that lexical scoping was introduced precisely to make
    > lambda functions less cumbersome to
    > employ, eliminating the need to explicitly initialise them using the
    > "identity" default parameters trick, but obviously introducing the
    > consequences and potential misunderstandings described above.


    As I meant when I wrote "Even without closures, the global scope of a
    function is the module it is defined in.", the concept of lexical
    scoping applies even to non-nested top-level functions.

    Yes, closures for nested functions were introduced in part to replace
    the pseudo-default parameter trick (which does not require the confusion
    of using the same name in both local namespaces). But this applies to
    both syntactical forms of function definition and had nothing to do in
    particular with easing the quite limited lambda form, which Guido does
    not like and once thought to delete in 3.0.

    Closures were also introduced because default parameters are limited to
    capturing a value at the time of function definition instead of at
    function invocation. Consider your example which I have moved down to here.

    > This is how I understand it, too. The confusing part involves the
    > definition of any inner function and how any "external" names are
    > bound or defined. As we've seen...
    >
    > def f(x):
    > def g():
    > return x
    > x += 1 # added for illustration
    > return g
    >
    > ...it might look at first glance like the function known as g (within
    > f) should return the initial value of x (as known within f), since
    > that was the value x had when g was defined.


    Only if one is confused about the difference between default parameter
    expressions, which are evaluated when the function is defined, and body
    expressions, which are evaluated when the function is called. If you
    want g to use the 'initial' value that x has when g is defined, then say so.

    def f(x):
    def g(y=x) # or x=x if you really prefer
    return y # or x, with the consequent possibility of confusion
    x += 1
    return g

    If 'x' is fixed, this is a difference without effect. If 'x' is not, as
    in this pair of examples, the difference is important.

    The introduction of the 'nonlocal' keyword will make the latter case
    more common and hence the second reason more important. Consider code like

    def f(*args)
    x = [0]
    def g()
    ...
    x[0] += 1
    ...
    <code calling g some variable number of times>
    return x[0], ...

    This has x fixedly bound to a particular list. The default argument
    trick would have worked here. In the future, this can and will be
    written more naturally as

    def f(*args)
    x = 0
    def g()
    nonlocal x
    ...
    x += 1
    ...
    <code calling g some variable number of times>
    return x, ...

    With x being rebound, the default argument trick would *not* work. And,
    of course, lambda expressions are not in the picture.

    Terry Jan Reedy
    Terry Reedy, Sep 29, 2008
    #14
  15. On Sep 29, 9:14 am, Paul Boddie <> wrote:
    > On 29 Sep, 05:56, Terry Reedy <> wrote:
    >
    >
    >
    > > As I understand it, partly from postings here years ago...

    >
    > > Lexical: The namespace scope of 'n' in inner is determined by where
    > > inner is located in the code -- where is is compiled.  This is Python
    > > (and nearly all modern languages).  Even without closures, the global
    > > scope of a function is the module it is defined in.

    >
    > This is how I understand it, too. The confusing part involves the
    > definition of any inner function and how any "external" names are
    > bound or defined. As we've seen...
    >
    >   def f(x):
    >     def g():
    >       return x
    >     x += 1 # added for illustration
    >     return g
    >
    > ...it might look at first glance like the function known as g (within
    > f) should return the initial value of x (as known within f), since
    > that was the value x had when g was defined. The following is an
    > example execution trace based on that mental model:
    >
    > fn = f(1)
    > -> def f(1):
    > ->   def g(): # g defined with x as 1
    > ->     return x # would be 1
    > ->   x += 1 # x becomes 2
    > ->   return g
    > fn()
    > -> def g():
    > ->   return x # would still be 1
    >
    > However, as we know, this isn't the case in real Python since g isn't
    > initialised with the value of x at the time of its definition - it
    > instead maintains access to the namespace providing x. We must
    > therefore revise the example:
    >
    > fn = f(1)
    > -> def f(1):
    > ->   def g(): # g refers to x within f(1)
    > ->     return x # would be the current value of x within f(1)
    > ->   x += 1 # x becomes 2
    > ->   return g
    > fn()
    > -> def g(): # g refers to x within f(1)
    > ->   return x # would be the current value of x within f(1), which is
    > 2
    >
    > This is the dynamic aspect of closures: values aren't used to
    > initialise inner functions; names are looked up from their origin.
    >
    > > Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
    > > determined by where inner is called from. This is what you seemed to be
    > > suggesting -- look up 'n' based on the scope it is *used* in.

    >
    > Indeed. Dynamic scoping is confusing in that one has to set up an
    > appropriate "environment" for the closure to access so that references
    > to names can be meaningfully satisfied; obviously, this creates all
    > sorts of encapsulation problems. The confusing aspect of lexical
    > scoping, however, especially if non-local names can be changed, is
    > that the effects of closures are potentially distant - one doesn't
    > always invoke inner functions in the place where they were defined, as
    > we see above - and such effects may thus happen within "abandoned
    > namespaces" - that is, namespaces which can no longer be revisited and
    > used in their original context. So, in the above, once f has been
    > invoked, the namespace for that invocation effectively lives on, but
    > that namespace is not a general one for f - if we invoke f again, we
    > get another namespace as one should reasonably expect.
    >
    > A somewhat separate issue is illustrated by the modification of x
    > within f. Although for most statements, we would expect the value of x
    > to evolve following from a top-to-bottom traversal of the code within
    > a unit, function definition statements do not behave like, say, "for",
    > "if" or "while" statements. Now although this should be obvious at the
    > module global level, I feel that it can be easy to overlook within a
    > function where one normally expects to find plain old control-flow
    > constructs, expressions, assignments and so on. It is pertinent to
    > note, with respect to the original inquiry, that lambda functions are
    > subject to the same caveats, and I believe that lexical scoping was
    > introduced precisely to make lambda functions less cumbersome to
    > employ, eliminating the need to explicitly initialise them using the
    > "identity" default parameters trick, but obviously introducing the
    > consequences and potential misunderstandings described above.
    >
    > Paul


    I'm thinking of a comparison to try to see an example.

    I tried this which still surprised me at first.

    >>> g= [0]
    >>> f= lambda: g
    >>> del g
    >>> f()

    NameError: global name 'g' is not defined

    It took a little thinking. The namespace in 'f' is the global
    namespace. They are one in the same dictionary. id( f.namespace ) ==
    id( main.namespace ). f.namespace is main.namespace.

    When f gets a parameter, its namespace is different by one (or two
    when two parameters, etc).

    >>> g=[0]
    >>> f= lambda h: lambda: h
    >>> i= f( g )
    >>> del g
    >>> i()

    [0]

    The statement 'del g' removes 'g' from the module/global namespace,
    but it lives on in the namespace of 'f', or more accurately, 'i'.

    >>> g=[0]
    >>> f= lambda h: lambda: h
    >>> i= f( g )
    >>> j= f( g )
    >>> del g
    >>> i().append(1)
    >>> j()

    [0, 1]

    Here, the namespaces of 'main', 'i', and 'j', all have references to
    'g'. 'i' and 'j' still have them when 'main' loses its.

    The lambda might be confusing. Take a simpler example of a namespace:

    >>> g= [0]
    >>> def h():

    .... class A:
    .... a= g
    .... return A
    ....
    >>> i= h()
    >>> del g
    >>> i.a

    [0]

    Or even:

    >>> g=[0]
    >>> class A:

    .... a= g
    ....
    >>> del g
    >>> A.a

    [0]

    The class is executed immediately so 'A.a' gets a hold of 'g'.
    Function definitions keep a reference to the -identity-, not value, of
    the namespace they were defined in, and parameters are part of that
    namespace.

    >>> def h( a ):

    .... #new namespace in here, new on each call
    .... def i():
    .... return a
    .... return i

    'i' is not defined in the global namespace, it's defined in a brand
    new one that is created on every call of 'h'. 'a' is defined in that,
    so that's what 'a' when 'i' is called refers to.

    To be specific, the namespace 'h' defines only includes the names that
    are different from its enclosing scope. Regardless of what's defined
    below 'h', 'h' only defines one new variable, 'a'. Its value is
    passed in at call-time. 'i' needs to know what that namespace is--
    that's how closures work-- so it saves a reference. That saved
    reference is to a namespace, not a variable though, which
    distinguishes it from the 'class A: a= g' statement that's executes
    immediately. There, 'a' stores the value of g. By contrast, 'i'
    stores 'h's entire namespace.

    In 'class A: a= g', the name 'a' is assigned to the contents of 'g'.
    In 'def i(): return a', 'a' is the value of a look up in a namespace
    by its name.

    >>> def h( a ):

    .... #new namespace in here
    .... def i():
    .... return a
    .... return i
    ....
    >>> g=[0]
    >>> j= h(g)
    >>> hex(id(g))

    '0x9ff440'
    >>> del g
    >>> hex(id(j()))

    '0x9ff440'
    >>> j.func_closure

    (<cell at 0x009FDF50: list object at 0x009FF440>,)

    By the time 'g' is deleted, 'j' has already hijacked a reference to
    it, which lives in the namespace of the 'i' it defined that time
    through. 'j()', originally 'g', and 'j's namespace all refer to the
    same object.

    Variables are keys in a namespace, even if it's an "abandoned
    namespace", adopting the term.
    Aaron \Castironpi\ Brady, Sep 29, 2008
    #15
  16. Aaron \Castironpi\ Brady

    Paul Boddie Guest

    On 29 Sep, 19:26, Terry Reedy <> wrote:
    >
    > Please: Python does not have 'lambda functions'. Def statements and
    > lambda expressions both define instances of the function class. So this
    > amounts to saying "functions are subject to the same caveats as functions."


    I myself am aware of the nature of "lambda expressions", for want of a
    better term, but it's important to emphasise their nature to anyone
    reading who isn't fully aware of what they represent. My closing
    paragraph touches on the issues of readability and programmer
    expectation when I write that 'function definition statements do not
    behave like, say, "for", "if" or "while" statements'. Although this
    may seem obvious, a newcomer might overlook lambda expressions in this
    regard.

    Personally, I'm not a great enthusiast of closures, anyway. Perhaps I
    spent too long writing Java to be able to look at them as being
    anything other than a fairly imprecise way of encapsulating state in a
    callable.

    Paul
    Paul Boddie, Sep 29, 2008
    #16
  17. Aaron \Castironpi\ Brady

    jhermann Guest

    I didn't see this mentioned in the thread yet: the double-lambda is
    unnecessary (and a hack). What you should do when you need early
    binding is... early binding. ;)

    Namely:

    f = [lambda n=n: n for n in range(10)]
    print f[0]()
    print f[1]()

    Note the "n=n", this prints 0 and 1 instead of 9/9.
    jhermann, Oct 1, 2008
    #17
  18. Aaron \Castironpi\ Brady

    Paul Boddie Guest

    On 1 Okt, 12:43, jhermann <> wrote:
    >
    > f = [lambda n=n: n for n in range(10)]
    > print f[0]()
    > print f[1]()
    >
    > Note the "n=n", this prints 0 and 1 instead of 9/9.


    Yes, Terry mentioned this in his response to my first message. Not
    with lambdas, however, but he did state that he didn't believe that
    such a distinction needed to be made clear to everyone.

    Paul
    Paul Boddie, Oct 1, 2008
    #18
  19. On Oct 1, 5:43 am, jhermann <> wrote:
    > I didn't see this mentioned in the thread yet: the double-lambda is
    > unnecessary (and a hack). What you should do when you need early
    > binding is... early binding. ;)
    >
    > Namely:
    >
    > f = [lambda n=n: n for n in range(10)]
    > print f[0]()
    > print f[1]()
    >
    > Note the "n=n", this prints 0 and 1 instead of 9/9.


    Yes it was mentioned earlier. I think its similar. They both create
    ten new namespaces. You could do something like this (I hit a bump
    with eval and globals() when I tried it):

    def early( string ):
    return eval( string, current_namespace )

    f = [early( 'lambda: n' ) for n for n in range(10)]
    print f[0]()
    print f[1]()

    Furthermore, I don't think the binding semantics of the language are
    completely static binding. What does that definition say about
    mutating a value? I think it's ambiguous and there's no obvious use
    case that favors either one.
    Aaron \Castironpi\ Brady, Oct 1, 2008
    #19
  20. Aaron \Castironpi\ Brady

    greg Guest

    jhermann wrote:

    > I didn't see this mentioned in the thread yet: the double-lambda is
    > unnecessary (and a hack).


    Well, the alternative -- abusing default argument values --
    is seen by many to be a hack as well, possibly a worse one.
    It doesn't work in general, e.g. it fails if the function
    needs to be called with a variable number of arguments.

    The double lambda is conceptually more sound in some
    ways, and can be made to work correctly in all cases.

    The root of the problem actually has nothing to do with
    lambdas or static vs. non-static scoping. It's the fact
    that Python's for-loop doesn't create a new environment
    for the loop variable each time around, but re-uses a
    slot in the containing environment.

    Contrast this with Scheme, where the equivalent of a
    for-loop *does* create a new environment for each
    value of the loop variable. Effectively it's using a
    double lambda, except that one of the lambdas is
    folded into the syntax of the loop, so you don't
    notice it.

    So if anything were to be done to the language to
    fix this, it really should be focused on fixing the
    semantics of the for-loop. Unfortunately, the
    fact that the loop variable leaks out of the scope
    of the loop is regarded as a feature, so anything
    which changes that seems to be a non-starter.

    --
    Greg
    greg, Oct 3, 2008
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matthew Wilson
    Replies:
    6
    Views:
    286
    Bjoern Schliessmann
    Sep 27, 2007
  2. Fernando Perez

    Confused about closures and scoping rules

    Fernando Perez, Nov 7, 2007, in forum: Python
    Replies:
    2
    Views:
    302
    Fernando Perez
    Nov 7, 2007
  3. Chris Mellon
    Replies:
    4
    Views:
    278
    Jakub Hegenbart
    Nov 8, 2007
  4. Alejandro Dubrovsky

    exec and closures

    Alejandro Dubrovsky, Feb 21, 2008, in forum: Python
    Replies:
    2
    Views:
    257
  5. Vivek Nallur

    value binding and function binding

    Vivek Nallur, Sep 25, 2003, in forum: Ruby
    Replies:
    0
    Views:
    122
    Vivek Nallur
    Sep 25, 2003
Loading...

Share This Page