# closures and dynamic binding

Discussion in 'Python' started by Aaron \Castironpi\ Brady, Sep 28, 2008.

Hello all,

To me, this is a somewhat unintuitive behavior. I want to discuss the
parts of it I don't understand.

>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= lambda: n
....
>>> f[0]()

9
>>> f[1]()

9

I guess I can accept this part so far, though it took a little getting
used to. I'm writing some code and found the following workaround,
but I don't think it should give different results. Maybe I'm not
understanding some of the details of closures.

>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= (lambda n: ( lambda: n ) )( n )
....
>>> f[0]()

0
>>> f[1]()

1

Which is of course the desired effect. Why doesn't the second one
just look up what 'n' is when I call f[0], and return 9?

Aaron \Castironpi\ Brady, Sep 28, 2008

2. ### Marc 'BlackJack' RintschGuest

On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:

> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= lambda: n
> ...
>>>> f[0]()

> 9
>>>> f[1]()

> 9

`n` is looked up at the time ``f[0]`` is called. At that time it is
bound to 9.

>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= (lambda n: ( lambda: n ) )( n ) ...
>>>> f[0]()

> 0
>>>> f[1]()

> 1
>
> Which is of course the desired effect. Why doesn't the second one just
> look up what 'n' is when I call f[0], and return 9?

It *does* look up `n` at the time you call ``f[0]`` but this time it's
the local `n` of the outer ``lambda`` function and that is bound to
whatever the function was called with. At the time it was called the
global `n` was bound to 0. Maybe it get's more clear if you don't name
it `n`:

In [167]: f = [None] * 10

In [168]: for n in xrange(10):
.....: f[n] = (lambda x: lambda: x)(n)
.....:

In [169]: f[0]()
Out[169]: 0

Ciao,
Marc 'BlackJack' Rintsch

Marc 'BlackJack' Rintsch, Sep 28, 2008

On Sep 28, 1:14 am, Marc 'BlackJack' Rintsch <> wrote:
> On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:
> > To me, this is a somewhat unintuitive behavior.  I want to discuss the
> > parts of it I don't understand.

>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ...     f[ n ]= lambda: n
> > ...
> >>>> f[0]()

> > 9
> >>>> f[1]()

> > 9

>
> `n` is looked up at the time ``f[0]`` is called.  At that time it is
> bound to 9.
>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ...     f[ n ]= (lambda n: ( lambda: n ) )( n ) ...
> >>>> f[0]()

> > 0
> >>>> f[1]()

> > 1

>
> > Which is of course the desired effect.  Why doesn't the second one just
> > look up what 'n' is when I call f[0], and return 9?

>
> It *does* look up `n` at the time you call ``f[0]`` but this time it's
> the local `n` of the outer ``lambda`` function and that is bound to
> whatever the function was called with.  At the time it was called the
> global `n` was bound to 0.  Maybe it get's more clear if you don't name
> it `n`:
>
> In [167]: f = [None] * 10
>
> In [168]: for n in xrange(10):
>    .....:     f[n] = (lambda x: lambda: x)(n)
>    .....:
>
> In [169]: f[0]()
> Out[169]: 0
>
> Ciao,
>         Marc 'BlackJack' Rintsch

Hi Marc,

It's my understanding that 'n' gets a new value every time through the
loop. n= 0 on the first pass, n= 1 on the second pass, and so on. n=
9 by the end, and that's why `lambda: n` always returns 9. It queries
the variable 'n', and finds 9. (This got lengthy. I started thinking
aloud.)

In your version of the indirect example, it queries the variable 'x',
which it finds in a new distinct scope in each f element. In f[0], x=
0. In f[1], x= 1. There are 10 different 'x' variables throughout
the contents of f. The direct example does not do this allocation of
ten different 'x's.

It's sort of helping. I think I feel like the following is more like
what I'm looking for:

(Non-standard)
>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= n
....
>>> f[0]

9
>>> f[1]

9

because when you access f[0], it looks up the variable 'n'. Obviously
not.

(Unproduced)
>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= late( n ) #late/lazy
....
>>> f[0]

9
>>> f[1]

9

>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= early( n ) #early/eager
....
>>> f[0]

0
>>> f[1]

1

For the functions, I want a function that returns 'n', either late or
early.

(Unproduced)
>>> for n in range( 10 ):

.... f[ n ]= lambda: late( n )
>>> f[0]()

9
>>> for n in range( 10 ):

.... f[ n ]= lambda: early( n )
>>> f[0]()

0

I don't think you could pull this off. 'late' and 'early' succeed
with quotes around n, early('n') and late('n'), in the direct
assignments, but the functions aren't so lucky. 'n' has gone on to a
better world by the time 'early' gets any control.

This might have some success.

(Unproduced)
>>> for n in range( 10 ):

.... f[ n ]= late( lambda: n )
>>> f[0]()

9
>>> for n in range( 10 ):

.... f[ n ]= early( lambda: n )
>>> f[0]()

0

Though it's beyond my foresight to tell if it's feasible as stated, if
you need quotes, how much introspection you would need, etc. Plus,
'late' and 'early' were accepting quoted parameters earlier. How
would they look inside a function? Could a simple decorator provide
the service?

On a tangent about mutables, it's not that numbers, strings, and
tuples are 'immutable' per se, it's just that they don't have any
methods which mutate them (or assignable properties). Lists and
dictionaries do. It's up to the user whether a custom class does.

Aaron \Castironpi\ Brady, Sep 28, 2008
4. ### Terry ReedyGuest

> Hello all,
>
> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= lambda: n

This is equivalent to

for n in range(10):
def g(): return n
f[n] = g

which is equivalent to

def g(): return n
f = [g]*10
n = 9

>>>> f[0]()

> 9
>>>> f[1]()

> 9

which make this not so surprising as the original lambda version is to
some people.

> I guess I can accept this part so far, though it took a little getting
> used to. I'm writing some code and found the following workaround,
> but I don't think it should give different results. Maybe I'm not
> understanding some of the details of closures.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= (lambda n: ( lambda: n ) )( n )

This is equivalent to

for n in range(10):
def g(n):
def h:
return n
return h
f[n] = g(n)

Now, to avoid the needless confusion of 'n's, g is equivalent to

def g(x):
def h:
return x
return h

(One could do the same change in the lambdas, too, of course).
so that g(n)() == n, with n stored in each closure h...

> ...
>>>> f[0]()

> 0
>>>> f[1]()

> 1

to be regurgitated when each is called.

Terry Jan Reedy

Terry Reedy, Sep 28, 2008
5. ### Steven D'ApranoGuest

On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:

> Hello all,
>
> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= lambda: n
> ...
>>>> f[0]()

> 9
>>>> f[1]()

> 9
>
> I guess I can accept this part so far, though it took a little getting
> used to. I'm writing some code and found the following workaround, but
> I don't think it should give different results. Maybe I'm not
> understanding some of the details of closures.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= (lambda n: ( lambda: n ) )( n )
> ...
>>>> f[0]()

> 0
>>>> f[1]()

> 1
>
> Which is of course the desired effect. Why doesn't the second one just
> look up what 'n' is when I call f[0], and return 9?

That's an awfully complicated solution. A much easier way to get the
result you are after is to give each function its own local copy of n:

f[n] = lambda n=n: n

As for why the complicated version works, it may be clearer if you expand
it from a one-liner:

# expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

inner = lambda: n
outer = lambda n: inner
f[n] = outer(n)

outer(0) => inner with a local scope of n=0
outer(1) => inner with a local scope of n=1 etc.

Then, later, when you call inner() it grabs the local scope and returns
the number you expected.

--
Steven

Steven D'Aprano, Sep 28, 2008

On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:
> On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:
> > Hello all,

>
> > To me, this is a somewhat unintuitive behavior.  I want to discuss the
> > parts of it I don't understand.

>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ...     f[ n ]= lambda: n
> > ...
> >>>> f[0]()

> > 9
> >>>> f[1]()

> > 9

>
> > I guess I can accept this part so far, though it took a little getting
> > used to.  I'm writing some code and found the following workaround, but
> > I don't think it should give different results.  Maybe I'm not
> > understanding some of the details of closures.

>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ...     f[ n ]= (lambda n: ( lambda: n ) )( n )
> > ...
> >>>> f[0]()

> > 0
> >>>> f[1]()

> > 1

>
> > Which is of course the desired effect.  Why doesn't the second one just
> > look up what 'n' is when I call f[0], and return 9?

>
> That's an awfully complicated solution. A much easier way to get the
> result you are after is to give each function its own local copy of n:
>
> f[n] = lambda n=n: n
>
> As for why the complicated version works, it may be clearer if you expand
> it from a one-liner:
>
> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>
> inner = lambda: n
> outer = lambda n: inner
> f[n] = outer(n)
>
> outer(0) => inner with a local scope of n=0
> outer(1) => inner with a local scope of n=1 etc.
>
> Then, later, when you call inner() it grabs the local scope and returns
> the number you expected.
>
> --
> Steven

Steven,

I must have misunderstood. Here's my run of your code:

>>> inner = lambda: n
>>> outer = lambda n: inner
>>> outer(0)

<function <lambda> at 0x00A01170>
>>> a=outer(0)
>>> b=outer(1)
>>> a()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
NameError: global name 'n' is not defined

Why doesn't 'inner' know it's been used in two different scopes, and
look up 'n' based on the one it's in?

Aaron \Castironpi\ Brady, Sep 28, 2008
7. ### Terry ReedyGuest

> On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

>> As for why the complicated version works, it may be clearer if you expand
>> it from a one-liner:
>>
>> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>
>> inner = lambda: n
>> outer = lambda n: inner
>> f[n] = outer(n)
>>
>> outer(0) => inner with a local scope of n=0
>> outer(1) => inner with a local scope of n=1 etc.

For this to work, the 'expansion' has to be mental and not actual.
Which is to say, inner must be a text macro to be substituted back into
outer.

>> Then, later, when you call inner() it grabs the local scope and returns
>> the number you expected.

>
> I must have misunderstood. Here's my run of your code:

I cannot speak to what Steven meant, but

>>>> inner = lambda: n

when inner is actually compiled outside of outer, it is no longer a
closure over outer's 'n' and 'n' will be looked for in globals instead.

>>>> outer = lambda n: inner
>>>> outer(0)

> <function <lambda> at 0x00A01170>
>>>> a=outer(0)
>>>> b=outer(1)
>>>> a()

> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "<stdin>", line 1, in <lambda>
> NameError: global name 'n' is not defined
>
> Why doesn't 'inner' know it's been used in two different scopes, and
> look up 'n' based on the one it's in?

That would be dynamic rather than lexical scoping.

Terry Reedy, Sep 28, 2008
8. ### Steven D'ApranoGuest

On Sun, 28 Sep 2008 17:47:44 -0400, Terry Reedy wrote:

>> On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

>
>>> As for why the complicated version works, it may be clearer if you
>>> expand it from a one-liner:
>>>
>>> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>>
>>> inner = lambda: n
>>> outer = lambda n: inner
>>> f[n] = outer(n)
>>>
>>> outer(0) => inner with a local scope of n=0 outer(1) => inner with a
>>> local scope of n=1 etc.

>
> For this to work, the 'expansion' has to be mental and not actual. Which
> is to say, inner must be a text macro to be substituted back into outer.

Er, yes, that's what I meant, sorry for not being more explicit. That's
why it wasn't a copy and paste of actual running code.

Or perhaps I just confused myself and was talking nonsense.

--
Steven

Steven D'Aprano, Sep 29, 2008

On Sep 28, 4:47 pm, Terry Reedy <> wrote:
> > On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
> >> As for why the complicated version works, it may be clearer if you expand
> >> it from a one-liner:

>
> >> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

>
> >> inner = lambda: n
> >> outer = lambda n: inner
> >> f[n] = outer(n)

>
> >> outer(0) => inner with a local scope of n=0
> >> outer(1) => inner with a local scope of n=1 etc.

>
> For this to work, the 'expansion' has to be mental and not actual.
> Which is to say, inner must be a text macro to be substituted back into
> outer.
>
> >> Then, later, when you call inner() it grabs the local scope and returns
> >> the number you expected.

>
> > I must have misunderstood.  Here's my run of your code:

>
> I cannot speak to what Steven meant, but
>
> >>>> inner = lambda: n

>
> when inner is actually compiled outside of outer, it is no longer a
> closure over outer's 'n' and 'n' will be looked for in globals instead.
>
> >>>> outer = lambda n: inner
> >>>> outer(0)

> > <function <lambda> at 0x00A01170>
> >>>> a=outer(0)
> >>>> b=outer(1)
> >>>> a()

> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File "<stdin>", line 1, in <lambda>
> > NameError: global name 'n' is not defined

>
> > Why doesn't 'inner' know it's been used in two different scopes, and
> > look up 'n' based on the one it's in?

>
> That would be dynamic rather than lexical scoping.

I couldn't find how those apply on the wikipedia website. It says:
"dynamic scoping can be dangerous and almost no modern languages use
it", but it sounded like that was what closures use. Or maybe it was
what 'inner' in Steven's example would use. I'm confused.

Actually, I'll pick this apart a little bit. See above when I
suggested 'late' and 'early' functions which control (or simulate)
different bindings. I get the idea that 'late' bound functions would
use a dangerous "dynamic scope", but I could be wrong; that's just my
impression.

> >> inner = lambda: n
> >> outer = lambda n: inner
> >> f[n] = outer(n)

>
> >> outer(0) => inner with a local scope of n=0
> >> outer(1) => inner with a local scope of n=1 etc.

If you defined these as:

inner= late( lambda: n )
outer= lambda n: inner

You could get the right results. It's not even clear you need
quotes. Perhaps 'late' could carry the definition of 'n' with it when
it's returned from 'outer'.

In my proposal, it makes a copy of the "localest" namespace, at least
all the variables used below it, then returns its argument in an
original closure.

Aaron \Castironpi\ Brady, Sep 29, 2008
10. ### Terry ReedyGuest

> On Sep 28, 4:47 pm, Terry Reedy <> wrote:

>>>>>> inner = lambda: n

>> when inner is actually compiled outside of outer, it is no longer a
>> closure over outer's 'n' and 'n' will be looked for in globals instead.
>>
>>>>>> outer = lambda n: inner
>>>>>> outer(0)
>>> <function <lambda> at 0x00A01170>
>>>>>> a=outer(0)
>>>>>> b=outer(1)
>>>>>> a()
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> File "<stdin>", line 1, in <lambda>
>>> NameError: global name 'n' is not defined
>>> Why doesn't 'inner' know it's been used in two different scopes, and
>>> look up 'n' based on the one it's in?

>> That would be dynamic rather than lexical scoping.

>
> I couldn't find how those apply on the wikipedia website. It says:
> "dynamic scoping can be dangerous and almost no modern languages use
> it", but it sounded like that was what closures use. Or maybe it was
> what 'inner' in Steven's example would use. I'm confused.

As I understand it, partly from postings here years ago...

Lexical: The namespace scope of 'n' in inner is determined by where
inner is located in the code -- where is is compiled. This is Python
(and nearly all modern languages). Even without closures, the global
scope of a function is the module it is defined in.

Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
determined by where inner is called from. This is what you seemed to be
suggesting -- look up 'n' based on the scope it is *used* in.

Even without closures, dynamic scoping would be if the global scope of a
function for each call were the module it is called in.

tjr

Terry Reedy, Sep 29, 2008
11. ### Lawrence D'OliveiroGuest

In message
<>,

> [Wikipedia] says:
> "dynamic scoping can be dangerous and almost no modern languages use
> it", but it sounded like that was what closures use.

Closures will use whatever the language says they use. LISP used dynamic
binding, but this caused some interesting problems as mentioned above. Perl
allows both, depending on whether the local is declared with "local"
or "my". Python uses only lexical, though there's probably some way to get
dynamic behaviour if you want.

Lawrence D'Oliveiro, Sep 29, 2008
12. ### Paul BoddieGuest

On 29 Sep, 05:56, Terry Reedy <> wrote:
>
> As I understand it, partly from postings here years ago...
>
> Lexical: The namespace scope of 'n' in inner is determined by where
> inner is located in the code -- where is is compiled.  This is Python
> (and nearly all modern languages).  Even without closures, the global
> scope of a function is the module it is defined in.

This is how I understand it, too. The confusing part involves the
definition of any inner function and how any "external" names are
bound or defined. As we've seen...

def f(x):
def g():
return x
x += 1 # added for illustration
return g

....it might look at first glance like the function known as g (within
f) should return the initial value of x (as known within f), since
that was the value x had when g was defined. The following is an
example execution trace based on that mental model:

fn = f(1)
-> def f(1):
-> def g(): # g defined with x as 1
-> return x # would be 1
-> x += 1 # x becomes 2
-> return g
fn()
-> def g():
-> return x # would still be 1

However, as we know, this isn't the case in real Python since g isn't
initialised with the value of x at the time of its definition - it
therefore revise the example:

fn = f(1)
-> def f(1):
-> def g(): # g refers to x within f(1)
-> return x # would be the current value of x within f(1)
-> x += 1 # x becomes 2
-> return g
fn()
-> def g(): # g refers to x within f(1)
-> return x # would be the current value of x within f(1), which is
2

This is the dynamic aspect of closures: values aren't used to
initialise inner functions; names are looked up from their origin.

> Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
> determined by where inner is called from. This is what you seemed to be
> suggesting -- look up 'n' based on the scope it is *used* in.

Indeed. Dynamic scoping is confusing in that one has to set up an
appropriate "environment" for the closure to access so that references
to names can be meaningfully satisfied; obviously, this creates all
sorts of encapsulation problems. The confusing aspect of lexical
scoping, however, especially if non-local names can be changed, is
that the effects of closures are potentially distant - one doesn't
always invoke inner functions in the place where they were defined, as
we see above - and such effects may thus happen within "abandoned
namespaces" - that is, namespaces which can no longer be revisited and
used in their original context. So, in the above, once f has been
invoked, the namespace for that invocation effectively lives on, but
that namespace is not a general one for f - if we invoke f again, we
get another namespace as one should reasonably expect.

A somewhat separate issue is illustrated by the modification of x
within f. Although for most statements, we would expect the value of x
to evolve following from a top-to-bottom traversal of the code within
a unit, function definition statements do not behave like, say, "for",
"if" or "while" statements. Now although this should be obvious at the
module global level, I feel that it can be easy to overlook within a
function where one normally expects to find plain old control-flow
constructs, expressions, assignments and so on. It is pertinent to
note, with respect to the original inquiry, that lambda functions are
subject to the same caveats, and I believe that lexical scoping was
introduced precisely to make lambda functions less cumbersome to
employ, eliminating the need to explicitly initialise them using the
"identity" default parameters trick, but obviously introducing the
consequences and potential misunderstandings described above.

Paul

Paul Boddie, Sep 29, 2008
13. ### Michele SimionatoGuest

On Sep 28, 6:43 am, "Aaron \"Castironpi\" Brady"
<> wrote:
> Hello all,
>
> To me, this is a somewhat unintuitive behavior.  I want to discuss the
> parts of it I don't understand.
>
> >>> f= [ None ]* 10
> >>> for n in range( 10 ):

>
> ...     f[ n ]= lambda: n
> ...>>> f[0]()
> 9
> >>> f[1]()

>
> 9
>

You may want to read this old thread that goes in detail explaining
scoping issues
with closures in for loops:

Michele Simionato, Sep 29, 2008
14. ### Terry ReedyGuest

Paul Boddie wrote:
> On 29 Sep, 05:56, Terry Reedy <> wrote:

....
>> Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
>> determined by where inner is called from. This is what you seemed to be
>> suggesting -- look up 'n' based on the scope it is *used* in.

....

> A somewhat separate issue is illustrated by the modification of x
> within f. Although for most statements, we would expect the value of x
> to evolve following from a top-to-bottom traversal of the code within
> a unit, function definition statements do not behave like, say, "for",
> "if" or "while" statements. Now although this should be obvious at the
> module global level, I feel that it can be easy to overlook within a
> function where one normally expects to find plain old control-flow
> constructs, expressions, assignments and so on. It is pertinent to
> note, with respect to the original inquiry, that lambda functions are
> subject to the same caveats,

Please: Python does not have 'lambda functions'. Def statements and
lambda expressions both define instances of the function class. So this
amounts to saying "functions are subject to the same caveats as functions."

> and I believe that lexical scoping was introduced precisely to make
> lambda functions less cumbersome to
> employ, eliminating the need to explicitly initialise them using the
> "identity" default parameters trick, but obviously introducing the
> consequences and potential misunderstandings described above.

As I meant when I wrote "Even without closures, the global scope of a
function is the module it is defined in.", the concept of lexical
scoping applies even to non-nested top-level functions.

Yes, closures for nested functions were introduced in part to replace
the pseudo-default parameter trick (which does not require the confusion
of using the same name in both local namespaces). But this applies to
both syntactical forms of function definition and had nothing to do in
particular with easing the quite limited lambda form, which Guido does
not like and once thought to delete in 3.0.

Closures were also introduced because default parameters are limited to
capturing a value at the time of function definition instead of at
function invocation. Consider your example which I have moved down to here.

> This is how I understand it, too. The confusing part involves the
> definition of any inner function and how any "external" names are
> bound or defined. As we've seen...
>
> def f(x):
> def g():
> return x
> x += 1 # added for illustration
> return g
>
> ...it might look at first glance like the function known as g (within
> f) should return the initial value of x (as known within f), since
> that was the value x had when g was defined.

Only if one is confused about the difference between default parameter
expressions, which are evaluated when the function is defined, and body
expressions, which are evaluated when the function is called. If you
want g to use the 'initial' value that x has when g is defined, then say so.

def f(x):
def g(y=x) # or x=x if you really prefer
return y # or x, with the consequent possibility of confusion
x += 1
return g

If 'x' is fixed, this is a difference without effect. If 'x' is not, as
in this pair of examples, the difference is important.

The introduction of the 'nonlocal' keyword will make the latter case
more common and hence the second reason more important. Consider code like

def f(*args)
x = [0]
def g()
...
x[0] += 1
...
<code calling g some variable number of times>
return x[0], ...

This has x fixedly bound to a particular list. The default argument
trick would have worked here. In the future, this can and will be
written more naturally as

def f(*args)
x = 0
def g()
nonlocal x
...
x += 1
...
<code calling g some variable number of times>
return x, ...

With x being rebound, the default argument trick would *not* work. And,
of course, lambda expressions are not in the picture.

Terry Jan Reedy

Terry Reedy, Sep 29, 2008

On Sep 29, 9:14 am, Paul Boddie <> wrote:
> On 29 Sep, 05:56, Terry Reedy <> wrote:
>
>
>
> > As I understand it, partly from postings here years ago...

>
> > Lexical: The namespace scope of 'n' in inner is determined by where
> > inner is located in the code -- where is is compiled.  This is Python
> > (and nearly all modern languages).  Even without closures, the global
> > scope of a function is the module it is defined in.

>
> This is how I understand it, too. The confusing part involves the
> definition of any inner function and how any "external" names are
> bound or defined. As we've seen...
>
>   def f(x):
>     def g():
>       return x
>     x += 1 # added for illustration
>     return g
>
> ...it might look at first glance like the function known as g (within
> f) should return the initial value of x (as known within f), since
> that was the value x had when g was defined. The following is an
> example execution trace based on that mental model:
>
> fn = f(1)
> -> def f(1):
> ->   def g(): # g defined with x as 1
> ->     return x # would be 1
> ->   x += 1 # x becomes 2
> ->   return g
> fn()
> -> def g():
> ->   return x # would still be 1
>
> However, as we know, this isn't the case in real Python since g isn't
> initialised with the value of x at the time of its definition - it
> therefore revise the example:
>
> fn = f(1)
> -> def f(1):
> ->   def g(): # g refers to x within f(1)
> ->     return x # would be the current value of x within f(1)
> ->   x += 1 # x becomes 2
> ->   return g
> fn()
> -> def g(): # g refers to x within f(1)
> ->   return x # would be the current value of x within f(1), which is
> 2
>
> This is the dynamic aspect of closures: values aren't used to
> initialise inner functions; names are looked up from their origin.
>
> > Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
> > determined by where inner is called from. This is what you seemed to be
> > suggesting -- look up 'n' based on the scope it is *used* in.

>
> Indeed. Dynamic scoping is confusing in that one has to set up an
> appropriate "environment" for the closure to access so that references
> to names can be meaningfully satisfied; obviously, this creates all
> sorts of encapsulation problems. The confusing aspect of lexical
> scoping, however, especially if non-local names can be changed, is
> that the effects of closures are potentially distant - one doesn't
> always invoke inner functions in the place where they were defined, as
> we see above - and such effects may thus happen within "abandoned
> namespaces" - that is, namespaces which can no longer be revisited and
> used in their original context. So, in the above, once f has been
> invoked, the namespace for that invocation effectively lives on, but
> that namespace is not a general one for f - if we invoke f again, we
> get another namespace as one should reasonably expect.
>
> A somewhat separate issue is illustrated by the modification of x
> within f. Although for most statements, we would expect the value of x
> to evolve following from a top-to-bottom traversal of the code within
> a unit, function definition statements do not behave like, say, "for",
> "if" or "while" statements. Now although this should be obvious at the
> module global level, I feel that it can be easy to overlook within a
> function where one normally expects to find plain old control-flow
> constructs, expressions, assignments and so on. It is pertinent to
> note, with respect to the original inquiry, that lambda functions are
> subject to the same caveats, and I believe that lexical scoping was
> introduced precisely to make lambda functions less cumbersome to
> employ, eliminating the need to explicitly initialise them using the
> "identity" default parameters trick, but obviously introducing the
> consequences and potential misunderstandings described above.
>
> Paul

I'm thinking of a comparison to try to see an example.

I tried this which still surprised me at first.

>>> g= [0]
>>> f= lambda: g
>>> del g
>>> f()

NameError: global name 'g' is not defined

It took a little thinking. The namespace in 'f' is the global
namespace. They are one in the same dictionary. id( f.namespace ) ==
id( main.namespace ). f.namespace is main.namespace.

When f gets a parameter, its namespace is different by one (or two
when two parameters, etc).

>>> g=[0]
>>> f= lambda h: lambda: h
>>> i= f( g )
>>> del g
>>> i()

[0]

The statement 'del g' removes 'g' from the module/global namespace,
but it lives on in the namespace of 'f', or more accurately, 'i'.

>>> g=[0]
>>> f= lambda h: lambda: h
>>> i= f( g )
>>> j= f( g )
>>> del g
>>> i().append(1)
>>> j()

[0, 1]

Here, the namespaces of 'main', 'i', and 'j', all have references to
'g'. 'i' and 'j' still have them when 'main' loses its.

The lambda might be confusing. Take a simpler example of a namespace:

>>> g= [0]
>>> def h():

.... class A:
.... a= g
.... return A
....
>>> i= h()
>>> del g
>>> i.a

[0]

Or even:

>>> g=[0]
>>> class A:

.... a= g
....
>>> del g
>>> A.a

[0]

The class is executed immediately so 'A.a' gets a hold of 'g'.
Function definitions keep a reference to the -identity-, not value, of
the namespace they were defined in, and parameters are part of that
namespace.

>>> def h( a ):

.... #new namespace in here, new on each call
.... def i():
.... return a
.... return i

'i' is not defined in the global namespace, it's defined in a brand
new one that is created on every call of 'h'. 'a' is defined in that,
so that's what 'a' when 'i' is called refers to.

To be specific, the namespace 'h' defines only includes the names that
are different from its enclosing scope. Regardless of what's defined
below 'h', 'h' only defines one new variable, 'a'. Its value is
passed in at call-time. 'i' needs to know what that namespace is--
that's how closures work-- so it saves a reference. That saved
reference is to a namespace, not a variable though, which
distinguishes it from the 'class A: a= g' statement that's executes
immediately. There, 'a' stores the value of g. By contrast, 'i'
stores 'h's entire namespace.

In 'class A: a= g', the name 'a' is assigned to the contents of 'g'.
In 'def i(): return a', 'a' is the value of a look up in a namespace
by its name.

>>> def h( a ):

.... #new namespace in here
.... def i():
.... return a
.... return i
....
>>> g=[0]
>>> j= h(g)
>>> hex(id(g))

'0x9ff440'
>>> del g
>>> hex(id(j()))

'0x9ff440'
>>> j.func_closure

(<cell at 0x009FDF50: list object at 0x009FF440>,)

By the time 'g' is deleted, 'j' has already hijacked a reference to
it, which lives in the namespace of the 'i' it defined that time
through. 'j()', originally 'g', and 'j's namespace all refer to the
same object.

Variables are keys in a namespace, even if it's an "abandoned

Aaron \Castironpi\ Brady, Sep 29, 2008
16. ### Paul BoddieGuest

On 29 Sep, 19:26, Terry Reedy <> wrote:
>
> Please: Python does not have 'lambda functions'. Def statements and
> lambda expressions both define instances of the function class. So this
> amounts to saying "functions are subject to the same caveats as functions."

I myself am aware of the nature of "lambda expressions", for want of a
better term, but it's important to emphasise their nature to anyone
reading who isn't fully aware of what they represent. My closing
paragraph touches on the issues of readability and programmer
expectation when I write that 'function definition statements do not
behave like, say, "for", "if" or "while" statements'. Although this
may seem obvious, a newcomer might overlook lambda expressions in this
regard.

Personally, I'm not a great enthusiast of closures, anyway. Perhaps I
spent too long writing Java to be able to look at them as being
anything other than a fairly imprecise way of encapsulating state in a
callable.

Paul

Paul Boddie, Sep 29, 2008
17. ### jhermannGuest

I didn't see this mentioned in the thread yet: the double-lambda is
unnecessary (and a hack). What you should do when you need early
binding is... early binding.

Namely:

f = [lambda n=n: n for n in range(10)]
print f[0]()
print f[1]()

Note the "n=n", this prints 0 and 1 instead of 9/9.

jhermann, Oct 1, 2008
18. ### Paul BoddieGuest

On 1 Okt, 12:43, jhermann <> wrote:
>
> f = [lambda n=n: n for n in range(10)]
> print f[0]()
> print f[1]()
>
> Note the "n=n", this prints 0 and 1 instead of 9/9.

Yes, Terry mentioned this in his response to my first message. Not
with lambdas, however, but he did state that he didn't believe that
such a distinction needed to be made clear to everyone.

Paul

Paul Boddie, Oct 1, 2008

On Oct 1, 5:43 am, jhermann <> wrote:
> I didn't see this mentioned in the thread yet: the double-lambda is
> unnecessary (and a hack). What you should do when you need early
> binding is... early binding.
>
> Namely:
>
> f = [lambda n=n: n for n in range(10)]
> print f[0]()
> print f[1]()
>
> Note the "n=n", this prints 0 and 1 instead of 9/9.

Yes it was mentioned earlier. I think its similar. They both create
ten new namespaces. You could do something like this (I hit a bump
with eval and globals() when I tried it):

def early( string ):
return eval( string, current_namespace )

f = [early( 'lambda: n' ) for n for n in range(10)]
print f[0]()
print f[1]()

Furthermore, I don't think the binding semantics of the language are
completely static binding. What does that definition say about
mutating a value? I think it's ambiguous and there's no obvious use
case that favors either one.

Aaron \Castironpi\ Brady, Oct 1, 2008
20. ### gregGuest

jhermann wrote:

> I didn't see this mentioned in the thread yet: the double-lambda is
> unnecessary (and a hack).

Well, the alternative -- abusing default argument values --
is seen by many to be a hack as well, possibly a worse one.
It doesn't work in general, e.g. it fails if the function
needs to be called with a variable number of arguments.

The double lambda is conceptually more sound in some
ways, and can be made to work correctly in all cases.

The root of the problem actually has nothing to do with
lambdas or static vs. non-static scoping. It's the fact
that Python's for-loop doesn't create a new environment
for the loop variable each time around, but re-uses a
slot in the containing environment.

Contrast this with Scheme, where the equivalent of a
for-loop *does* create a new environment for each
value of the loop variable. Effectively it's using a
double lambda, except that one of the lambdas is
folded into the syntax of the loop, so you don't
notice it.

So if anything were to be done to the language to
fix this, it really should be focused on fixing the
semantics of the for-loop. Unfortunately, the
fact that the loop variable leaks out of the scope
of the loop is regarded as a feature, so anything
which changes that seems to be a non-starter.

--
Greg

greg, Oct 3, 2008