Lambda forms and scoping

[Márcio Faustino]

Hi,

Executing the example below doesn't produce the expected behavior, but
using the commented code does. Is this normal, or is it a problem with
Python? I've tested it with version 2.6.1 on Windows XP.

Thanks,

--

from abc import *
from types import *
import re

class Base (ObjectType):
__metaclass__ = ABCMeta

def __init__(self):
for option in self.get_options().keys():
method = 'get_%s_option' % re.sub(' ', '_', option.lower
())
setattr(self.__class__, method, lambda self:
self.get_option(option))

#def create_method(option):
# method = 'get_%s_option' % re.sub(' ', '_', option.lower
())
# setattr(self.__class__, method, lambda self:
self.get_option(option))
#
#map(create_method, self.get_options().keys())

@abstractmethod
def get_options(self):
raise NotImplementedError()

def get_option(self, option):
return self.get_options()[option]

class Derived (Base):
def get_options(self):
return {
'Message': 'Hello world!',
'Web site': 'http://www.example.com',
}

object = Derived()
print object.get_message_option()
print object.get_web_site_option()

 

[Benjamin Peterson]

Hi,

Executing the example below doesn't produce the expected behavior, but
using the commented code does. Is this normal, or is it a problem with
Python? I've tested it with version 2.6.1 on Windows XP.

Thanks,

--

from abc import *
from types import *
import re

class Base (ObjectType):
__metaclass__ = ABCMeta

def __init__(self):
for option in self.get_options().keys():
method = 'get_%s_option' % re.sub(' ', '_', option.lower
())
setattr(self.__class__, method, lambda self:
self.get_option(option))

This is because the closure over option is changed when it is reassigned in the
for loop. For example:
.... return [lambda: num for num in xrange(2)]
....

f()

f()[0]

g = f()
g[0]() 1
g[1]()

1

 

[Márcio Faustino]

So simple thanks!

 

[R. David Murray]

Executing the example below doesn't produce the expected behavior, but
using the commented code does. Is this normal, or is it a problem with
Python? I've tested it with version 2.6.1 on Windows XP.

Thanks,

--

from abc import *
from types import *
import re

class Base (ObjectType):
__metaclass__ = ABCMeta

def __init__(self):
for option in self.get_options().keys():
method = 'get_%s_option' % re.sub(' ', '_', option.lower
())
setattr(self.__class__, method, lambda self:
self.get_option(option))

This is because the closure over option is changed when it is reassigned in the
for loop. For example:
... return [lambda: num for num in xrange(2)]
...

f()

f()[0]

g = f()
g[0]() 1
g[1]()

1

Here's the way I find it useful to think about this:

When your lambda is created in your for loop inside your __init__ method,
it acquires a pointer to __init__'s local namespace. (That's how I
understand what "closure" means in this case, though I imagine "closure"
probably means something slightly different in computer-science-ese

So, when any of those lambdas is executed, they all have a pointer to
the exact same namespace, that of __init__. And when they are called,
__init__'s namespace is in whatever state it was left in when __init__
ended. In this case, that means that 'option' is pointing to the value
it had at the _end_ of the for loop.

Hope this helps. I find that thinking in terms of namespaces helps
me understand how Python works better than any other mental model
I've come across.

 

[Michele Simionato]

Hi,

Executing the example below doesn't produce the expected behavior, but
using the commented code does. Is this normal, or is it a problem with
Python?

It is a common gotcha. Notice that it has nothing to do with lambda
functions, you
would get the same issue with regular functions.

This post http://www.artima.com/forums/flat.jsp?forum=106&thread=251156
and especially the comments
below should share some light on why it is the way it is (short
answer: for loops are
implemented by mutating the loop variable at each iteration).

 

[Gabriel Genellina]

En Fri, 20 Mar 2009 09:28:08 -0300, R. David Murray

Here's the way I find it useful to think about this:

When your lambda is created in your for loop inside your __init__ method,
it acquires a pointer to __init__'s local namespace. (That's how I
understand what "closure" means in this case, though I imagine "closure"
probably means something slightly different in computer-science-ese

Well, some people would say that the function "closes over" part of its
environment, but that's the idea.

So, when any of those lambdas is executed, they all have a pointer to
the exact same namespace, that of __init__. And when they are called,
__init__'s namespace is in whatever state it was left in when __init__
ended. In this case, that means that 'option' is pointing to the value
it had at the _end_ of the for loop.

That's a pretty good explanation -- I would just substitute "pointer" by
"reference".

Hope this helps. I find that thinking in terms of namespaces helps
me understand how Python works better than any other mental model
I've come across.

Good to know! The Python execution model *is* based on namespaces - any
other mental model won't be accurate past certain point (although some
people insist...)

 

[Márcio Faustino]

Hope this helps.  I find that thinking in terms of namespaces helps
me understand how Python works better than any other mental model
I've come across.

It does, thanks.

This post http://www.artima.com/forums/flat.jsp?forum=106&thread=251156
and especially the comments
below should share some light on why it is the way it is (short
answer: for loops are
implemented by mutating the loop variable at each iteration).

Thanks for the link.

 

[Gabriel Genellina]

En Fri, 20 Mar 2009 23:16:00 -0300, alex goretoy

i looks at lambdas as unbound functions(or super function), in the case
above we create the functions in a list places it in memory unboud, once
binding a call to the memory address space it returns the value

it is basically same as doing this:
def f():
print "f"

a=f #unbound function, same as rename function
a() #bind call to address space

Mmm, I don't quite understand what you said. lambda creates functions that
aren't different than functions created by def: apart from the name,
they're really the same thing.

And if you imply that *where* you call a function does matter, it does
not. A function carries its own local namespace, its own closure, and its
global namespace. At call time, no additional "binding" is done (except
parameters -> arguments).

(and the address space is always the one of the running process)

 

[R. David Murray]

En Fri, 20 Mar 2009 23:16:00 -0300, alex goretoy


Mmm, I don't quite understand what you said. lambda creates functions that
aren't different than functions created by def: apart from the name,
they're really the same thing.

Oh, good, I'm not the only one for whom the above didn't make sense
I feel a little less dense now.

And if you imply that *where* you call a function does matter, it does
not. A function carries its own local namespace, its own closure, and its
global namespace. At call time, no additional "binding" is done (except
parameters -> arguments).

(and the address space is always the one of the running process)

I poked around in the API docs and experimented with func_closure and
related attributes, and after bending my brain for a while I think I
understand this. The actual implementation of the closure is a single
list of 'cell' objects which represent namespace slots in the nested
scopes in which the closed-over function is defined.

But the fact that it is a single list is an implementation detail, and
the implementation is in fact carefully designed so that conceptually
we can think of the closure as giving the function access to those
nested-scope namespaces in almost(*) the same sense that it has a
reference to the global and local namespaces. That is, if what a name
in _any_ of those namespaces points to is changed, then the closed-over
function sees those changes.

In this way, we understand the original example: when defining a
lambda having a 'free variable' (that is, one not defined in either the
local or global scope) that was a name in the surrounding function's
local namespace, the lambda is going to see any changes made by the
surrounding function with regards to what that name points to. Thus,
the final value that the lambda uses is whatever the final value of the
for loop variable was when the surrounding function finished executing.

However, I think that a Python closure is not quite the same thing as a
'computer science' closure, for the same reason that people coming from a
language with variables-and-values as opposed to namespaces get confused
when dealing with Python function call semantics. Consider:

http://en.wikipedia.org/wiki/Closure_(computer_science)

That says that a closure can be used to provide a function with a private
set of variables that persist from one invocation to the next, so that
a value established in one call can be accessed in the next. The last
part of that sentence is not true in Python, since any assignment inside
a function affects only the local (per-invocation) namespace or (given
a global statement) the global namespace. A function cannot change the
thing pointed to by a name in the closure. Only the outer function,
for whom that name is in its local namespace, can do that.

(*) That last sentence in the previous paragraph is why I said '_almost_
the same sense' earlier: a function can modify what names point to in
its local and global namespaces, but cannot modify what names point to
in the closure namespace.

Of course, we can produce the same _effect_ as a computer science closure
in Python by using mutable objects...which is exactly parallel to the
difference between passing mutable or immutable objects in a function
call.

 

[Gabriel Genellina]

En Sun, 22 Mar 2009 20:43:02 -0300, alex goretoy

Sorry to have confused yall. What I meant was that you can do something
like
this, where the fucntion isn't called until it is bount to () with the
right
params
... print "inside a"
...... print "inside b"
...... a()
... b()
...

d={ca,b)}
d[c][0]() inside a
d[c][1]() inside b
d[c(d[c][0],d[c][1])]

inside a
inside b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: None

where function a and b are bound in function c

Ah, so this is a terminology issue. I'd say that a and b are *called* in
function c, not *bound*. I've never seen "bind" used in this sense before,
but as Humpty Dumpty said to Alice:

- When I use a word, it means just what I choose it to mean -- neither
more nor less.
- The question is, whether you can make words mean so many different
things.
- The question is, which is to be master -- that's all.

(Lewis Carroll, Through the Looking Glass, ch. VI)

 

[Gabriel Genellina]

En Sun, 22 Mar 2009 16:42:21 -0300, R. David Murray

I poked around in the API docs and experimented with func_closure and
related attributes, and after bending my brain for a while I think I
understand this. The actual implementation of the closure is a single
list of 'cell' objects which represent namespace slots in the nested
scopes in which the closed-over function is defined.

But the fact that it is a single list is an implementation detail, and
the implementation is in fact carefully designed so that conceptually
we can think of the closure as giving the function access to those
nested-scope namespaces in almost(*) the same sense that it has a
reference to the global and local namespaces. That is, if what a name
in _any_ of those namespaces points to is changed, then the closed-over
function sees those changes.

In this way, we understand the original example: when defining a
lambda having a 'free variable' (that is, one not defined in either the
local or global scope) that was a name in the surrounding function's
local namespace, the lambda is going to see any changes made by the
surrounding function with regards to what that name points to. Thus,
the final value that the lambda uses is whatever the final value of the
for loop variable was when the surrounding function finished executing.
Exactly.

However, I think that a Python closure is not quite the same thing as a
'computer science' closure, for the same reason that people coming from a
language with variables-and-values as opposed to namespaces get confused
when dealing with Python function call semantics. Consider:

http://en.wikipedia.org/wiki/Closure_(computer_science)

That says that a closure can be used to provide a function with a private
set of variables that persist from one invocation to the next, so that
a value established in one call can be accessed in the next. The last
part of that sentence is not true in Python, since any assignment inside
a function affects only the local (per-invocation) namespace or (given
a global statement) the global namespace. A function cannot change the
thing pointed to by a name in the closure. Only the outer function,
for whom that name is in its local namespace, can do that.

That's true in Python 2.x, but 3.x has the "nonlocal" keyword - so you can
modify variables in outer scopes too:

p3> z = 1
p3> def o():
.... z = 2
.... def i():
.... nonlocal z
.... print("z in i:", z)
.... z = 5
.... print("z in o:", z)
.... i()
.... print("z in o:", z)
.... z=3
.... print("z in o at exit:", z)
.... return i
....
p3> i=o()
z in o: 2
z in i: 2
z in o: 5
z in o at exit: 3
p3> z
1
p3> i()
z in i: 3
p3> i()
z in i: 5

(Anyway I think the inability to "modify" a variable doesn't invalidate
the "closure" concept...)

 

[R. David Murray]

That's true in Python 2.x, but 3.x has the "nonlocal" keyword - so you can
modify variables in outer scopes too:

p3> z = 1
p3> def o():
... z = 2
... def i():
... nonlocal z
... print("z in i:", z)
... z = 5
... print("z in o:", z)
... i()
... print("z in o:", z)
... z=3
... print("z in o at exit:", z)
... return i
...
p3> i=o()
z in o: 2
z in i: 2
z in o: 5
z in o at exit: 3
p3> z
1
p3> i()
z in i: 3
p3> i()
z in i: 5

(Anyway I think the inability to "modify" a variable doesn't invalidate
the "closure" concept...)

Invalidate, no, but it does mean that the word meant something slightly
different to a Python 2.x programmer than to, say, a Scheme programmer.
We could say that a Python 2.x closure is a "read-only closure".

But now with Python 3.x we can really have fun (thank you for that info):
.... def a(x):
.... nonlocal z
.... z = z + x
.... def b(x):
.... nonlocal z
.... z = z - x
.... def p():
.... print(z)
.... z = 1
.... return a, b, p
.... 6

So, as the wikipedia article says, we could, if we wanted to, use python 3
closures to reimplement objects, in a very confusing fashion