Something is rotten in Denmark...

[harrismh777]

fs=[]

fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good

( that was a big surprise! . . . )
( let's try it another way . . . )

fs =[]
def g(i): return (lambda n: i + n)
fs = [g(i) for i in range(10)]
[fs(1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

(aaah, that's better . . . )
(hmmm, let's try another . . . )

fs =[]
for i in range(10): fs.append(lambda n, i=i: i + n)
[fs(1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

(aaaah, that works too . . . )
(... and another... )

fs=[]
fs = [(lambda n, i=i: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

(great! . . . )
(now then, what's wrong with this picture ?)

lambda? closure? scope? bug?

What is going on with the binding in the first construct... this
seems to reduce the usefulness of lambda to a considerable extent?





kind regards,
m harris

 

[Chris Rebert]

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10]         <=== not good

   ( that was a big surprise! . . . )

    lambda?  closure?  scope?   bug?

    What is going on with the binding in the first construct...this seems
to reduce the usefulness of lambda to a considerable extent?

http://stackoverflow.com/questions/233673/lexical-closures-in-python
(See 1st and 2nd answers)

Cheers,
Chris

 

[Ian Kelly]

    What is going on with the binding in the first construct... this seems
to reduce the usefulness of lambda to a considerable extent?

I don't see why; as you've shown there are a couple of simple ways to
avoid this problem. The trick is just to recognize that when you have
a closure around a variable, and that variable needs to change, but
the value in the closure needs to be constant, then what you really
need are two separate variables -- the cell variable needs to be
promoted to a local. How you accomplish that is not terribly
important.

One other technique that is sometimes preferable is functools.partial, e.g.:

fs = [functools.partial(operator.add, i) for i in range(10)]

Tangentially, I'd like to point out that this line:

[fs(1) for i in range(10)]

is more naturally written as:

[f(1) for f in fs]

 

[Jussi Piitulainen]

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good

( that was a big surprise! . . . )
( let's try it another way . . . )

The ten functions share the same i. The list comprehension changes the
value of that i. At the time when the functions are called, the value
is 9.

A different list comprehension mechanism could create a fresh i for
each element instead of changing the value of one i. Then each of the
functions would have a private i which would have the value it had at
the time of the creation of the closure. That is not the Python
mechanism.

The same sharing-an-i thing happens here:

fs = []
for i in range(4):

.... fs.append(lambda n : i + n)
....

fs[0](0)

3

And the different private-j thing happens here:

gs = []
for i in range(4):

.... gs.append((lambda j : lambda n : j + n)(i))
....

gs[0](0)

0

You used the lambda itself to introduce its private i in your other
examples, in (lambda n, i=i : i + n). In its i=i, the i to the left is
a different i - will be a fresh i every time the function is called, I
think - while the i to the right gets resolved to the value of the i
that the list comprehension is stepping at the time when the closure
is created.

What is going on with the binding in the first
construct... this seems to reduce the usefulness of lambda to a
considerable extent?

The lambda is doing its lambda thing exactly. The list comprehension
just updates the one i in whatever you call it that each of the ten
closures remember, and they all observe the updates, so to say.

It's a bit subtle. Using different names might help, like I used j.

 

[Thomas Rachel]

Am 31.05.2011 12:08 schrieb Jussi Piitulainen:

The same sharing-an-i thing happens here:

fs = []
for i in range(4):

... fs.append(lambda n : i + n)
...

fs[0](0)

3

And the different private-j thing happens here:

gs = []
for i in range(4):

... gs.append((lambda j : lambda n : j + n)(i))
...

gs[0](0)

0

There is a simpler way: with

fs = []
for i in range(4):

... fs.append(lambda n, i=i: i + n)
...

you give each lambda a different default argument.

fs[0](0)

0

Thomas

 

[Jussi Piitulainen]

Am 31.05.2011 12:08 schrieb Jussi Piitulainen:

The same sharing-an-i thing happens here:

fs = []
for i in range(4):

... fs.append(lambda n : i + n)
...

fs[0](0)

3

And the different private-j thing happens here:

gs = []
for i in range(4):

... gs.append((lambda j : lambda n : j + n)(i))
...

gs[0](0)

0

There is a simpler way: with

fs = []
for i in range(4):

... fs.append(lambda n, i=i: i + n)
...

you give each lambda a different default argument.

fs[0](0)

0

I know, and harrismh777 knows. I find it an unnecessary distraction
when explaining why the different closures in the initial example
behave identically, but I did discuss it at the end of my post.

 

[Terry Reedy]

fs=[]

Irrelevant here since you immediately rebind 'fs'.

fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

Same as [f(1) for f in fs]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good

( that was a big surprise! . . . )

You have been hypnotizeed by lambda. (lambda n: i+n) is a *constant
expression*, so you get 10 'equal' functions. To see this better

fs = [(lambda n: i + n) for i in range(10)]
from dis import dis
for f in fs: dis(f)

1 0 LOAD_DEREF 0 (i)
3 LOAD_FAST 0 (n)
6 BINARY_ADD
7 RETURN_VALUE

1 0 LOAD_DEREF 0 (i)
3 LOAD_FAST 0 (n)
6 BINARY_ADD
7 RETURN_VALUE
....

All have the same bytecode and all retrieve the same last value of i in
the nonlocal listcomp scope when you call them *after* the listcomp
scope has otherwise disappeared.

Your code is equivalent to

fs = []
for i in range(10):
fs.append(lambda n: i + n)
print([f(1) for f in fs])

which is equivalent (except for naming the functions) to

fs = []
for i in range(10):
def f(n): return i + n
fs.append(f)
print([f(1) for f in fs])

Does [10, 10, 10, 10, 10, 10, 10, 10, 10, 10] still surprise?
Because the def is a constant expression, we can move it out of the
loop, and get the equivalent (except for identity)

def f(n): return i + n
fs = []
for i in range(10):
fs.append(f)
print([f(1) for f in fs])

This in turn is equivalent to

def f(n): return i + n
fs = []
for _ in range(10):
fs.append(f)
i=9
print([f(1) for f in fs])

which in turn is equivalent in output to

def f(n): return i + n
i = 9
print([f(1) for _ in range(10)])

Note that:

def f(n): return i+n # or f = lambda n: i+n
fs = [f for i in range(10)]
print([f(1) for f in fs])

works in 2.7, with the same output, but not in 3.2 because in 3.x, i is
local to the list comp and the later call raises unbound global error.

( let's try it another way . . . )

All these other ways create 10 *different* (unequal) functions that are
different because they have captured 10 different values of i when
defined instead of deferring lookup of i to when they are called.

def g(i): return (lambda n: i + n)
fs = [g(i) for i in range(10)]
print([f.__closure__[0].cell_contents for f in fs])

fs = [(lambda n, i=i: i + n) for i in range(10)]
print([f.__defaults__[0] for f in fs])

# CPython 3.2 dependent code !!!

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

 

[harrismh777]

You have been hypnotizeed by lambda. (lambda n: i+n) is a *constant
expression*, so you get 10 'equal' functions.

'hypnotized' indeed! ... ok, so let me see if I get this... the lambda
defers lookup|bind of its references until such time as the lambda is
'called' and not at the time (as I thought) that the anonymous
function(s) are returned?

If I'm understanding that correctly, then that means lambda is working
as designed, and that there are very subtle nuances to be aware of. In
my little case

(lambda n: i + n)

... if the i goes out of scope before the anonymous function gets
called then we have a problem... or if i as a reference is mutable or
refers to a different object before the anonymous function is called
then we have a problem?

What I am discovering is that 'yes' I can use lambda syntactically
where I might not be able to code a def statement; however, if I do use
it (as in a list comprehension) then I may get unexpected results if any
of my lambda's references go out of scope or are mutable...(?)

Question:

What are the ramifications of making the lookup|binding happen at
the time the anonymous function is 'returned' vs 'called'? Has anyone
suggested this? Is this PEP-able....? Are there side-effects in the
other direction?


PS Thanks Chris, Ian, Jussi, Thomas, Terry... I appreciate your
teaching and patience !



kind regards,
m harris

 

[Martin Manns]

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good

( that was a big surprise! . . . )
( let's try it another way . . . )

After being confused I figured out it is a 3.x example:

----

$ python
Python 2.6.6 (r266:84292, Apr 20 2011, 11:58:30)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

----

$ python3.1
Python 3.1.3 (r313:86834, Nov 28 2010, 11:28:10)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10]

 

[Ian Kelly]

$ python
Python 2.6.6 (r266:84292, Apr 20 2011, 11:58:30)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

This works by accident.

[fs(1) for i in range(10)] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[fs[0](1) for i in range(10)] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[f(1) for f in fs]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10]

The i variable is part of the global scope, and as you iterate over
range(10) again it coincidentally takes on the same values as in the
original list comprehension. You don't see this in Python 3 because
the scope of i is limited to the list comprehension, not global.

Cheers,
Ian

 

[harrismh777]

After being confused I figured out it is a 3.x example:

Actually, it is a compatibility example between 2.x and 3.x, compare
below for different behavior from two seemingly identical compatible
constructs, one from 3.2, and the other from 2.6.4:

Python 3.2 (r32:88445, Mar 29 2011, 21:33:57)
[GCC 4.3.3] on linux2

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)] [10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=========== compare

Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2

fs=[]
fs = [(lambda n: i + n) for i in range(10)]
[fs(1) for i in range(10)] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] <=========== compare

Having compared the two, someone please tell me whether the two
are incompatible, mostly compatible, completely incompatible, or
different languages...


,.,, I realize how 3.2 is working (at the moment) but as compared
with the books, and the behavior of 2.6, it sure looks 'broke' to me...

.... why would we want to defer lookup of the 'i' in range(10) until the
anonymous function is called, instead of the time that the function
object is returned... inquiring minds want to know...


PS Ian calls the second construct "working by mistake..."







kind regards,
m harris

 

[harrismh777]

PS Ian calls the second construct "working by mistake..."

oops, actually he called it, "working by accident... "

 

[Ian Kelly]

If I'm understanding that correctly, then that means lambda is working as
designed, and that there are very subtle nuances to be aware of. In my
little case

  (lambda n: i + n)

  ... if the  i  goes out of scope before the anonymous function gets called
then we have a problem... or if  i   as a reference is mutable or refers to
a different object before the anonymous function is called then we have a
problem?

Actually, if i merely goes out of scope, there is no problem. It just
creates a closure. It's only when the i within that scope is modified
that we run into problems.

In fact, in Python 3 the scope of a list comprehension variable is the
list comprehension itself, so in your original post i was already out
of scope by the time you started calling the lambda functions.

Cheers,
Ian

 

[Chris Angelico]

    Having compared the two, someone please tell me whether the two are
incompatible, mostly compatible, completely incompatible, or different
languages...

By implication, every version of Python is incompatible with every
other. The 2.7.1 revision notes include:

- Issue #1713: Fix os.path.ismount(), which returned true for symbolic links
across devices.

Suppose some program were depending on this bug. It works under 2.7.0,
fails under 2.7.1. Does that mean that 2.7.1 is incompatible with
2.7.0?

Chris Angelico

 

[Chris Angelico]

- Issue #1713: Fix os.path.ismount(), which returned true for symbolic links
 across devices.

PS. I know nothing about this particular issue, I just skimmed down
the release notes and stopped when something caught my eye. Choose
another example if you know of a better one.

Chris Angelico

 

[Terry Reedy]

'hypnotized' indeed!

I say 'hypnotized' ;-) because people have posted examples almost
exactly like the one you did, with a list of 10 (usually) lambda-defined
functions, at least yearly for over a decade. Never have I seen the same
puzzlement when the functions are defined with def statements instead.


What you did differently is to continue investigating and discover some
of the alternatives for yourself. Hence you get more effort from others
in response.

I think part of the problem with lambda is this: the body of a def
statement is indented and nicely set apart from surrounding code. The
body of a lambda expression is prefixed by the header but there is no
terminator to visually indicate the end of the nested scope. I think
there should have been. I often add unneeded parens, as you did, but
that is not lambda specific. Generator expressions have to be set apart
with parentheses, and other comprehensions are all fenced.

There also seems to be a bit of lambda (oc)cultism and mystique of the
anonymous. However, if a function raises an exception, anonymity is a
defect.

What you did differently is to continue investigating and discover some
of the alternatives for yourself.

... ok, so let me see if I get this... the lambda
defers lookup|bind of its references until such time as the lambda is
'called' and not at the time (as I thought) that the anonymous
function(s) are returned?

In Python, there are no lambda objects to be called.

A def statement, when executed, produces an instance of the user-defined
function class. Its .__name__ attribute is the one given in the header
of the statement.
A lambda expression, when executed, produces an instance of the
user-defined function class. Its .__name__ attribute is '<lambda>' (at
least in CPython).
See the single difference? The objects produced by equivalent def
statements and lambda expressions are otherwise exactly the same.

statement with lambda arg-list: expression

is an abbreviation for

def new-name(arg-list): return expression
statement with new-name

assuming that the lambda in in the same scope as the statement and not
nested in another scope-creating expression

If I'm understanding that correctly, then that means lambda is working
as designed, and that there are very subtle nuances to be aware of.

I would say that there is no subtle nuance. Except for not providing a
name, a lambda expression is purely an optional syntactic abbreviation.

In my little case

(lambda n: i + n)

... if the i goes out of scope before the anonymous function gets called
then we have a problem...

There was no problem; nonlocal names and their associations with objects
get saved (in 'cells', for CPython) association with the function than
needs them. One of my examples showed how to retrieve them.

or if i as a reference is mutable

Objects are mutable, names can be rebound.

or refers to a different object before the anonymous function
is called then we have a problem?

Anonymity is completely irrelevant to all of the above. Everything is
the same as for def f(n): return i + n. When a function is compiled,
each name is classified as local, non-local, or global. When the
function is called and a name is evaluated, the corresponding object
from its scope is retrieved, if there is one, or an error is raised.

What I am discovering is that 'yes' I can use lambda syntactically where
I might not be able to code a def statement;

If the lambda expression is nested in an abbreviated nested scope
(lambda or comprehension), then you have to unabbreviate the outer
nesting before you can unabbreviate the lambda. Several example of this
have been shown.

however, if I do use it (as in a list comprehension) then I may get
unexpected results if any of my
lambda's references go out of scope or are mutable...(?)

Nothing needed goes out of scope and all names are rebindable (not
mutable).

What are the ramifications of making the lookup|binding happen at the
time the anonymous function is 'returned' vs 'called'?

This is early-binding versus late-binding. Python is a late-binding
language.

Are you asking about changing all function compilation or only when
functions are defined with lambda expressions? The latter *would* make
there be a 'suble nuance' that Python now lacks and does not need.

Has anyone suggested this?

Of course, more than once, in multiple variations. All have so far been
rejected, more than once. The most sensible ideas are for earlier
binding of builtins to make Python run faster.

You have already discovered and been given some of the solutions.
Another is to define a class with a __call__ statement and save the
desired early bindings as instance data attributes. Your g(i): return
lambda ... as well as the default arg trick are functional equivalents
of the same idea.

In general, if you want to immediately grab and save an object, then
explicity grab and save the object.

 

[harrismh777]

This is early-binding versus late-binding. Python is a late-binding
language.

ok ...

Are you asking about changing all function compilation or only when
functions are defined with lambda expressions?

At least lambda expressions, but (see below) any other built-in
where 'early-binding' makes sense for intuitive results or
performance--- possibly, not commenting.

The latter *would* make

there be a 'suble nuance' that Python now lacks and does not need.

How so, and why not?

Of course, more than once, in multiple variations. All have so far been
rejected, more than once. The most sensible ideas are for earlier
binding of builtins to make Python run faster.

At the moment I'm only speaking about my OP and that particular list
comprehension... the thing that happened (at least for me) is that the
intuitive sense that each 'i' somehow becomes a part of the anonymous
function (I know, not so) is built-in. There is little to nothing
indicating in the docs that this is not so. Again, what we have here is
the 'i' being saved in a cell and looked up at call time (an
implementation detail, 'late-binding') that is critical for the
user-coder to understand. This *may* be true for other built-ins also,
I'm not commenting on that, but it seems to me that if lambda is going
to remain in the language at all that 'early-binding' in the lambda
specific case would make sense; at least make the lambda more useful
generally.

.... just saying,


kind regards,
m harris

 

[Martin Manns]

The i variable is part of the global scope, and as you iterate over
range(10) again it coincidentally takes on the same values as in the
original list comprehension. You don't see this in Python 3 because
the scope of i is limited to the list comprehension, not global.

I read about the scope change.
However, list comprehension scope being global still feels "right" for
me because I am still using Python 2.

I feel that I should start worrying more about future migration issues.

Cheers

Martin

 

[Terry Reedy]

At the moment I'm only speaking about my OP and that particular list
comprehension... the thing that happened (at least for me) is that the
intuitive sense that each 'i' somehow becomes a part of the anonymous
function (I know, not so) is built-in. There is little to nothing
indicating in the docs that this is not so

On the contrary, the docs very explicitly say that a lambda expression
is equivalent to a def statement.

"[Lambda forms (lambda expressions)] are a shorthand to create anonymous
functions; the expression lambda arguments: expression yields a function
object. The unnamed object behaves like a function object defined with

def <lambda>(arguments):
return expression"

? Again, what we have here is

the 'i' being saved in a cell and looked up at call time (an
implementation detail, 'late-binding') that is critical for the
user-coder to understand.

Again, exactly the same as if the function were created with a def
statement.

I'm not commenting on that, but it seems to me that if lambda is going
to remain in the language at all that 'early-binding' in the lambda
specific case would make sense; at least make the lambda more useful
generally.

I disagree. Currently, late-binding is the default, with early-binding
an option through a few different mechanisms. Making early binding the
default would *reduce* the usefulness by eliminating the late-binding
option and would add nothing that cannot be done now.

There are some people whose 'intuition' is the opposite of yours. They
instead want to eliminate the early-binding option of default argument
expressions. They want to reduce flexibility in the other direction.
Both proposals are equally bad.

 

[harrismh777]

On the contrary, the docs very explicitly say that a lambda expression
is equivalent to a def statement.

Allow me to clarify... I'm not speaking about whether the lambda is
short-hand for def, ... that part of the docs I understand well!... no
problems there.

The part that I don't see much about in the docs (some books, that is)
is that the lambda lookups occur late (the lambda is evaluated at the
time it is called). The Python docs on-line *do say* this (I found too
late) but its one quick phrase that can be missed. So, the i in
range(10) is sitting there at '9' by the time *any* of the ten lambdas
get called. This is not intuitive, nor good. IMHO

Please allow me to whine a little bit, ... but the *whole point* of
iterating is to be able to implicitly grab each iterated value as it
flies by (by the lambda or anything else!) and there is not much point
to having a 'late-binding' on an iterable particularly range(n).

Yes, I can explicitly grab each 'i' as it flies by with a little clever
coding of the default value for the lambda n, i=i: i + n but that
'trick' is not intuitive, nor is it clear reading. It 'works' is just
about all one can say for it (not very elegant).

I'm not sure what the answer is, but I think all of us need to think
through it some more. Placing lambdas in a list comprehension is just
delicious, except for the explicit kludges we have to code to get it to
work. I'm wondering if whether it would make some sense to put some
'binding smarts' into the interpreter to allow for 'interpreter
intuition' (say AI ) that would presume to understand when early vs late
binding makes sense and apply early binding in those cases where the
context is not ambiguous and when it is clear that an iterable is being
passed to the constant lambda function??


kind regards,
m harris