Something is rotten in Denmark...

[harrismh777]

Allow me to clarify... I'm not speaking about whether the lambda is
short-hand for def, ... that part of the docs I understand well!... no
problems there.

Allow me to clarify a little further... the docs are misleading in
that they state that the lambda can be coded (as an expression) where
the def 'statement' cannot be coded. Well, I know, this is speaking to
the syntax rules not the binding rules, but the point is that it implies
that the lambda can be used where the def cannot... and this is where
the hypnosis takes place... we assume that something 'additional' is
happening with the lambda that is *not* happening with the def.

And the truth is that the def (save its coding syntax) is the 'same'
critter as the lambda. It seems, in fact, that the only difference is
two ... that 1) the lambda does not automatically bind to a name, and
2) the lambda is a constant expression rather than a statement.


<sigh>


thanks for listening...


m harris

 

[Steven D'Aprano]

Allow me to clarify a little further... the docs are misleading in
that they state that the lambda can be coded (as an expression) where
the def 'statement' cannot be coded. Well, I know, this is speaking to
the syntax rules not the binding rules, but the point is that it implies
that the lambda can be used where the def cannot...

And so it can.

and this is where
the hypnosis takes place... we assume that something 'additional' is
happening with the lambda that is *not* happening with the def.

This is not a failure of the docs, but of your assumption. The only
difference is that lambda is an expression , and is limited to a single
expression. The leap from "lambda is an expression" to "...and therefore
the thing created by lambda has 'additional' stuff beyond ordinary def
functions" is unjustified.

Nevertheless, it does seem to be awfully common. You're hardly alone.

 

[Steven D'Aprano]

The part that I don't see much about in the docs (some books, that is)
is that the lambda lookups occur late (the lambda is evaluated at the
time it is called). The Python docs on-line *do say* this (I found too
late) but its one quick phrase that can be missed. So, the i in
range(10) is sitting there at '9' by the time *any* of the ten lambdas
get called. This is not intuitive, nor good. IMHO

I agree it's not intuitive. But where does it say that programming
language semantics must always be intuitive? Whose intuition? Mine?
Yours? Linus Torvalds'? Donald Knuth's? My auntie Rose's?

Please allow me to whine a little bit, ... but the *whole point* of
iterating is to be able to implicitly grab each iterated value as it
flies by (by the lambda or anything else!) and there is not much point
to having a 'late-binding' on an iterable particularly range(n).

What do you expect this code to do?

a = 42
funcs = [(lambda x: x+a) for i in range(10)]
funcs[0](1)

a = 23
funcs[0](1)


Do you agree that `a` should be late bound in this situation?

If so, why do you think that `i` should be early bound here?

funcs = [(lambda x: x+i) for i in range(10)]


Oh, the fact that it works at all in Python 2.5 is a side-effect of i
leaking from the list comprehension:

funcs = [(lambda x: x+i) for i in range(10)]
del i
funcs[0](1)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
NameError: global name 'i' is not defined


We can see that the closure isn't created:

funcs[0].func_closure is None

True


However, with a generator expression, i does not leak, a closure is
created, but it is still late bound:

funcs = list((lambda x: x+i) for i in range(10))
funcs[0].func_closure

del i

Traceback (most recent call last):

funcs[0](1) 10
funcs[1](1)

10


Yes, I can explicitly grab each 'i' as it flies by with a little clever
coding of the default value for the lambda n, i=i: i + n but that
'trick' is not intuitive, nor is it clear reading. It 'works' is just
about all one can say for it (not very elegant).

It might be more clear reading if you do it this way:

funcs = [(lambda x, i=j: x+i) for j in range(10)]

Now the reader is no longer distracted by the "i=i" ugliness.

I'm not sure what the answer is, but I think all of us need to think
through it some more. Placing lambdas in a list comprehension is just
delicious, except for the explicit kludges we have to code to get it to
work. I'm wondering if whether it would make some sense to put some
'binding smarts' into the interpreter to allow for 'interpreter
intuition' (say AI ) that would presume to understand when early vs late
binding makes sense and apply early binding in those cases where the
context is not ambiguous and when it is clear that an iterable is being
passed to the constant lambda function??

The problem with Do What I Mean is that it so rarely Does What You Mean.
At best it Does What Some Other Guy Imagined I'd Probably Mean In This
Situation. Let's not go there.

 

[Terry Reedy]

The part that I don't see much about in the docs (some books, that is)
is that the lambda lookups occur late (the lambda is evaluated at the
time it is called). The Python docs on-line *do say* this (I found too
late) but its one quick phrase that can be missed. So, the i in
range(10) is sitting there at '9' by the time *any* of the ten lambdas
get called. This is not intuitive, nor good. IMHO

I readily admit that the docs might be improved. Part of the reason I
spend time responding to issues like this is to practive giving
explanations that I might use in my own writings or even in the doc. I
have at least one small idea already.

Yes, I can explicitly grab each 'i' as it flies by with a little clever
coding of the default value for the lambda n, i=i: i + n but that
'trick' is not intuitive, nor is it clear reading. It 'works' is just
about all one can say for it (not very elegant).

You found the alternative of using a closure (the g(i) that returns a
*new* function for each i). You happened to use lambda to create each
new function; a nested def statement would have served just as well.
Replacing the default arg trick was one of the reasons to introduce
closures (in about 2.3). It might be considered the recommended solution
to this use case.

work. I'm wondering if whether it would make some sense to put some
'binding smarts' into the interpreter to allow for 'interpreter
intuition' (say AI ) that would presume to understand when early vs late
binding makes sense and apply early binding in those cases where the
context is not ambiguous and when it is clear that an iterable is being
passed to the constant lambda function?

Oh the irony of this proposal. You scolded us for breaking code with 2
to 3 changes, and here you propose a change more radical than anything
done in Python 3, and certain to break code, introduce bugs, complicate
the language, and reduce its functionality. Most of Guido's design
decision are rejections of such changes ;-).

 

[Chris Angelico]

The problem with Do What I Mean is that it so rarely Does What You Mean.
At best it Does What Some Other Guy Imagined I'd Probably Mean In This
Situation. Let's not go there.

+1

One of my biggest "threats" to my coworkers goes along the lines of
"Well, go ahead, but you have to document it for the internal wiki".
If you can't document something clearly, it's probably a bad idea.

Chris Angelico

 

[Alain Ketterlin]

I agree it's not intuitive. But where does it say that programming
language semantics must always be intuitive?

Nowhere. But going against generally accepted semantics should at least
be clearly indicated. Lambda is one of the oldest computing abstraction,
and they are at the core of any functional programming language. Adding
a quick hack to python and call it "lambda" is just abuse of terminology
(I don't say python is the only abuser, implementing lambda is
difficult; see, e.g., Apple C extension called "blocks" and their
implementation of binding).

What do you expect this code to do?

a = 42
funcs = [(lambda x: x+a) for i in range(10)]
funcs[0](1) [...]
Do you agree that `a` should be late bound in this situation?

No, absolutely, definitely not. But hey, I didn't learn programming with
python.

-- Alain.

 

[Jussi Piitulainen]

Nowhere. But going against generally accepted semantics should at
least be clearly indicated. Lambda is one of the oldest computing
abstraction, and they are at the core of any functional programming
language. Adding a quick hack to python and call it "lambda" is just
abuse of terminology (I don't say python is the only abuser,
implementing lambda is difficult; see, e.g., Apple C extension
called "blocks" and their implementation of binding).

As far as I can see, and that may not be far enough, Python's lambda
expressions do implement a generally accepted semantics. It seems to
be essentially the same semantics that Scheme uses. The value of a
lambda expression is closed in the environment where it is evaluated.
When called, a function looks up the values of its free variables in
the environment where it is closed. (*)

Alonzo Church in his lambda calculus did not deal with variable
assignment at all, as far as I know - he was a logician, not a
programming language designer - and so he also did not need to talk
about _when_ things happen.

Purely functional programming languages also do not have variable
assignment and the whole issue of this thread simply cannot arise.

(*) Some Python folks insist that Python does not have variables or
variable assignment. They talk of names and rebinding of names. And
they call environments namespaces. But the important point is the
underlying reality:
2

Or the same with a closed-over variable that is only accessible
through the values of the lambda expressions at the time they are
called:
... f1=lambda : n
... n += 1
... f2=lambda : n
... n += 1
... return f1, f2
...(3, 3)

As expected.

Too long, didn't read? A Schemer finds no fault with Python's lambda.

 

[harrismh777]

funcs = [(lambda x, i=j: x+i) for j in range(10)]

Now the reader is no longer distracted by the "i=i" ugliness.

That's a good idea, in fact, change made!

The problem with Do What I Mean is that it so rarely Does What You Mean.
At best it Does What Some Other Guy Imagined I'd Probably Mean In This
Situation. Let's not go there.

Oh, I agree. This is one of those things where I agree
(tongue-in-cheek) in all cases except the one I'm whining about!

--just kidding, here. Yes, ambiguous is bad.

 

[harrismh777]

What do you expect this code to do?

a = 42
funcs = [(lambda x: x+a) for i in range(10)]
funcs[0](1)

I do see your point with this... truly... but it did get me to think
about what I *do* expect... and that is that 'a' (for the lambda) will
be whatever 'a' is (now) at the time when the anonymous function is
returned, not later when it is called (not really). If I understand
things correctly, if 'a' references a different simple int object
'later' (before the anonymous function is called) then the result of the
lambda may not be what was expected. In your example, of course, the 'i'
is not relevant. On the other hand, as in callbacks, the whole reason
we want to use the lambda in the first place is because we don't know
what the data will be 'later,' and in fact we really do want
'late-binding' after all. (I guess, I want my cake on a nice china
saucer, with a silver spoon, 'and' I want to eat it too... ) 'magine that!

I know, you're saying "well, duh". But here's the thing... that's
the beauty and the curse of pure functional programming (like haskell)
which (by the way) doesn't have this problem, because doesn't have
mutables as in Python ( nor other languages, non functional ). So, those
of us attempting to morph functional programming over python are having
a little difficulty because what we expect to happen with the lambda in
a list comprehension is adversely affected by our misunderstanding of
python's 'late-binding'. See the whine, here?



kind regards,
m harris

 

[harrismh777]

Oh the irony of this proposal. You scolded us for breaking code with 2
to 3 changes, and here you propose a change more radical than anything
done in Python 3, and certain to break code, introduce bugs, complicate
the language, and reduce its functionality. Most of Guido's design
decision are rejections of such changes ;-).

Yeah, I knew that was coming... and I deserved it too!

Yes, I'm torn a bit over this surprise. Because I really don't want
anything like ambiguous AI intuitions going on in the interpreter... its
just me whining out loud and realizing that explicit really is better
than implicit... much of the time.


kind regards,
m harris

 

[Steven D'Aprano]

What do you expect this code to do?

a = 42
funcs = [(lambda x: x+a) for i in range(10)] funcs[0](1)

I do see your point with this... truly... but it did get me to think
about what I *do* expect... and that is that 'a' (for the lambda) will
be whatever 'a' is (now) at the time when the anonymous function is
returned, not later when it is called (not really).

Arguably, there's nothing wrong with early binding as a strategy. It's
just a different strategy from late binding.

It seems to me that early binding is less flexible than late, because
with late binding you have a chance to simulate early binding by saving a
reference of the variable elsewhere, such as in a default value, or an
instance attribute. But with early binding, you're stuck. There's no
simple or straightforward way to simulate late binding in an early
binding language.

 

[Ian Kelly]

It seems to me that early binding is less flexible than late, because
with late binding you have a chance to simulate early binding by saving a
reference of the variable elsewhere, such as in a default value, or an
instance attribute. But with early binding, you're stuck. There's no
simple or straightforward way to simulate late binding in an early
binding language.

Well, you can always do something like this:
55

I realize that's not exactly the same thing since you're stuck working
with "a.value" instead of just "a", but it does produce the desired
result.

Cheers,
Ian

 

[Terry Reedy]

Nowhere. But going against generally accepted semantics should at least
be clearly indicated. Lambda is one of the oldest computing abstraction,
and they are at the core of any functional programming language. Adding
a quick hack to python and call it "lambda" is just abuse of terminology

Whether or not it is abuse, I agree it was a big mistake. The keyword
should have been something like 'func': an abbreviated term for a
construct with abbreviated capabilities. End of argument over whether
Python's function expressions matche the semantics of lambda in other
languages. They are definitely highly limited in the scope of what they
can do.

 

[Gregory Ewing]

But going against generally accepted semantics should at least
be clearly indicated. Lambda is one of the oldest computing abstraction,
and they are at the core of any functional programming language.

Yes, and Python's lambdas behave exactly the *same* way as
every other language's lambdas in this area. Changing it to
do early binding would be "going against generally accepted
semantics".

It's not the lambda that's different from other languages,
it's the for-loop. In languages that encourage a functional
style of programming, the moral equivalent of a for-loop is
usually some construct that results in a new binding of the
control variable each time round, so the problem doesn't
arise very often.

If anything should be changed here, it's the for-loop, not
lambda.

 

[rusi]

Yes, and Python's lambdas behave exactly the *same* way as
every other language's lambdas in this area. Changing it to
do early binding would be "going against generally accepted
semantics".

It's not the lambda that's different from other languages,
it's the for-loop. In languages that encourage a functional
style of programming, the moral equivalent of a for-loop is
usually some construct that results in a new binding of the
control variable each time round, so the problem doesn't
arise very often.

If anything should be changed here, it's the for-loop, not
lambda.

I also thought so

So I tried:
Recast the comprehension as a map
Rewrite the map into a fmap (functionalmap) to create new bindings

def fmap(f,lst):
if not lst: return []
return [f(lst[0])] + fmap(f, lst[1:])

Still the same effects.

Obviously I am changing it at the wrong place...

 

[Jussi Piitulainen]

So I tried:
Recast the comprehension as a map
Rewrite the map into a fmap (functionalmap) to create new bindings

def fmap(f,lst):
if not lst: return []
return [f(lst[0])] + fmap(f, lst[1:])

Still the same effects.

Obviously I am changing it at the wrong place...

fs = [(lambda n : n + i) for i in range(10)]
[f(1) for f in fs]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

 

[Thomas Rachel]

Am 03.06.2011 01:43 schrieb Gregory Ewing:

It's not the lambda that's different from other languages,
it's the for-loop. In languages that encourage a functional
style of programming, the moral equivalent of a for-loop is
usually some construct that results in a new binding of the
control variable each time round, so the problem doesn't
arise very often.

If anything should be changed here, it's the for-loop, not
lambda.

In my opinion, it is rather the closure thing which confused me at some
time, and that's exactly what is the subject of the thread.


On one hand, a closure can be quite handy because I have access to an
"outer" vaiable even it changes.

But on the other hand, I might want to have exactly the value the
variable had when defining the function. So there should be a way to
exactly do so:

funcs=[]
for i in range(100):
def f(): return i
funcs.append(f)

for f in funcs: f()

Here, i should not be transported as "look what value i will get", but
"look what value i had when defining the function".

So there should be a way to replace the closure of a function with a
snapshot of it at a certain time. If there was an internal function with
access to the readonly attribute func_closure and with the capability of
changing or creating a cell object and thus hbeing capable of doing so,
it could be used a a decorator for a function to be "closure-snapshotted".

So in

funcs=[]
for i in range(100):
@closure_snapshot
def f(): return i
funcs.append(f)

each f's closure content cells would just be changed not to point to the
given variables, but to a cell referenced nowhere else and initialized
with the reference pointed to by the original cells at the given time.


Thomas

 

[Alain Ketterlin]

Yes, and Python's lambdas behave exactly the *same* way as
every other language's lambdas in this area. Changing it to
do early binding would be "going against generally accepted
semantics".

You must be kidding. Like many others, you seem to think that Scheme is
a typical functional language, which it is not. Learn about ML (i.e.,
SML or CaML), Erlang, Haskell, etc. You can read, e.g.,
http://en.wikipedia.org/wiki/Closure_(computer_science)

The reason why we have the kind of lambdas we have in python (and
scheme, and javascript, etc.) is just that it is way easier to
implement. That's all I've said. And people have gotten used to it,
without ever realizing they are using something completely different
from what Church called the "lambda abstraction".

Whether the python/... concept of lambda is useful or not is another,
subjective question, that I'm not intersted in. If you're pleased with
it, go ahead.

(End of discussion for me.)

-- Alain.

 

[Jussi Piitulainen]

You must be kidding. Like many others, you seem to think that Scheme is
a typical functional language, which it is not. Learn about ML (i.e.,
SML or CaML), Erlang, Haskell, etc. You can read, e.g.,
http://en.wikipedia.org/wiki/Closure_(computer_science)

That seems a good read, but I don't see how it supports your
contention that Python goes against generally accepted semantics.

The reason why we have the kind of lambdas we have in python (and
scheme, and javascript, etc.) is just that it is way easier to
implement. That's all I've said. And people have gotten used to it,
without ever realizing they are using something completely different
from what Church called the "lambda abstraction".

Church did not deal with assignment statements and order of execution.
Python has to.

Whether the python/... concept of lambda is useful or not is another,
subjective question, that I'm not intersted in. If you're pleased with
it, go ahead.

(End of discussion for me.)

Oh well.

 

[Nobody]

Yes, and Python's lambdas behave exactly the *same* way as
every other language's lambdas in this area. Changing it to
do early binding would be "going against generally accepted
semantics".

In Lisp, it depends upon whether the free variable is bound:

$ clisp
[1]> (setq f (lambda (x) (+ x i)))
#<FUNCTION :LAMBDA (X) (+ X I)>
[2]> (setq i 7)
7
[3]> (apply f (list 4))
11
[4]> (setq i 12)
12
[5]> (apply f (list 4))
16
^D
$ clisp
[1]> (let ((i 7)) (setq f (lambda (x) (+ x i))))
#<FUNCTION :LAMBDA (X) (+ X I)>
[2]> (apply f (list 4))
11
[3]> (setq i 12)
12
[4]> (apply f (list 4))
11
^D

If the variable is bound, then the lambda creates a closure.

And Python behaves the same way:

f = (lambda i: lambda x: x + i)(7)
f(4) 11
i = 12
f(4)

11

# If you really want a "let", this syntax is closer:
# f = (lambda i = 7: lambda x: x + i)()

The original semantics (from the lambda calculus) don't deal with the
concept of mutable state.

If anything should be changed here, it's the for-loop, not lambda.

Right. The for loop should bind the variable rather than set it.