Lambda going out of fashion

[Stephen Thorne]

Hi guys,

I'm a little worried about the expected disappearance of lambda in
python3000. I've had my brain badly broken by functional programming
in the past, and I would hate to see things suddenly become harder
than they need to be.

An example of what I mean is a quick script I wrote for doing certain
actions based on a regexp, which I will simlify in this instance to
make the pertanant points more relevent.

{
'one': lambda x:x.blat(),
'two': lambda x:x.blah(),
}.get(someValue, lambda x:0)(someOtherValue)

The alternatives to this, reletively simple pattern, which is a rough
parallel to the 'switch' statement in C, involve creating named
functions, and remove the code from the context it is to be called
from (my major gripe).

So, the questions I am asking are:
Is this okay with everyone?
Does anyone else feel that lambda is useful in this kind of context?
Are there alternatives I have not considered?

merrily-yr's
Stephen.

 

[Keith Dart]

Hi guys,

I'm a little worried about the expected disappearance of lambda in
python3000. I've had my brain badly broken by functional programming
in the past, and I would hate to see things suddenly become harder
than they need to be.

I use Python lambda quite a bit, and I don't understand the recent noise
about problems with it, and its removal. I don't have a problem with
lambdas.

My personal gripe is this. I think the core language, as of 2.3 or 2.4
is very good, has more features than most people will ever use, and they
(Guido, et al.) can stop tinkering with it now and concentrate more on
the standard libraries.

 

[Steven Bethard]

{
'one': lambda x:x.blat(),
'two': lambda x:x.blah(),
}.get(someValue, lambda x:0)(someOtherValue)

The alternatives to this, reletively simple pattern, which is a rough
parallel to the 'switch' statement in C, involve creating named
functions, and remove the code from the context it is to be called
from (my major gripe).

Here's what my code for a very similar situation usually looks like:

py> class Foo(object):
.... def blat(self):
.... print "blat"
.... def blah(self):
.... print "blah"
....
py> key, foo = 'two', Foo()
py> try:
.... result = dict(one=Foo.blat, two=Foo.blah)[key](foo)
.... except KeyError:
.... result = 0
....
blah

As you can see, I just use the unbound methods of the parent class
directly. Of course this means that your code won't work if 'foo' isn't
actually a Foo instance. So you lose a little generality, but you gain
a bit in conciceness of expression (IMHO). Note also that I don't use
dict.get, relying on the KeyError instead. Usually, my case for when no
function applies is complex enough to not be expressable in a lambda
anyway, so this is generally more appropriate for my code.

While I don't generally find that I need lambda, I'm not particularly
arguing against it here. I just thought it might be helpful to
demonstrate how this code might be written concicely without lambdas.

Steve

 

[tanghaibao]

This is NOT true. Functional programming, AFAIKC, is a cool thing, and
in-fashion, increases productivity & readability, and an indispensable
thing for a flexible scripting lang. The inline/lambda function is
feature is shared by many lang. which I frequently use,
MATLAB/R/Python/etc. Well, you can say apply() is 'deprecated' now,
(which is another functional thing I like), but I am still using it. I
am not a desperate person who uses higher-order function factory a lot,
but if lambda keyword is removed, I swear I will not use the python
anymore.

 

[Craig Ringer]

I use Python lambda quite a bit, and I don't understand the recent noise
about problems with it, and its removal. I don't have a problem with
lambdas.

My personal gripe is this. I think the core language, as of 2.3 or 2.4
is very good, has more features than most people will ever use, and they
(Guido, et al.) can stop tinkering with it now and concentrate more on
the standard libraries.

As someone working with the Python/C API, I'd have to argue that the
Python language may (I express no opinion on this) be "Done", but
CPython doesn't look like it is.

In my view a bunch of minor, but irritating, issues exist:

It's hard to consistently support Unicode in extension modules without
doing a lot of jumping through hoops. Unicode in docstrings is
particularly painful. This may not be a big deal for normal extension
modules, but when embedding Python it's a source of considerable
frustration. It's also not easy to make docstrings translatable.
Supporting an optional encoding argument for docstrings in the
PyMethodDef struct would help a lot, especially for docstrings returned
by translation mechanisms.

Py_NewInterpreter/Py_EndInterpreter used in the main thread doesn't
work with a threaded Python debug build, calling abort(). If a
non-threaded Python is used, or a non-debug build, they appear to work
fine. There are some indications on the mailing list that this is due to
Python's reliance on thread-local-storage for per-interpreter data, but
there are no suggestions on how to work around this or even solid
explanations of it. This sucks severely when embedding Python in Qt
applications, as GUI calls may only be made from the main thread but
subinterepreters may not be created in the main thread. It'd be
fantastic if the subinterpreter mechanism could be fleshed out a bit. I
looked at it, but my C voodo just isn't up there. The ability to run
scripts in private interpreters or subinterpreters (that are disposed of
when the script terminates) would be immensely useful when using Python
as a powerful glue/scripting language in GUI apps.

IMO the reference behaviour of functions in the C API could be
clearer. One often has to simply know, or refer to the docs, to tell
whether a particular call steals a reference or is reference neutral.
Take, for example, PyDict_SetItemString vs PyMapping_SetItemString . Is
it obvious that one of those steals a reference, and one is reference
neutral? Is there any obvious rationale behind this? I'm not overflowing
with useful suggestions about this, but I do think it'd be nice if there
was a way to more easily tell how functions behave in regard to
reference counts.

Const. I know there's a whole giant can of worms here, but even so -
some parts of the Python/C API take arguments that will almost always be
string literals, such as format values for Py_BuildValue and
PyArg_ParseTuple or the string arguments to Py*_(Get|Set)String calls.
Many of these are not declared const, though they're not passed on to
anywhere else. This means that especially in c++, one ends up doing a
lot of swearing and including a lot of ugly and unnecessary string
copies or const_cast<char*>()s to silence the compiler's whining. It
would be nice if functions in the Python/C API would declare arguments
(not return values) const if they do not pass them on and do not change
them.

Of course, all these are just my opinion in the end, but I'd still have
to argue that using Python from C could be a lot nicer than it is. The
API is still pretty good, but the solution of these issues would make it
a fair bit nicer again, especially for people embedding Python in apps
(a place were it can seriously excel as a scripting/extension/glue
language).

 

[Steven Bethard]

Well, you can say apply() is 'deprecated' now,
(which is another functional thing I like), but I am still using it.

Interesting. Could you explain why? Personally, I find the
*expr/**expr syntax much simpler, so I'd be interested in knowing what
motivates you to continue to use apply...

Steve

 

[Craig Ringer]

This is NOT true. Functional programming, AFAIKC, is a cool thing, and
in-fashion, increases productivity & readability, and an indispensable
thing for a flexible scripting lang.

Couldn't agree more. One of the things I find most valuable about Python
is the ability to use functional style where it's the most appropriate
tool to solve a problem - WITHOUT being locked into a pure-functional
purist language where I have to fight the language to get other things
done.

The inline/lambda function is
feature is shared by many lang. which I frequently use,
MATLAB/R/Python/etc. Well, you can say apply() is 'deprecated' now,
(which is another functional thing I like), but I am still using it. I
am not a desperate person who uses higher-order function factory a lot,
but if lambda keyword is removed, I swear I will not use the python
anymore.

I also make signficant use of Lambda functions, but less than I used to.
I've recently realised that they don't fit too well with Python's
indentation-is-significant syntax, and that in many cases my code is
much more readable through the use of a local def rather than a lambda.

In other words, I increasingly find that I prefer:

def myfunction(x):
return x**4
def mysecondfuntion(x):
return (x * 4 + x**2) / x
make_some_call(myfunction, mysecondfunction)

to:

make_some_call( lambda x: x**4, lambda x: (x * 4 + x**2) / x)

finding the former more readable. The function names are after all just
temporary local bindings of the function object to the name - no big
deal. Both the temporary function and the lambda can be used as
closures, have no impact outside the local function's scope, etc. I'd be
interested to know if there's anything more to it than this (with the
side note that just don't care if my temporary functions are anonymous
or not).

One of the things I love about Python is the ability to mix functional,
OO, and procedural style as appropriate. I can write a whole bunch of
functions that just return the result of list comprehensions, then use
them to operate on instances of an object from a procedural style main
function. In fact, that's often the cleanest way to solve the problem.
The fact that I have that option is something I really like.

As for apply(), while it is gone, the f(*args) extension syntax is now
available so I don't really see the issue. After all, these two are the
same:

def callfunc(function,args):
return apply(function,args)

and

def callfunc(function,args):
return function(*args)

its just an (IMO trivial) difference in syntax. I'd be interested in
knowing if there is in fact more to it than this.

 

[Fredrik Lundh]

It's hard to consistently support Unicode in extension modules without
doing a lot of jumping through hoops. Unicode in docstrings is
particularly painful. This may not be a big deal for normal extension
modules, but when embedding Python it's a source of considerable
frustration. It's also not easy to make docstrings translatable.
Supporting an optional encoding argument for docstrings in the
PyMethodDef struct would help a lot, especially for docstrings returned
by translation mechanisms.

docstrings should be moved out of the C modules, and into resource
files. (possibly via macros and an extractor). when I ship programs
to users, I should be able to decide whether or not to include docstrings
without having to recompile the darn thing.

and yes, docstrings should support any encoding supported by python's
unicode system.

Const. I know there's a whole giant can of worms here, but even so -
some parts of the Python/C API take arguments that will almost always be
string literals, such as format values for Py_BuildValue and
PyArg_ParseTuple or the string arguments to Py*_(Get|Set)String calls.
Many of these are not declared const, though they're not passed on to
anywhere else. This means that especially in c++, one ends up doing a
lot of swearing and including a lot of ugly and unnecessary string
copies or const_cast<char*>()s to silence the compiler's whining. It
would be nice if functions in the Python/C API would declare arguments
(not return values) const if they do not pass them on and do not change
them.

I think the only reason that this hasn't already been done is to reduce the
amount of swearing during the conversion process (both for the core developer
and C extension developers...).

</F>

 

[Alex Martelli]

My personal gripe is this. I think the core language, as of 2.3 or 2.4
is very good, has more features than most people will ever use, and they

Indeed, it has _too many_ features. Look at the PEP about 3.0, and
you'll see that removing redundant features and regularizing a few
oddities is what it's meant to be all about.

(Guido, et al.) can stop tinkering with it now and concentrate more on
the standard libraries.

No doubt the standard library needs more work, particularly if you count
the built-ins as being part of the library (which, technically, they
are). Indeed, much of the redundancy previously mentioned is there
rather than in the core language strictly speaking -- e.g., all of the
things returning lists (from keys, values, items methods in dicts, to
the range built-in) mostly-duplicated with things returning iterators
(iterkeys, etc) or nearly so (xrange). These are things that can't
change in 2.5 to avoid breaking backwards compatibility. Other things,
where bw compat is not threatened, are no doubt going to be targeted in
2.5.


Alex

 

[Alan Gauld]

I'm a little worried about the expected disappearance of lambda in
python3000. I've had my brain badly broken by functional programming
in the past, and I would hate to see things suddenly become harder
than they need to be.

Me too.
But its not only about becoming harder, I actually like the fact
that lamda exists as a tie in to the actual concept of an
anonymous function. When you read a math book on lambda calculus
its nice to transform those concepts directly to the language.

The current Pythonic lambda has its limitations, but being able
to explicitly create lambdas is a nice feature IMHO. Its much
better than having to create lots of one-off single use functions
with meaningless names.

It can't be that hard to maintain the lambda code, why not just
leave it there for the minority of us who like the concept?

Alan G.
Author of the Learn to Program website
http://www.freenetpages.co.uk/hp/alan.gauld

 

[Alan Gauld]

but if lambda keyword is removed, I swear I will not use the python
anymore.

While I would be dissappointed to lose lambda, I think losing
Python would hurt me a lot more! After all we aren't losing
functionality here, just adding sonme extra work and losing some
readability. Pythonic lambdas are just syntactic sugar in
practice, they just make the code look more like true
functional code.

Alan G.
Author of the Learn to Program website
http://www.freenetpages.co.uk/hp/alan.gauld

 

[Paul Rubin]

readability. Pythonic lambdas are just syntactic sugar in
practice,

Actually it's the other way around: it's named functions that are the
syntactic sugar.

 

[Alex Martelli]

Couldn't agree more. One of the things I find most valuable about Python
is the ability to use functional style where it's the most appropriate
tool to solve a problem - WITHOUT being locked into a pure-functional
purist language where I have to fight the language to get other things
done.

By the way, if that's very important to you, you might enjoy Mozart
(http://www.mozart-oz.org/) -- I'm looking at it and it does appear to
go even further in this specific regard (rich support for multi -
paradigm programming). It's also blessed with a great book,
<http://www.info.ucl.ac.be/people/PVR/book.html> -- I've just started
browsing it, but it appears to be worthy of being called "SICP for the
21st century"...!-). Just like SICP made it worthwhile to learn a
little Scheme even if you'd never use it in production, so does CTMCP
(acronym for this new book by Van Roy and Haridi) work for Oz, it
appears to me.

def callfunc(function,args):
return apply(function,args)

and

def callfunc(function,args):
return function(*args)

its just an (IMO trivial) difference in syntax. I'd be interested in
knowing if there is in fact more to it than this.

No, the semantics are indeed the same.


Alex

 

[Stephen Thorne]

Actually it's the other way around: it's named functions that are the
syntactic sugar.

Not true, you can't re-write

def f():
raise ValueError, "Don't call f"

as a lambda. Lambdas contain only a single expression. Even the py3k
wiki page ignores this critical difference. A single expression means
no statements of any kind can be included. Assignment, if/elif/else,
while, for, try/except, etc are not catered for in lambdas.

There has been a case in the past for a lambda that contains
statements, but they have been stomped on due to problems with syntax.
I don't like lambdas that have more than a single expression myself
(if it's more complex than "lambda x:baz(x.foo(y))", I would prefer to
write a named function).

ultimate-ly yr's.
Stephen Thorne.

 

[Fredrik Lundh]

Not true, you can't re-write

def f():
raise ValueError, "Don't call f"

f = lambda: eval(compile("raise ValueError(\"Don't call f\")", "", "exec"))

note that lambdas have names too, btw:

print f.func_name
<lambda>

</F>

 

[Alex Martelli]

It can't be that hard to maintain the lambda code, why not just
leave it there for the minority of us who like the concept?

I guess the key motivator is the usual set of design principles:

3. Keep the language small and simple.
4. Provide only one way to do an operation.

Well, this phrasing is actually from the "Spirit of C" section of the
introduction to the ISO C standard, but the Zen of Python phrasing,
"""There should be one-- and preferably only one --obvious way to do
it.""" isn't all that different.

Having lambda in the language in addition to def provides two ways to do
an operation (make a function), and for that it makes the language a
little bit less simple and less small than if only def was there. I
guess that's why Guido now regrets ever having accepted lambda into the
language, and hopes to take it away when backwards compatibility can be
broken (i.e., in 3.0).


Having just reviewed, edited and merged over 1000 recipes to select
about 1/3 of those for the 2nd edition of the Cookbook, I think I'm in a
good position to state that, at least as far as CB contributors are
representative of the Python community, uses of lambda that are dubious
to very dubious to absurd outnumber those which are decent to good by
around 4:1. If I see another case of:

somename = lambda x: ...

instead of the obvious

def somename(x): ...

I think I'll scream, though not quite as loud as for my next seeing:

map(lambda x: f(x), ...

instead of

map(f, ...


I don't know what it IS about lambda that prompts so much dubious to
absurd use, but that's what I observed. I don't know if that plays any
role in Guido's current thinking, though -- I have no idea how much
"dubious Python" he's had to struggle with. One could argue that the
typical abuses of redundant lambda as shown above are closely related,
e.g., to typical abuses of booleans, such as:

if (x>y) == True:

Funny enough, though, I've seen the equivalent of this latter abuse very
often in C and C++, but not in Python (again judging e.g. from the
submissions to the cookbook site). Maybe it's bool's relatively recent
introduction in the language; after all, one DOES often see:

if len(somecontainer) > 0:

instead of the obvious

if somecontainer:

so it's not as if pythonistas in general are blessed with some magical
"redundancy avoidance spell"...


Alex

 

[Nick Coghlan]

It can't be that hard to maintain the lambda code, why not just
leave it there for the minority of us who like the concept?

Because one of the points of Py3K is to clean up the language concepts and
syntax, and the current lambda just doesn't fit cleanly. If it was proposed in a
PEP, the syntax would be rejected as not being sufficiently Pythonic and for
being a castrated near-synonym for the def statement.

Now, Python 3K will retain the statement/expression distinction, so anonymous
functions will indeed be impossible without lambda - the link from a statement
to an expression will need to be made through the local namespace.

So, rather than pushing to retain lambda for Py3K, it might be more productive
to find a better statement -> expression translation for function definitions.
Guido seems to prefer named functions, so it would still be tough to gain his
acceptance. However, a more Pythonic syntax is the only way I can see anonymous
functions making into 3.0

The current best example of a statement->expression translation is generator
expressions:

def squares(seq)
for x in seq:
yield x * x

total = sum(squares(seq))

versus:

total = sum(x * x for x in seq)

If we consider a function definition (omitting decorators and docstrings) we get:

def foo(a, b, c):
return f(a) + o(b) - o(c)

accepts_func(foo)

What would a Pythonic 'function as expression' look like?

Perhaps something like:

accepts_func( (def (a, b, c) to f(a) + o(b) - o(c)) )

Here we simply omit the function name and use 'to' instead of ':' (the colon is
omitted to avoid making our expression look like it might be a statement). We
also don't need a return statement, since our target is an expression. The
surrounding parentheses would be required.

The "def (arg-tuple) to (result)" form is intended to show that we're talking
about a function in the mathematical sense of a single expression, rather than
the general Python sense of a suite of statements. That is, it has the same
behaviour as the current lambda - if you want a real Python function, write a
real Python function

Personally, I don't see the restriction of anonymous functions to single
expressions any more of a wart than the restriction of the value portion of a
generator expression to a single expression. If an algorithm is too complex for
a single expression, it's probably worth giving a name.

Some more examples:

(def (x) to x * x) # Map values to their squares
(def () to x) # No arguments, always map to x
(def (*a, **k) to x.bar(*a, **k)) # Look up the method now, call it later

And in combination with a generator expression:

( (def () to x(*a, **k)) for x, a, k in funcs_and_args_list)

Replacing the 'to' with -> might actually read better:

(def (a, b, c) -> f(a) + o(b) - o(c))
(def (x) -> x * x)
(def () -> x)
(def (*a, **k) -> x.bar(*a, **k))
( (def () -> x(*a, **k)) for x, a, k in func_list)

Anyway, thats just some ideas if you're concerned about the plan to have lambda
disappear in 3K.

Cheers,
Nick.

 

[Fredrik Lundh]

I think I'll scream, though not quite as loud as for my next seeing:

map(lambda x: f(x), ...

instead of

map(f, ...

note that if you replace "map" with "some function that takes a callable", the difference
between these two constructs may be crucially important.

</F>

 

[Peter Otten]

if len(somecontainer) > 0:

instead of the obvious

if somecontainer:

so it's not as if pythonistas in general are blessed with some magical
"redundancy avoidance spell"...

That is not always equivalent:

z = Numeric.zeros(5)
z array([0, 0, 0, 0, 0])
bool(z) False
len(z) > 0

True

Peter

 

[Alex Martelli]

note that if you replace "map" with "some function that takes a callable",
the difference between these two constructs may be crucially important.

Sure -- if global name f gets rebound during the execution of the ``some
function'' (or if that function stashes the callable away, and that name
gets rebound later, etc), the semantics of passing f are different -- to
get the same semantics with a lambda, you'd have to use the old trick:

somefunc(lambda x, f=f: f(x), ...


I don't recall ever having seen the late-binding semantics (of the
lambda I originally showed) ``used properly'' -- I _have_ sometimes seen
that semantics cause subtle bugs, though. Do you have any real-life
examples where expecting and allowing for rebinding of global name `f'
is proper and desirable behavior? Being able to show a "good" use of
this late-binding could be helpful, if I knew of any.


Alex