Securing a future for anonymous functions in Python

[Ian Bicking]

This is my primary use case for lambda's nowadays as well - typically
just to provide a way to convert the input to a callback into a call
to some other routine. I do a lot of Twisted stuff, whose deferred
objects make heavy use of single parameter callbacks, and often you
just want to call the next method in sequence, with some minor change
(or to ignore) the last result.

So for example, an asynchronous sequence of operations might be like:

d = some_deferred_function()
d.addCallback(lambda x: next_function())
d.addCallback(lambda blah: third_function(otherargs, blah))
d.addCallback(lambda x: last_function())

Steven proposed an ignoreargs function, and the partial function offers
the other side (http://www.python.org/peps/pep-0309.html). So this
would become:

d = some_deferred_function()
d.addCallback(ignoreargs(next_function, 1))
d.addCallback(partial(third_function, otherargs))
d.addCallback(ignoreargs(last_function, 1))

I'm not sure this is "better" than it is with lambda. It's actually
considerably less readable to me. Hmm... well, that makes me less
excited about those...

 

[Scott David Daniels]

So for example, an asynchronous sequence of operations might be like:

d = some_deferred_function()
d.addCallback(lambda x: next_function())
d.addCallback(lambda blah: third_function(otherargs, blah))
d.addCallback(lambda x: last_function())

which to me is more readable (in terms of seeing the sequence of
operations being performed in their proper order), then something like:

def cb_next(x):
return next_function()
def cb_third(blah, otherargs):
return third_function(otherargs, blah)
def cb_last(x):
return last_function()

d = some_deferred_function()
d.addCallback(cb_next)
d.addCallback(cb_third, otherargs)
d.addCallback(cb_next)

which has an extra layer of naming (the callback functions), and
requires more effort to follow the flow of what is really just a simple
sequence of three functions being called.

But this sequence contains an error of the same form as the "fat":

while test() != False:
...code...

The right sequence using lambda is:
d = some_deferred_function()
d.addCallback(next_function)
d.addCallback(lambda blah: third_function(otherargs, blah))
d.addCallback(last_function)

And I would write it as:

def third_function_fixed_blah(blah):
def call_third(otherargs):
return third_function(otherargs, blah)
return call_third

d = some_deferred_function()
d.addCallback(next_function)
d.addCallback(third_function_fixed_blah, otherargs)
d.addCallback(last_function)

The name gives you the chance to point out that the argument order is
tweaked. In many such cases, I use curry (ASPN recipe #52549), which
should show up in Python as "partial" in the "functional" module
according to PEP 309 <http://www.python.org/peps/pep-0309.html>
(accepted but not included). I suppose it will show up in Python 2.5.

Programming is a quest is for clear, easy-to-read code, not quick,
easy-to-write code. Choosing a name is a chance to explain what you
are doing. lambda is used too often in lieu of deciding what to write.

--Scott David Daniels
(e-mail address removed)

 

[David Bolen]

But this sequence contains an error of the same form as the "fat":

"this" being which of the two scenarios you quote above?

while test() != False:
...code...

I'm not sure I follow the "error" in this snippet...

The right sequence using lambda is:
d = some_deferred_function()
d.addCallback(next_function)
d.addCallback(lambda blah: third_function(otherargs, blah))
d.addCallback(last_function)

By what metric are you judging "right"?

In my scenario, the functions next_function and last_function are not
written to expect any arguments, so they can't be passed straight into
addCallback because any deferred callback will automatically receive
the result of the prior deferred callback in the chain (this is how
Twisted handles asynchronous callbacks for pending operations).
Someone has to absorb that argument (either the lambda, or
next_function itself, which if it is an existing function, needs to be
handled by a wrapper, ala my second example).

Your "right" sequence simply isn't equivalent to what I wrote.
Whether or not next_function is fixable to be used this way is a
separate point, but then you're discussing two different scenarios,
and not two ways to write one scenario.

-- David

 

[Scott David Daniels]

I'm not sure I follow the "error" in this snippet...

The code is "fat" -- clearer is:
while test():
...code...

By what metric are you judging "right"?

By a broken metric that requires you to mis-understand the original code
in the same way that I did. It was an idiotic response that required
more careful reading than I am doing this morning. The thing I've seen
in too much code (and though I saw in your code) is code like:

requires_function(lambda: function())

rather than:

requires_function(function)

It happens quite often, and I'm sure you've seen it. But I got your
code wrong, and for that I apologize.

--Scott David Daniels
(e-mail address removed)

 

[Simo Melenius]

Closure is the name for the whole thing, apparently, not just the
environment the procedure body needs, which was the aspect that I
(mis)attached the name to.

Which brings me to the point where I'd welcome more flexibility in
writing to variables outside the local scope. This limitation most
often kicks in in closed-over code in function objects, although it's
a more general issue in Python's scoping.

As we know, you can't write to variables that are both non-local and
non-global (globals you can declare "global"). Now that effectively
makes free variables read-only (although, the objects they point to
can _still_ be mutated).

Allowing write access to variables in a closed-over lexical scope
outside the innermost scope wouldn't hurt because:

1) if you need it, you can already do it -- just practice some
cumbersome tricks make suitable arrangements (e.g. the classical
accumulator example uses an array to hold the counter value instead
of binding it directly to the free variable;

2) if you don't need or understand it, you don't have to use it;

3) and at least in function instances: if you accidentally do, it'll
change the bindings within your closure only which is definitely
less dangerous than mutating objects that are bound inside the
closure.

It must be noted, however, that such behaviour would change the way of
hiding nested variable names:

Now it's safe (though maybe lexically confusing) to use the same
variable names in inner functions. This could happen with common names
for temporary variables like "i", "x", "y".

On the other hand, one could introduce a way to declare variables from
global scope or from local scope, with default from lexical scope. (If
you want to explicitly hide an outer binding, you'd declare "local
foo", for example. You can already do "global foo".)

I see what you are saying (I think), but I think I'd still like a
full anonymous def, whatever adapter you come up with. And I prefer
to be persuaded ;-)

I elaborated on this one in a post a few days ago. Indeed, it is
mostly a minor issue that _can_ be worked around(1). The problem is
that it eventually becomes irritating, when repeated all the time, to
name functions even if the name isn't used elsewhere.

It also creates an implicit dependency from the function call (one of
whose arguments points to the once-named function) to the once-named
function. That is, when you refactor some of your code, you must keep
two things paired all the time in your cut+paste maneuvers.


br,
S

(1) Everything can be worked around. In contrast: you can work around
the lack of a syntactic language by typing in machine code manually.
Sound stupid? Yes, it was done decades ago. How about using C to write
a "shell script" equivalent that you need, as a workaround to the
problem of lacking a shell? Stupid? Yes, but less -- it's been done.
How about writing callbacks by passing in a function pointer and a
data pointer, if you don't want to use a language like Python that
automates that task for you? Stupid? Yes, but not much -- it's been
done all the time. How about implementing an _anonymous_ function by
naming it, if you can't do it otherwise?

 

[Simo Melenius]

But I do think there's other ways to approach this. Function
expressions could get really out of hand, IMHO, and could easily lead
to twenty-line "expressions". That's aesthetically incompatible with
Python source, IMHO.

You can already write unaesthetic hundred-line Python functions, if
you want to. Python's syntax doesn't yet impose a restriction on the
number of sequential statements It sounds artificial to impose
such restrictions on these hypothetical "inline blocks", even if by
only allowing them to be plain expressions.

IMHO, the most pythonic way to write an "inline-block" is by reusing
existing keywords, using Python-like start-of-blocks and ending it by
indentation rules:

map (def x:
if foo (x):
return baz_1 (x)
elif bar (x):
return baz_2 (x)
else:
global hab
hab.append (x)
return baz_3 (hab),
[1,2,3,4,5,6])

and for one-liners:

map (def x: return x**2, [1,2,3,4,5,6])

As a side-effect, we also

- got rid of the "lambda" keyword;

- strenghtened the semantics of "def": a "def" already defines a
function so it's only logical to use it to define anonymous
functions, too;

- unified the semantics: function is always a function, and functions
return values by using "return". When learning Python, I learned the
hard way that "lambda"s are expressions, not functions. I'd pay the
burden of writing "return" more often in exchange for better
consistency.


my two cents,
br,
S

 

[Steven Bethard]

map (def x:
if foo (x):
return baz_1 (x)
elif bar (x):
return baz_2 (x)
else:
global hab
hab.append (x)
return baz_3 (hab),
[1,2,3,4,5,6])

I think this would probably have to be written as:

map (def x:
if foo(x):
return baz_1(x)
elif bar(x):
return baz_2(x)
else:
global hab
hab.append(x)
return baz_3(hab)
, [1,2,3,4,5,6])

or:

map (def x:
if foo(x):
return baz_1(x)
elif bar(x):
return baz_2(x)
else:
global hab
hab.append(x)
return baz_3(hab)
,
[1,2,3,4,5,6])

Note the placement of the comma. As it is,
return baz_3(hab),
returns the tuple containing the result of calling baz_3(hab):

py> def f(x):
.... return float(x),
....
py> f(1)
(1.0,)

It's not horrible to have to put the comma on the next line, but it
isn't as pretty as your version that doesn't. Unfortunately, I don't
think anyone's gonna want to revise the return statement syntax just to
introduce anonymous functions.

Steve

 

[Simo Melenius]

map (def x:
if foo (x):
return baz_1 (x)
elif bar (x):
return baz_2 (x)
else:
global hab
hab.append (x)
return baz_3 (hab),
[1,2,3,4,5,6])

I think this would probably have to be written as: ....
return baz_3(hab)
, [1,2,3,4,5,6])
or: ....
return baz_3(hab)
,
[1,2,3,4,5,6])

Note the placement of the comma. As it is,
return baz_3(hab),
returns the tuple containing the result of calling baz_3(hab):

That one didn't occur to me; creating a one-item tuple with (foo,) has
been odd enough for me: only few times I've seen also the parentheses
omitted.

I did ponder the unambiguousness of the last line, though. One could
suggest a new keyword like "end", but keyword bloat is bad.

(Of course, if we trade the "lambda" keyword for another, new keyword
we're not exactly _adding_ keywords... )

It's not horrible to have to put the comma on the next line, but it
isn't as pretty as your version that doesn't. Unfortunately, I don't
think anyone's gonna want to revise the return statement syntax just
to introduce anonymous functions.

There might not be a return statement: the anonymous function might
conditionally return earlier and have side-effects at the end of the
block (to implicitly return None). So the block-ending would need to
fit after any statement and be strictly unambiguous.


br,
S

 

[Doug Holton]

map (def x:
if foo (x):
return baz_1 (x)
elif bar (x):
return baz_2 (x)
else:
global hab
hab.append (x)
return baz_3 (hab),
[1,2,3,4,5,6])

I think this would probably have to be written as:

Right the comma plus other things make this difficult for a parser to
handle correctly. Other people have already come up with working solutions.

We have a special way to pass a multiline closure as a parameter to a
function. Put it outside the parameter list.

First, the single-line way using curly braces:

newlist = map({x as int | return x*x*x}, [1,2,3,4,5,6])

Then the multi-line way. I had to add an overload of map to support
reversing the order of parameters (list first, then the closure):

newlist = map([1,2,3,4,5,6]) def (x as int):
return x*x*x

for item in newlist:
print item

 

[Bengt Richter]

map (def x:
if foo (x):
return baz_1 (x)
elif bar (x):
return baz_2 (x)
else:
global hab
hab.append (x)
return baz_3 (hab),
[1,2,3,4,5,6])

I think this would probably have to be written as: ...
return baz_3(hab)
, [1,2,3,4,5,6])
or: ...
return baz_3(hab)
,
[1,2,3,4,5,6])

Note the placement of the comma. As it is,
return baz_3(hab),
returns the tuple containing the result of calling baz_3(hab):

That one didn't occur to me; creating a one-item tuple with (foo,) has
been odd enough for me: only few times I've seen also the parentheses
omitted.

I did ponder the unambiguousness of the last line, though. One could
suggest a new keyword like "end", but keyword bloat is bad.

ISTM you don't need "end" -- just put the def expression in parens,
and let the closing paren end it, e.g.:

map((def x:
if foo (x):
return baz_1 (x)
elif bar (x):
return baz_2 (x)
else:
global hab
hab.append (x)
return baz_3 (hab)), [1,2,3,4,5,6])

(Of course, if we trade the "lambda" keyword for another, new keyword
we're not exactly _adding_ keywords... )


There might not be a return statement: the anonymous function might
conditionally return earlier and have side-effects at the end of the
block (to implicitly return None). So the block-ending would need to
fit after any statement and be strictly unambiguous.

Just use parens as necessary or when in doubt ;-)

Regards,
Bengt Richter

 

[Terry Reedy]

(e-mail address removed) (Bengt Richter) writes:
Which brings me to the point where I'd welcome more flexibility in
writing to variables outside the local scope.

This idea was discussed extensively on PyDev perhaps 2 years ago. (You can
check the archived summaries if interested enough ;-) As I remember, there
was no consensus either on the desireability of such a mechanism or on the
syntax (several were proposed).

Terry J. Reedy

 

[Simo Melenius]

ISTM you don't need "end" -- just put the def expression in parens,
and let the closing paren end it, e.g.:

I first rejected any parens as not being native to how
classes/toplevel functions/control blocks are written in Python.
However, this looks quite clear to me. I'd expect that in most cases
it would be difficult to mistake a block of statements in parentheses
with e.g. tuple notation. And parensing works with the old "lambda"
already, of course.


br,
S

 

[Simo Melenius]

Oops, I found a typo alreay. I meant to write "def (x):" -- no name
for anonymous functions but just the argument list, please

Right the comma plus other things make this difficult for a parser to
handle correctly. Other people have already come up with working

It is difficult since in Python we don't enjoy the ugly luxury of
stuffing statements inside {}s. But as Bengt pointed out, parentheses
don't actually look bad at all in this case.

Then the multi-line way. I had to add an overload of map to support
reversing the order of parameters (list first, then the closure):

newlist = map([1,2,3,4,5,6]) def (x as int):
return x*x*x

That doesn't seem to scale if you want to pass more than one anonymous
function arguments to the function or call an arbitrary function with
parameters in arbitrary order.


br,
S

 

[Paul Rubin]

Do you consider generator expressions or list comprehensions deficient
because they don't allow several statements in the body of the for
loop?

I don't see what it would mean to do otherwise.

 

[Nick Coghlan]

I don't see what it would mean to do otherwise.

Exactly the same as a suite would in the innermost portion of the equivalent
explicit generator (i.e. where the implicit "yield <expr>" currently ends up).

If you could put a suite inside a function expression (i.e. replacing the
implicit "return <expr>") why not inside generator expressions as well?

Cheers,
Nick.

 

[Alan Gauld]

GvR has commented that he want to get rid of the lambda keyword for Python 3.0.
Getting rid of lambda seems like a worthy goal,

Can I ask what the objection to lambda is?
1) Is it the syntax?
2) Is it the limitation to a single expression?
3) Is it the word itself?

I can sympathise with 1 and 2 but the 3rd seems strange since a
lambda is a well defined name for an anonymous function used in
several programming languages and originating in lambda calculus
in math. Lambda therefore seems like a pefectly good name to
choose.

So why not retain the name lambda but extend or change the syntax
to make it more capable rather than invent a wholly new syntax
for lambdas?

Slightly confused, but since I only have time to read these
groups regularly when I'm at home I have probably missed the bulk
of the discussion over the years.

Alan G.
Author of the Learn to Program website
http://www.freenetpages.co.uk/hp/alan.gauld

 

[Jeff Shannon]

Can I ask what the objection to lambda is?
1) Is it the syntax?
2) Is it the limitation to a single expression?
3) Is it the word itself?

I can sympathise with 1 and 2 but the 3rd seems strange since a
lambda is a well defined name for an anonymous function used in
several programming languages and originating in lambda calculus
in math. Lambda therefore seems like a pefectly good name to
choose.

I think that the real objection is a little bit of 1), and something
that's kinda close to 2), but has nothing to do with 3).

The issue isn't that lambdas are bad because they're limited to a
single expression. The issue is that they're an awkward special case
of a function, which was added to the language to mollify
functional-programming advocates but which GvR never felt really "fit"
into Python. Other, more pythonic functional-programming features
have since been added (like list comprehensions and iterators).

It seems to me that in other, less-dynamic languages, lambdas are
significantly different from functions in that lambdas can be created
at runtime. In Python, *all* functions are created at runtime, and
new ones can be defined at any point in execution, so lambdas don't
get that advantage. Thus, their advantages are limited to the fact
that they're anonymous (but names are treated differently in Python
than in most other languages, so this is of marginal utility), and
that they can be created inline. This last bit makes them suitable
for creating quick closures (wrapping a function and tweaking its
parameters/return values) and for creating a delayed-execution object
(e.g. callbacks), so there's a lot of pressure to keep them, but
they're still a special case, and "special cases aren't special enough
to break the rules".

Jeff Shannon
Technician/Programmer
Credit International

 

[Paul Rubin]

It seems to me that in other, less-dynamic languages, lambdas are
significantly different from functions in that lambdas can be created
at runtime.

What languages are those, where you can create anonymous functions
at runtime, but not named functions?! That notion is very surprising
to me.

 

[Doug Holton]

Can I ask what the objection to lambda is?
1) Is it the syntax?
2) Is it the limitation to a single expression?
3) Is it the word itself?

I can sympathise with 1 and 2 but the 3rd seems strange since a
lambda is a well defined name for an anonymous function used in
several programming languages and originating in lambda calculus
in math. Lambda therefore seems like a pefectly good name to
choose.

I agree with keeping lambda functionality, and I don't care what name is
used, but there are people who do not like "lambda":
http://lambda-the-ultimate.org/node/view/419#comment-3069
The word "lambda" is meaningless to most people. Of course so is "def",
which might be why Guido van Robot changed it to "define":
http://gvr.sourceforge.net/screen_shots/

Even a simple word like "type" can be difficult to explain to beginners:
http://lambda-the-ultimate.org/node/view/337

Python is easier for beginners to learn than other mainstream
programming languages (like java or C++), but that's not to say it
doesn't have some stumbling blocks for beginners of course:
http://www.linuxjournal.com/article/5028

So why not retain the name lambda but extend or change the syntax
to make it more capable rather than invent a wholly new syntax
for lambdas?

Yes, I agree, and either keep the "lambda" keyword or else reuse the
"def" keyword for anonymous methods. See this page Steven Bethard
created: http://www.python.org/moin/AlternateLambdaSyntax

I really don't think anyone should worry about lambda disappearing.

By the way, you've done great work with your learning to program site
and all the help you've given on the python-tutor list:

 

[Jeff Shannon]

What languages are those, where you can create anonymous functions
at runtime, but not named functions?! That notion is very surprising
to me.

Hm, I should have been more clear that I'm inferring this from things
that others have said about lambdas in other languages; I'm sadly
rather language-deficient (especially as regards *worthwhile*
languages) myself. This particular impression was formed from a
recent-ish thread about lambdas....

http://groups-beta.google.com/group...1bd9f2?thread_id=3afee62f7ed7094b&mode=thread

(line-wrap's gonna mangle that, but it's all one line...)

Looking back, I see that I've mis-stated what I'd originally
concluded, and that my original conclusion was a bit questionable to
begin with. In the referenced thread, it was the O.P.'s assertion
that lambdas made higher-order and dynamic functions possible. From
this, I inferred (possibly incorrectly) a different relationship
between functions and lambdas in other (static) languages than exists
in Python.

Jeff Shannon
Technician/Programmer
Credit International