What are python closures realy like?

[Karl Kofnarson]

Hi,
while writing my last program I came upon the problem
of accessing a common local variable by a bunch of
functions.
I wanted to have a function which would, depending on
some argument, return other functions all having access to
the same variable. An OO approach would do but why not
try out closures...
So here is a simplified example of the idea:
def fun_basket(f):
common_var = [0]
def f1():
print common_var[0]
common_var[0]=1
def f2():
print common_var[0]
common_var[0]=2
if f == 1:
return f1
if f == 2:
return f2
If you call f1 and f2 from the inside of fun_basket, they
behave as expected, so common_var[0] is modified by
whatever function operates on it.
However, calling f1=fun_basket(1); f2 = fun_basket(2) and
then f1(); f2() returns 0 and 0. It is not the way one would
expect closures to work, knowing e.g. Lisp make-counter.
Any ideas what's going on behind the scene?

 

[Felipe Almeida Lessa]

def fun_basket(f):
common_var = [0]
def f1():
print common_var[0]
common_var[0]=1
def f2():
print common_var[0]
common_var[0]=2
if f == 1:
return f1
if f == 2:
return f2

Everytime you call fun_basket you create another common_var.

However, calling f1=fun_basket(1); f2 = fun_basket(2) and
then f1(); f2() returns 0 and 0.

Two calls to fun_basket, two different common_var's, two f1's and two
f2's. Each f1/f2 pair have access to a different common_var, so it's
working as expected. To work as you expected, fun_basket should be on
the same block common_var is defined.

 

[Klaas]

Hi,
while writing my last program I came upon the problem
of accessing a common local variable by a bunch of
functions.
I wanted to have a function which would, depending on
some argument, return other functions all having access to
the same variable. An OO approach would do but why not
try out closures...
So here is a simplified example of the idea:
def fun_basket(f):
common_var = [0]
def f1():
print common_var[0]
common_var[0]=1
def f2():
print common_var[0]
common_var[0]=2
if f == 1:
return f1
if f == 2:
return f2
If you call f1 and f2 from the inside of fun_basket, they
behave as expected, so common_var[0] is modified by
whatever function operates on it.
However, calling f1=fun_basket(1); f2 = fun_basket(2) and
then f1(); f2() returns 0 and 0. It is not the way one would
expect closures to work, knowing e.g. Lisp make-counter.
Any ideas what's going on behind the scene?

Python can be read quite literally. "common_var" is a local variable
to fun_basket, hence it independent among invokations of fun_basket.
"def" is a statement that creates a function when it is executed. If
you execute the same def statement twice, two different functions are
created. Running fun_basket twice creates four closures, and the first
two have no relation to the second two. The two sets close over
different cell variables.

If you want to share data between function invokation, you need an
object which persists between calls. You can use a global variable, or
a default argument. But since the value is shared everytime the
function is called, I don't see the value in using a closure. I don't
know lisp very well, but in my mind the whole point of closures is that
you can reference a different unique cell each time.

-MIke

 

[Paul McGuire]

Hi,
while writing my last program I came upon the problem
of accessing a common local variable by a bunch of
functions.
I wanted to have a function which would, depending on
some argument, return other functions all having access to
the same variable. An OO approach would do but why not
try out closures...
So here is a simplified example of the idea:
def fun_basket(f):
common_var = [0]
def f1():
print common_var[0]
common_var[0]=1
def f2():
print common_var[0]
common_var[0]=2
if f == 1:
return f1
if f == 2:
return f2

Karl,

Usually when using this idiom, fun_basket would return a tuple of all of the
defined functions, rather than one vs. the other. So in place of:

if f == 1:
return f1
if f == 2:
return f2 Just do
return f1, f2

(For that matter, the argument f is no longer needed either.)

Then your caller will get 2 functions, who share a common var. You don't
call fun_basket any more, you've already created your two "closures". Call
fun_basket using something like:

z1,z2 = fun_basket(None)

And then call z1() and z2() at your leisure - they should have the desired
behavior.

-- Paul

 

[Carl Banks]

Hi,
while writing my last program I came upon the problem
of accessing a common local variable by a bunch of
functions.
I wanted to have a function which would, depending on
some argument, return other functions all having access to
the same variable. An OO approach would do but why not
try out closures...
So here is a simplified example of the idea:
def fun_basket(f):
common_var = [0]
def f1():
print common_var[0]
common_var[0]=1
def f2():
print common_var[0]
common_var[0]=2
if f == 1:
return f1
if f == 2:
return f2
If you call f1 and f2 from the inside of fun_basket, they
behave as expected, so common_var[0] is modified by
whatever function operates on it.
However, calling f1=fun_basket(1); f2 = fun_basket(2) and
then f1(); f2() returns 0 and 0. It is not the way one would
expect closures to work, knowing e.g. Lisp make-counter.

Lisp works the same way.

* (defun fun_basket (f)
(let ((common_var 0))
(defun f1 ()
(print common_var)
(setf common_var 1))
(defun f2 ()
(print common_var)
(setf common_var 2))
(if (eq f 1)
#'f1
#'f2)))

FUN_BASKET
* (setf (symbol-function 'f1a) (fun_basket 1))

; Converted F1.
; Converted F2.

#<Interpreted Function F1 {5807C9A1}>
* (setf (symbol-function 'f2a) (fun_basket 2))

#<Interpreted Function F2 {5807D409}>
* (f1a)

0
1
* (f2a)

0
2

Any ideas what's going on behind the scene?

Every time you call the function, a new closure is created.


Carl Banks

 

[Karl Kofnarson]

Karl,

Usually when using this idiom, fun_basket would return a tuple of all of the
defined functions, rather than one vs. the other. So in place of:
(For that matter, the argument f is no longer needed either.)

Then your caller will get 2 functions, who share a common var. You don't
call fun_basket any more, you've already created your two "closures". Call
fun_basket using something like:

z1,z2 = fun_basket(None)

And then call z1() and z2() at your leisure - they should have the desired
behavior.

-- Paul

Thanks a lot Paul and for the other answers. The things are now
clear to me. In fact, in the Lisp example that I mentioned, you
get a list (or let it be association list) of the internal
functions. Then you can call them separately and they work as
you expect but it's due to the fact only that you got them created
at the same time.

 

[Paddy]

Thanks a lot Paul and for the other answers. The things are now
clear to me. In fact, in the Lisp example that I mentioned, you
get a list (or let it be association list) of the internal
functions. Then you can call them separately and they work as
you expect but it's due to the fact only that you got them created
at the same time.

I played around a bit. The following is a 'borg' version in that there
is only one counter shared between all calls of the outer function:
.... def borg_var_inc(x=1):
.... fun_borg_var._n += x
.... return fun_borg_var._n
.... def borg_var_dec(x=1):
.... fun_borg_var._n -= x
.... return fun_borg_var._n
.... try:
.... fun_borg_var._n = fun_borg_var._n
.... except:
.... fun_borg_var._n = initial_val
.... return (borg_var_inc, borg_var_dec)
....

- Paddy.

 

[Fredrik Lundh]

I played around a bit. The following is a 'borg' version in that there
is only one counter shared between all calls of the outer function:

... def borg_var_inc(x=1):
... fun_borg_var._n += x

a drawback with the function attribute approach compared to a real closure
is that the function is no longer a self-contained callable:

def fun_borg_var(initial_val=0):
def borg_var_inc(x=1):
fun_borg_var._n += x
return fun_borg_var._n
def borg_var_dec(x=1):
fun_borg_var._n -= x
return fun_borg_var._n
try:
fun_borg_var._n = fun_borg_var._n
except:
fun_borg_var._n = initial_val
return (borg_var_inc, borg_var_dec)

up1, dn1 = fun_borg_var()

del fun_borg_var # won't need this any more

print up1() # oops!

so you might as well use a good old global variable, and initialize it as
usual.

</F>

 

[Paul Boddie]

I wanted to have a function which would, depending on
some argument, return other functions all having access to
the same variable. An OO approach would do but why not
try out closures...

I know that everyone will say that Python is a "multi-paradigm"
language and that one should feel free to use whatever technique seems
appropriate to solve the problem at hand, but it seems to me that
there's been an explosion in nested function usage recently, with lots
of code snippets showing them off either in the context of a debugging
exercise or as a proposed solution to a problem, and yet in many cases
their usage seems frivolous in comparison to plain old object-oriented
techniques.

I'm not pointing the finger at you here, Karl, since you seem to be
experimenting with closures, but why are they suddenly so fashionable?
Haven't the features supporting them existed in Python for a few
versions now? Don't people want to write classes any more?

Intrigued,

Paul

 

[Fredrik Lundh]

I know that everyone will say that Python is a "multi-paradigm"
language and that one should feel free to use whatever technique seems
appropriate to solve the problem at hand, but it seems to me that
there's been an explosion in nested function usage recently, with lots
of code snippets showing them off either in the context of a debugging
exercise or as a proposed solution to a problem, and yet in many cases
their usage seems frivolous in comparison to plain old object-oriented
techniques.

when doing some heavy optimization, I recently found myself writing:

def foobar(arg1, arg2, arg3):
def helper(arg):
do something with arg1 and argument
def foo():
do something with arg1 and arg3 and
call helper
def bar():
do something with arg1 and arg2
def zoo():
do something with arg2 and arg3 and
call helper
# oops; how do I return all these?
class bag(object):
pass
bag = bag()
bag.foo = foo
bag.bar = bar
bag.zoo = zoo
return bag

which, I think, deserves no further comment...

</F>

 

[Tim Chase]

def foobar(arg1, arg2, arg3):

def helper(arg):
do something with arg1 and argument
def foo():
do something with arg1 and arg3 and
call helper
def bar():
do something with arg1 and arg2
def zoo():
do something with arg2 and arg3 and
call helper
# oops; how do I return all these?
class bag(object):
pass
bag = bag()
bag.foo = foo
bag.bar = bar
bag.zoo = zoo
return bag

which, I think, deserves no further comment...

Could you please explain your reasoning behind why you opted to
create the bag, fill it, and then return it? Usually I see
something like

return foo,bar,zoo

and it would be helpful to understand the reasoning behind why
one would choose one over the other. Some of the off-the-cuff
thoughts in my trying to understand it:

-defining a bag, instantiating a bag and filling it takes a few
extra cycles for each call of foobar() so the returned tuple
should be a hair faster (though perhaps defining a bag class
outside the foobar() function might trim some of this?)

-returning a tuple requires that any additional methods you might
opt to add/return in the future involve adjusting all your code,
whereas the bag method allows you to toss extra methods in the
bag without adjusting every statement that calls foobar()

Are either of these correct? Are there additional reasons I'm
missing?

Thanks,

-tkc

 

[Paul Boddie]

when doing some heavy optimization, I recently found myself writing:

def foobar(arg1, arg2, arg3):
def helper(arg):
do something with arg1 and argument
def foo():
do something with arg1 and arg3 and
call helper
def bar():
do something with arg1 and arg2
def zoo():
do something with arg2 and arg3 and
call helper
# oops; how do I return all these?
class bag(object):
pass
bag = bag()
bag.foo = foo
bag.bar = bar
bag.zoo = zoo
return bag

which, I think, deserves no further comment...

Have I missed something deep here, or could you not have written the
above as follows...?

class foobar(object):
def __init__(self, arg1, arg2, arg3):
self.arg1, self.arg2, self.arg3 = arg1, arg2, arg3
def helper(self, arg):
do something with arg1 and argument
def foo(self):
do something with arg1 and arg3 and
call helper
def bar(self):
do something with arg1 and arg2
def zoo(self):
do something with arg2 and arg3 and
call helper

There's certainly some boilerplate required (in both forms, though) and
you'd have to use self in various places, but the above looks less
contorted to me. I'd like to hear your further comment, however, since
the principal inconvenience of making a class seems to be in the
verbose initialisation and explicit self, the latter of which obviously
being a feature that you won't find me complaining about, though. ;-)

Paul

 

[Michele Simionato]

I'm not pointing the finger at you here, Karl, since you seem to be
experimenting with closures, but why are they suddenly so fashionable?
Haven't the features supporting them existed in Python for a few
versions now? Don't people want to write classes any more?

Intrigued,

Paul

I believe decorators are in large part responsible for that. A callable
object does not work
as a method unless you define a custom __get__, so in decorator
programming it is
often easier to use a closure. OTOH closures a not optimal if you want
persistency
(you cannot pickle a closure) so in that case I use a callable object
instead.

Michele Simionato

 

[Klaas]

I believe decorators are in large part responsible for that. A callable
object does not work
as a method unless you define a custom __get__, so in decorator
programming it is
often easier to use a closure. OTOH closures a not optimal if you want
persistency
(you cannot pickle a closure) so in that case I use a callable object
instead.

Note that it isn't necessary to write the descriptor yourself. The
'new' module takes care of it:

In [1]: class A(object):
...: pass
In [2]: a = A()
In [3]: class Method(object):
...: def __call__(mself, oself):
...: print mself, oself
In [4]: import new
In [5]: a.method = new.instancemethod(Method(), a, A)
In [6]: a.method()
<__main__.Method object at 0xb7ab7f6c> <__main__.A object at
0xb7ab79ec>

-Mike

 

[John Nagle]

I know that everyone will say that Python is a "multi-paradigm"
language and that one should feel free to use whatever technique seems
appropriate to solve the problem at hand, but it seems to me that
there's been an explosion in nested function usage recently, with lots
of code snippets showing them off either in the context of a debugging
exercise or as a proposed solution to a problem, and yet in many cases
their usage seems frivolous in comparison to plain old object-oriented
techniques.

Most of the examples given here are kind of silly, but closures have
real uses. I used one today in Javascript because I was writing an
AJAX application, and I was using an API, the standard XMLHttpRequestObject,
which required a callback function with no arguments. A closure allowed
the code to pass relevant information with the callback function, which
would be called when a request to the server completed. A global
variable wouldn't have worked, because multiple instances of the object
making the callback are possible.

It's a useful tool, but not one you need frequently. Don't get
carried away.

John Nagle
Animats

 

[Fredrik Lundh]

Most of the examples given here are kind of silly, but closures have
real uses. I used one today in Javascript because I was writing an
AJAX application, and I was using an API, the standard XMLHttpRequestObject,
which required a callback function with no arguments. A closure allowed
the code to pass relevant information with the callback function, which
would be called when a request to the server completed. A global
variable wouldn't have worked, because multiple instances of the object
making the callback are possible.

the usual way of handling multiple callback context instances in Python
is of course to create multiple instances of the class that implements
the behaviour, and use a bound method as the callback.

</F>

 

[Roy Smith]

Most of the examples given here are kind of silly, but closures have
real uses. I used one today in Javascript because I was writing an
AJAX application, and I was using an API, the standard XMLHttpRequestObject,
which required a callback function with no arguments. A closure allowed
the code to pass relevant information with the callback function, which
would be called when a request to the server completed. A global
variable wouldn't have worked, because multiple instances of the object
making the callback are possible.

As another example, from a large C++ project I'm currently working on, we
use closures in many places to push actions onto queues.

One kind of queue we have is a timed queue. You push a closure and a time
onto it, and it executes the closure at the appointed time. Another kind
of queue is for inter-thread communication. You push a closure onto a
thread's input queue when you want it to do something. The thread just
keeps popping closures off the other end of the queue and executing them.

Think of a closure as a way of saying "do this". You can say, "At 5:00, do
this", or "Do this as soon as you get a chance".

# pseudocode
class husband (spouse):
def pickUpGroceries():
etc
def addToQueue():
etc

c = makeClosure (husband.pickUpGroceries, [milk, eggs, bread])
h = husband()
h.addToQueue (c)