# What are python closures realy like?

Discussion in 'Python' started by Karl Kofnarson, Dec 1, 2006.

1. ### Karl KofnarsonGuest

Hi,
while writing my last program I came upon the problem
of accessing a common local variable by a bunch of
functions.
I wanted to have a function which would, depending on
the same variable. An OO approach would do but why not
try out closures...
So here is a simplified example of the idea:
common_var = [0]
def f1():
print common_var[0]
common_var[0]=1
def f2():
print common_var[0]
common_var[0]=2
if f == 1:
return f1
if f == 2:
return f2
If you call f1 and f2 from the inside of fun_basket, they
behave as expected, so common_var[0] is modified by
whatever function operates on it.
then f1(); f2() returns 0 and 0. It is not the way one would
expect closures to work, knowing e.g. Lisp make-counter.
Any ideas what's going on behind the scene?

Karl Kofnarson, Dec 1, 2006

2. ### Felipe Almeida LessaGuest

On 12/1/06, Karl Kofnarson <> wrote:
[snip]
> common_var = [0]
> def f1():
> print common_var[0]
> common_var[0]=1
> def f2():
> print common_var[0]
> common_var[0]=2
> if f == 1:
> return f1
> if f == 2:
> return f2

Everytime you call fun_basket you create another common_var.

> then f1(); f2() returns 0 and 0.

Two calls to fun_basket, two different common_var's, two f1's and two
f2's. Each f1/f2 pair have access to a different common_var, so it's
working as expected. To work as you expected, fun_basket should be on
the same block common_var is defined.

--
Felipe.

Felipe Almeida Lessa, Dec 1, 2006

3. ### KlaasGuest

Karl Kofnarson wrote:
> Hi,
> while writing my last program I came upon the problem
> of accessing a common local variable by a bunch of
> functions.
> I wanted to have a function which would, depending on
> the same variable. An OO approach would do but why not
> try out closures...
> So here is a simplified example of the idea:
> common_var = [0]
> def f1():
> print common_var[0]
> common_var[0]=1
> def f2():
> print common_var[0]
> common_var[0]=2
> if f == 1:
> return f1
> if f == 2:
> return f2
> If you call f1 and f2 from the inside of fun_basket, they
> behave as expected, so common_var[0] is modified by
> whatever function operates on it.
> then f1(); f2() returns 0 and 0. It is not the way one would
> expect closures to work, knowing e.g. Lisp make-counter.
> Any ideas what's going on behind the scene?

Python can be read quite literally. "common_var" is a local variable
"def" is a statement that creates a function when it is executed. If
you execute the same def statement twice, two different functions are
created. Running fun_basket twice creates four closures, and the first
two have no relation to the second two. The two sets close over
different cell variables.

If you want to share data between function invokation, you need an
object which persists between calls. You can use a global variable, or
a default argument. But since the value is shared everytime the
function is called, I don't see the value in using a closure. I don't
know lisp very well, but in my mind the whole point of closures is that
you can reference a different unique cell each time.

-MIke

Klaas, Dec 1, 2006
4. ### Paul McGuireGuest

"Karl Kofnarson" <> wrote in message
news...
> Hi,
> while writing my last program I came upon the problem
> of accessing a common local variable by a bunch of
> functions.
> I wanted to have a function which would, depending on
> the same variable. An OO approach would do but why not
> try out closures...
> So here is a simplified example of the idea:
> common_var = [0]
> def f1():
> print common_var[0]
> common_var[0]=1
> def f2():
> print common_var[0]
> common_var[0]=2
> if f == 1:
> return f1
> if f == 2:
> return f2

Karl,

Usually when using this idiom, fun_basket would return a tuple of all of the
defined functions, rather than one vs. the other. So in place of:
> if f == 1:
> return f1
> if f == 2:
> return f2

Just do
> return f1, f2

(For that matter, the argument f is no longer needed either.)

Then your caller will get 2 functions, who share a common var. You don't

And then call z1() and z2() at your leisure - they should have the desired
behavior.

-- Paul

Paul McGuire, Dec 1, 2006
5. ### Carl BanksGuest

Karl Kofnarson wrote:
> Hi,
> while writing my last program I came upon the problem
> of accessing a common local variable by a bunch of
> functions.
> I wanted to have a function which would, depending on
> the same variable. An OO approach would do but why not
> try out closures...
> So here is a simplified example of the idea:
> common_var = [0]
> def f1():
> print common_var[0]
> common_var[0]=1
> def f2():
> print common_var[0]
> common_var[0]=2
> if f == 1:
> return f1
> if f == 2:
> return f2
> If you call f1 and f2 from the inside of fun_basket, they
> behave as expected, so common_var[0] is modified by
> whatever function operates on it.
> then f1(); f2() returns 0 and 0. It is not the way one would
> expect closures to work, knowing e.g. Lisp make-counter.

Lisp works the same way.

(let ((common_var 0))
(defun f1 ()
(print common_var)
(setf common_var 1))
(defun f2 ()
(print common_var)
(setf common_var 2))
(if (eq f 1)
#'f1
#'f2)))

* (setf (symbol-function 'f1a) (fun_basket 1))

; Converted F1.
; Converted F2.

#<Interpreted Function F1 {5807C9A1}>
* (setf (symbol-function 'f2a) (fun_basket 2))

#<Interpreted Function F2 {5807D409}>
* (f1a)

0
1
* (f2a)

0
2

> Any ideas what's going on behind the scene?

Every time you call the function, a new closure is created.

Carl Banks

Carl Banks, Dec 1, 2006
6. ### Karl KofnarsonGuest

> Karl,
>
> Usually when using this idiom, fun_basket would return a tuple of all of the
> defined functions, rather than one vs. the other. So in place of:
>> if f == 1:
>> return f1
>> if f == 2:
>> return f2

> Just do
>> return f1, f2

> (For that matter, the argument f is no longer needed either.)
>
> Then your caller will get 2 functions, who share a common var. You don't
>
>
> And then call z1() and z2() at your leisure - they should have the desired
> behavior.
>
> -- Paul

Thanks a lot Paul and for the other answers. The things are now
clear to me. In fact, in the Lisp example that I mentioned, you
get a list (or let it be association list) of the internal
functions. Then you can call them separately and they work as
you expect but it's due to the fact only that you got them created
at the same time.

Karl Kofnarson, Dec 2, 2006

Karl Kofnarson wrote:

> > Karl,
> >
> > Usually when using this idiom, fun_basket would return a tuple of all of the
> > defined functions, rather than one vs. the other. So in place of:
> >> if f == 1:
> >> return f1
> >> if f == 2:
> >> return f2

> > Just do
> >> return f1, f2

> > (For that matter, the argument f is no longer needed either.)
> >
> > Then your caller will get 2 functions, who share a common var. You don't
> > fun_basket using something like:
> >
> >
> > And then call z1() and z2() at your leisure - they should have the desired
> > behavior.
> >
> > -- Paul

>
> Thanks a lot Paul and for the other answers. The things are now
> clear to me. In fact, in the Lisp example that I mentioned, you
> get a list (or let it be association list) of the internal
> functions. Then you can call them separately and they work as
> you expect but it's due to the fact only that you got them created
> at the same time.

I played around a bit. The following is a 'borg' version in that there
is only one counter shared between all calls of the outer function:

>>> def fun_borg_var(initial_val=0):

.... def borg_var_inc(x=1):
.... fun_borg_var._n += x
.... return fun_borg_var._n
.... def borg_var_dec(x=1):
.... fun_borg_var._n -= x
.... return fun_borg_var._n
.... try:
.... fun_borg_var._n = fun_borg_var._n
.... except:
.... fun_borg_var._n = initial_val
.... return (borg_var_inc, borg_var_dec)
....
>>> up1, dn1 = fun_borg_var() # get an inc/decrementer
>>> up1(0)

0
>>> up1()

1
>>> up1()

2
>>> dn1()

1
>>> dn1()

0
>>> dn1()

-1
>>> up2, dn2 = fun_borg_var() # get another inc/decrementer
>>> up2(0) # looks like the same _n

-1
>>> up2(3)

2
>>> up1(3)

5
>>>

8. ### Fredrik LundhGuest

> I played around a bit. The following is a 'borg' version in that there
> is only one counter shared between all calls of the outer function:
>
>>>> def fun_borg_var(initial_val=0):

> ... def borg_var_inc(x=1):
> ... fun_borg_var._n += x

a drawback with the function attribute approach compared to a real closure
is that the function is no longer a self-contained callable:

def fun_borg_var(initial_val=0):
def borg_var_inc(x=1):
fun_borg_var._n += x
return fun_borg_var._n
def borg_var_dec(x=1):
fun_borg_var._n -= x
return fun_borg_var._n
try:
fun_borg_var._n = fun_borg_var._n
except:
fun_borg_var._n = initial_val
return (borg_var_inc, borg_var_dec)

up1, dn1 = fun_borg_var()

del fun_borg_var # won't need this any more

print up1() # oops!

so you might as well use a good old global variable, and initialize it as
usual.

</F>

Fredrik Lundh, Dec 6, 2006
9. ### Paul BoddieGuest

Karl Kofnarson wrote:
>
> I wanted to have a function which would, depending on
> the same variable. An OO approach would do but why not
> try out closures...

I know that everyone will say that Python is a "multi-paradigm"
language and that one should feel free to use whatever technique seems
appropriate to solve the problem at hand, but it seems to me that
there's been an explosion in nested function usage recently, with lots
of code snippets showing them off either in the context of a debugging
exercise or as a proposed solution to a problem, and yet in many cases
their usage seems frivolous in comparison to plain old object-oriented
techniques.

I'm not pointing the finger at you here, Karl, since you seem to be
experimenting with closures, but why are they suddenly so fashionable?
Haven't the features supporting them existed in Python for a few
versions now? Don't people want to write classes any more?

Intrigued,

Paul

Paul Boddie, Dec 6, 2006
10. ### Fredrik LundhGuest

Paul Boddie wrote:

> I know that everyone will say that Python is a "multi-paradigm"
> language and that one should feel free to use whatever technique seems
> appropriate to solve the problem at hand, but it seems to me that
> there's been an explosion in nested function usage recently, with lots
> of code snippets showing them off either in the context of a debugging
> exercise or as a proposed solution to a problem, and yet in many cases
> their usage seems frivolous in comparison to plain old object-oriented
> techniques.

when doing some heavy optimization, I recently found myself writing:

def foobar(arg1, arg2, arg3):
def helper(arg):
do something with arg1 and argument
def foo():
do something with arg1 and arg3 and
call helper
def bar():
do something with arg1 and arg2
def zoo():
do something with arg2 and arg3 and
call helper
# oops; how do I return all these?
class bag(object):
pass
bag = bag()
bag.foo = foo
bag.bar = bar
bag.zoo = zoo
return bag

which, I think, deserves no further comment...

</F>

Fredrik Lundh, Dec 6, 2006
11. ### Tim ChaseGuest

> def foobar(arg1, arg2, arg3):
> def helper(arg):
> do something with arg1 and argument
> def foo():
> do something with arg1 and arg3 and
> call helper
> def bar():
> do something with arg1 and arg2
> def zoo():
> do something with arg2 and arg3 and
> call helper
> # oops; how do I return all these?
> class bag(object):
> pass
> bag = bag()
> bag.foo = foo
> bag.bar = bar
> bag.zoo = zoo
> return bag
>
> which, I think, deserves no further comment...

create the bag, fill it, and then return it? Usually I see
something like

return foo,bar,zoo

and it would be helpful to understand the reasoning behind why
one would choose one over the other. Some of the off-the-cuff
thoughts in my trying to understand it:

-defining a bag, instantiating a bag and filling it takes a few
extra cycles for each call of foobar() so the returned tuple
should be a hair faster (though perhaps defining a bag class
outside the foobar() function might trim some of this?)

-returning a tuple requires that any additional methods you might
whereas the bag method allows you to toss extra methods in the
bag without adjusting every statement that calls foobar()

Are either of these correct? Are there additional reasons I'm
missing?

Thanks,

-tkc

Tim Chase, Dec 6, 2006
12. ### Paul BoddieGuest

Fredrik Lundh wrote:
>
> when doing some heavy optimization, I recently found myself writing:
>
> def foobar(arg1, arg2, arg3):
> def helper(arg):
> do something with arg1 and argument
> def foo():
> do something with arg1 and arg3 and
> call helper
> def bar():
> do something with arg1 and arg2
> def zoo():
> do something with arg2 and arg3 and
> call helper
> # oops; how do I return all these?
> class bag(object):
> pass
> bag = bag()
> bag.foo = foo
> bag.bar = bar
> bag.zoo = zoo
> return bag
>
> which, I think, deserves no further comment...

Have I missed something deep here, or could you not have written the
above as follows...?

class foobar(object):
def __init__(self, arg1, arg2, arg3):
self.arg1, self.arg2, self.arg3 = arg1, arg2, arg3
def helper(self, arg):
do something with arg1 and argument
def foo(self):
do something with arg1 and arg3 and
call helper
def bar(self):
do something with arg1 and arg2
def zoo(self):
do something with arg2 and arg3 and
call helper

There's certainly some boilerplate required (in both forms, though) and
you'd have to use self in various places, but the above looks less
contorted to me. I'd like to hear your further comment, however, since
the principal inconvenience of making a class seems to be in the
verbose initialisation and explicit self, the latter of which obviously
being a feature that you won't find me complaining about, though. ;-)

Paul

Paul Boddie, Dec 6, 2006
13. ### Michele SimionatoGuest

Paul Boddie wrote:
> I'm not pointing the finger at you here, Karl, since you seem to be
> experimenting with closures, but why are they suddenly so fashionable?
> Haven't the features supporting them existed in Python for a few
> versions now? Don't people want to write classes any more?
>
> Intrigued,
>
> Paul

I believe decorators are in large part responsible for that. A callable
object does not work
as a method unless you define a custom __get__, so in decorator
programming it is
often easier to use a closure. OTOH closures a not optimal if you want
persistency
(you cannot pickle a closure) so in that case I use a callable object

Michele Simionato

Michele Simionato, Dec 6, 2006
14. ### KlaasGuest

Michele Simionato wrote:

> I believe decorators are in large part responsible for that. A callable
> object does not work
> as a method unless you define a custom __get__, so in decorator
> programming it is
> often easier to use a closure. OTOH closures a not optimal if you want
> persistency
> (you cannot pickle a closure) so in that case I use a callable object

Note that it isn't necessary to write the descriptor yourself. The
'new' module takes care of it:

In [1]: class A(object):
...: pass
In [2]: a = A()
In [3]: class Method(object):
...: def __call__(mself, oself):
...: print mself, oself
In [4]: import new
In [5]: a.method = new.instancemethod(Method(), a, A)
In [6]: a.method()
<__main__.Method object at 0xb7ab7f6c> <__main__.A object at
0xb7ab79ec>

-Mike

Klaas, Dec 6, 2006
15. ### John NagleGuest

Paul Boddie wrote:
> I know that everyone will say that Python is a "multi-paradigm"
> language and that one should feel free to use whatever technique seems
> appropriate to solve the problem at hand, but it seems to me that
> there's been an explosion in nested function usage recently, with lots
> of code snippets showing them off either in the context of a debugging
> exercise or as a proposed solution to a problem, and yet in many cases
> their usage seems frivolous in comparison to plain old object-oriented
> techniques.

Most of the examples given here are kind of silly, but closures have
real uses. I used one today in Javascript because I was writing an
AJAX application, and I was using an API, the standard XMLHttpRequestObject,
which required a callback function with no arguments. A closure allowed
the code to pass relevant information with the callback function, which
would be called when a request to the server completed. A global
variable wouldn't have worked, because multiple instances of the object
making the callback are possible.

It's a useful tool, but not one you need frequently. Don't get
carried away.

John Nagle
Animats

John Nagle, Dec 12, 2006
16. ### Fredrik LundhGuest

John Nagle wrote:

> Most of the examples given here are kind of silly, but closures have
> real uses. I used one today in Javascript because I was writing an
> AJAX application, and I was using an API, the standard XMLHttpRequestObject,
> which required a callback function with no arguments. A closure allowed
> the code to pass relevant information with the callback function, which
> would be called when a request to the server completed. A global
> variable wouldn't have worked, because multiple instances of the object
> making the callback are possible.

the usual way of handling multiple callback context instances in Python
is of course to create multiple instances of the class that implements
the behaviour, and use a bound method as the callback.

</F>

Fredrik Lundh, Dec 12, 2006
17. ### Roy SmithGuest

John Nagle <> wrote:
> Most of the examples given here are kind of silly, but closures have
> real uses. I used one today in Javascript because I was writing an
> AJAX application, and I was using an API, the standard XMLHttpRequestObject,
> which required a callback function with no arguments. A closure allowed
> the code to pass relevant information with the callback function, which
> would be called when a request to the server completed. A global
> variable wouldn't have worked, because multiple instances of the object
> making the callback are possible.

As another example, from a large C++ project I'm currently working on, we
use closures in many places to push actions onto queues.

One kind of queue we have is a timed queue. You push a closure and a time
onto it, and it executes the closure at the appointed time. Another kind
of queue is for inter-thread communication. You push a closure onto a
thread's input queue when you want it to do something. The thread just
keeps popping closures off the other end of the queue and executing them.

Think of a closure as a way of saying "do this". You can say, "At 5:00, do
this", or "Do this as soon as you get a chance".

# pseudocode
class husband (spouse):
def pickUpGroceries():
etc
etc

c = makeClosure (husband.pickUpGroceries, [milk, eggs, bread])
h = husband()