What are python closures realy like?

Discussion in 'Python' started by Karl Kofnarson, Dec 1, 2006.

  1. Hi,
    while writing my last program I came upon the problem
    of accessing a common local variable by a bunch of
    functions.
    I wanted to have a function which would, depending on
    some argument, return other functions all having access to
    the same variable. An OO approach would do but why not
    try out closures...
    So here is a simplified example of the idea:
    def fun_basket(f):
    common_var = [0]
    def f1():
    print common_var[0]
    common_var[0]=1
    def f2():
    print common_var[0]
    common_var[0]=2
    if f == 1:
    return f1
    if f == 2:
    return f2
    If you call f1 and f2 from the inside of fun_basket, they
    behave as expected, so common_var[0] is modified by
    whatever function operates on it.
    However, calling f1=fun_basket(1); f2 = fun_basket(2) and
    then f1(); f2() returns 0 and 0. It is not the way one would
    expect closures to work, knowing e.g. Lisp make-counter.
    Any ideas what's going on behind the scene?
     
    Karl Kofnarson, Dec 1, 2006
    #1
    1. Advertising

  2. On 12/1/06, Karl Kofnarson <> wrote:
    [snip]
    > def fun_basket(f):
    > common_var = [0]
    > def f1():
    > print common_var[0]
    > common_var[0]=1
    > def f2():
    > print common_var[0]
    > common_var[0]=2
    > if f == 1:
    > return f1
    > if f == 2:
    > return f2


    Everytime you call fun_basket you create another common_var.

    > However, calling f1=fun_basket(1); f2 = fun_basket(2) and
    > then f1(); f2() returns 0 and 0.


    Two calls to fun_basket, two different common_var's, two f1's and two
    f2's. Each f1/f2 pair have access to a different common_var, so it's
    working as expected. To work as you expected, fun_basket should be on
    the same block common_var is defined.

    --
    Felipe.
     
    Felipe Almeida Lessa, Dec 1, 2006
    #2
    1. Advertising

  3. Karl Kofnarson

    Klaas Guest

    Karl Kofnarson wrote:
    > Hi,
    > while writing my last program I came upon the problem
    > of accessing a common local variable by a bunch of
    > functions.
    > I wanted to have a function which would, depending on
    > some argument, return other functions all having access to
    > the same variable. An OO approach would do but why not
    > try out closures...
    > So here is a simplified example of the idea:
    > def fun_basket(f):
    > common_var = [0]
    > def f1():
    > print common_var[0]
    > common_var[0]=1
    > def f2():
    > print common_var[0]
    > common_var[0]=2
    > if f == 1:
    > return f1
    > if f == 2:
    > return f2
    > If you call f1 and f2 from the inside of fun_basket, they
    > behave as expected, so common_var[0] is modified by
    > whatever function operates on it.
    > However, calling f1=fun_basket(1); f2 = fun_basket(2) and
    > then f1(); f2() returns 0 and 0. It is not the way one would
    > expect closures to work, knowing e.g. Lisp make-counter.
    > Any ideas what's going on behind the scene?


    Python can be read quite literally. "common_var" is a local variable
    to fun_basket, hence it independent among invokations of fun_basket.
    "def" is a statement that creates a function when it is executed. If
    you execute the same def statement twice, two different functions are
    created. Running fun_basket twice creates four closures, and the first
    two have no relation to the second two. The two sets close over
    different cell variables.

    If you want to share data between function invokation, you need an
    object which persists between calls. You can use a global variable, or
    a default argument. But since the value is shared everytime the
    function is called, I don't see the value in using a closure. I don't
    know lisp very well, but in my mind the whole point of closures is that
    you can reference a different unique cell each time.

    -MIke
     
    Klaas, Dec 1, 2006
    #3
  4. Karl Kofnarson

    Paul McGuire Guest

    "Karl Kofnarson" <> wrote in message
    news:p...
    > Hi,
    > while writing my last program I came upon the problem
    > of accessing a common local variable by a bunch of
    > functions.
    > I wanted to have a function which would, depending on
    > some argument, return other functions all having access to
    > the same variable. An OO approach would do but why not
    > try out closures...
    > So here is a simplified example of the idea:
    > def fun_basket(f):
    > common_var = [0]
    > def f1():
    > print common_var[0]
    > common_var[0]=1
    > def f2():
    > print common_var[0]
    > common_var[0]=2
    > if f == 1:
    > return f1
    > if f == 2:
    > return f2


    Karl,

    Usually when using this idiom, fun_basket would return a tuple of all of the
    defined functions, rather than one vs. the other. So in place of:
    > if f == 1:
    > return f1
    > if f == 2:
    > return f2

    Just do
    > return f1, f2

    (For that matter, the argument f is no longer needed either.)

    Then your caller will get 2 functions, who share a common var. You don't
    call fun_basket any more, you've already created your two "closures". Call
    fun_basket using something like:

    z1,z2 = fun_basket(None)

    And then call z1() and z2() at your leisure - they should have the desired
    behavior.

    -- Paul
     
    Paul McGuire, Dec 1, 2006
    #4
  5. Karl Kofnarson

    Carl Banks Guest

    Karl Kofnarson wrote:
    > Hi,
    > while writing my last program I came upon the problem
    > of accessing a common local variable by a bunch of
    > functions.
    > I wanted to have a function which would, depending on
    > some argument, return other functions all having access to
    > the same variable. An OO approach would do but why not
    > try out closures...
    > So here is a simplified example of the idea:
    > def fun_basket(f):
    > common_var = [0]
    > def f1():
    > print common_var[0]
    > common_var[0]=1
    > def f2():
    > print common_var[0]
    > common_var[0]=2
    > if f == 1:
    > return f1
    > if f == 2:
    > return f2
    > If you call f1 and f2 from the inside of fun_basket, they
    > behave as expected, so common_var[0] is modified by
    > whatever function operates on it.
    > However, calling f1=fun_basket(1); f2 = fun_basket(2) and
    > then f1(); f2() returns 0 and 0. It is not the way one would
    > expect closures to work, knowing e.g. Lisp make-counter.



    Lisp works the same way.

    * (defun fun_basket (f)
    (let ((common_var 0))
    (defun f1 ()
    (print common_var)
    (setf common_var 1))
    (defun f2 ()
    (print common_var)
    (setf common_var 2))
    (if (eq f 1)
    #'f1
    #'f2)))

    FUN_BASKET
    * (setf (symbol-function 'f1a) (fun_basket 1))

    ; Converted F1.
    ; Converted F2.

    #<Interpreted Function F1 {5807C9A1}>
    * (setf (symbol-function 'f2a) (fun_basket 2))

    #<Interpreted Function F2 {5807D409}>
    * (f1a)

    0
    1
    * (f2a)

    0
    2



    > Any ideas what's going on behind the scene?


    Every time you call the function, a new closure is created.


    Carl Banks
     
    Carl Banks, Dec 1, 2006
    #5
  6. > Karl,
    >
    > Usually when using this idiom, fun_basket would return a tuple of all of the
    > defined functions, rather than one vs. the other. So in place of:
    >> if f == 1:
    >> return f1
    >> if f == 2:
    >> return f2

    > Just do
    >> return f1, f2

    > (For that matter, the argument f is no longer needed either.)
    >
    > Then your caller will get 2 functions, who share a common var. You don't
    > call fun_basket any more, you've already created your two "closures". Call
    > fun_basket using something like:
    >
    > z1,z2 = fun_basket(None)
    >
    > And then call z1() and z2() at your leisure - they should have the desired
    > behavior.
    >
    > -- Paul


    Thanks a lot Paul and for the other answers. The things are now
    clear to me. In fact, in the Lisp example that I mentioned, you
    get a list (or let it be association list) of the internal
    functions. Then you can call them separately and they work as
    you expect but it's due to the fact only that you got them created
    at the same time.
     
    Karl Kofnarson, Dec 2, 2006
    #6
  7. Karl Kofnarson

    Paddy Guest

    Karl Kofnarson wrote:

    > > Karl,
    > >
    > > Usually when using this idiom, fun_basket would return a tuple of all of the
    > > defined functions, rather than one vs. the other. So in place of:
    > >> if f == 1:
    > >> return f1
    > >> if f == 2:
    > >> return f2

    > > Just do
    > >> return f1, f2

    > > (For that matter, the argument f is no longer needed either.)
    > >
    > > Then your caller will get 2 functions, who share a common var. You don't
    > > call fun_basket any more, you've already created your two "closures". Call
    > > fun_basket using something like:
    > >
    > > z1,z2 = fun_basket(None)
    > >
    > > And then call z1() and z2() at your leisure - they should have the desired
    > > behavior.
    > >
    > > -- Paul

    >
    > Thanks a lot Paul and for the other answers. The things are now
    > clear to me. In fact, in the Lisp example that I mentioned, you
    > get a list (or let it be association list) of the internal
    > functions. Then you can call them separately and they work as
    > you expect but it's due to the fact only that you got them created
    > at the same time.


    I played around a bit. The following is a 'borg' version in that there
    is only one counter shared between all calls of the outer function:

    >>> def fun_borg_var(initial_val=0):

    .... def borg_var_inc(x=1):
    .... fun_borg_var._n += x
    .... return fun_borg_var._n
    .... def borg_var_dec(x=1):
    .... fun_borg_var._n -= x
    .... return fun_borg_var._n
    .... try:
    .... fun_borg_var._n = fun_borg_var._n
    .... except:
    .... fun_borg_var._n = initial_val
    .... return (borg_var_inc, borg_var_dec)
    ....
    >>> up1, dn1 = fun_borg_var() # get an inc/decrementer
    >>> up1(0)

    0
    >>> up1()

    1
    >>> up1()

    2
    >>> dn1()

    1
    >>> dn1()

    0
    >>> dn1()

    -1
    >>> up2, dn2 = fun_borg_var() # get another inc/decrementer
    >>> up2(0) # looks like the same _n

    -1
    >>> up2(3)

    2
    >>> up1(3)

    5
    >>>



    - Paddy.
     
    Paddy, Dec 6, 2006
    #7
  8. "Paddy" wrote:

    > I played around a bit. The following is a 'borg' version in that there
    > is only one counter shared between all calls of the outer function:
    >
    >>>> def fun_borg_var(initial_val=0):

    > ... def borg_var_inc(x=1):
    > ... fun_borg_var._n += x


    a drawback with the function attribute approach compared to a real closure
    is that the function is no longer a self-contained callable:

    def fun_borg_var(initial_val=0):
    def borg_var_inc(x=1):
    fun_borg_var._n += x
    return fun_borg_var._n
    def borg_var_dec(x=1):
    fun_borg_var._n -= x
    return fun_borg_var._n
    try:
    fun_borg_var._n = fun_borg_var._n
    except:
    fun_borg_var._n = initial_val
    return (borg_var_inc, borg_var_dec)

    up1, dn1 = fun_borg_var()

    del fun_borg_var # won't need this any more

    print up1() # oops!

    so you might as well use a good old global variable, and initialize it as
    usual.

    </F>
     
    Fredrik Lundh, Dec 6, 2006
    #8
  9. Karl Kofnarson

    Paul Boddie Guest

    Karl Kofnarson wrote:
    >
    > I wanted to have a function which would, depending on
    > some argument, return other functions all having access to
    > the same variable. An OO approach would do but why not
    > try out closures...


    I know that everyone will say that Python is a "multi-paradigm"
    language and that one should feel free to use whatever technique seems
    appropriate to solve the problem at hand, but it seems to me that
    there's been an explosion in nested function usage recently, with lots
    of code snippets showing them off either in the context of a debugging
    exercise or as a proposed solution to a problem, and yet in many cases
    their usage seems frivolous in comparison to plain old object-oriented
    techniques.

    I'm not pointing the finger at you here, Karl, since you seem to be
    experimenting with closures, but why are they suddenly so fashionable?
    Haven't the features supporting them existed in Python for a few
    versions now? Don't people want to write classes any more?

    Intrigued,

    Paul
     
    Paul Boddie, Dec 6, 2006
    #9
  10. Paul Boddie wrote:

    > I know that everyone will say that Python is a "multi-paradigm"
    > language and that one should feel free to use whatever technique seems
    > appropriate to solve the problem at hand, but it seems to me that
    > there's been an explosion in nested function usage recently, with lots
    > of code snippets showing them off either in the context of a debugging
    > exercise or as a proposed solution to a problem, and yet in many cases
    > their usage seems frivolous in comparison to plain old object-oriented
    > techniques.


    when doing some heavy optimization, I recently found myself writing:

    def foobar(arg1, arg2, arg3):
    def helper(arg):
    do something with arg1 and argument
    def foo():
    do something with arg1 and arg3 and
    call helper
    def bar():
    do something with arg1 and arg2
    def zoo():
    do something with arg2 and arg3 and
    call helper
    # oops; how do I return all these?
    class bag(object):
    pass
    bag = bag()
    bag.foo = foo
    bag.bar = bar
    bag.zoo = zoo
    return bag

    which, I think, deserves no further comment...

    </F>
     
    Fredrik Lundh, Dec 6, 2006
    #10
  11. Karl Kofnarson

    Tim Chase Guest

    > def foobar(arg1, arg2, arg3):
    > def helper(arg):
    > do something with arg1 and argument
    > def foo():
    > do something with arg1 and arg3 and
    > call helper
    > def bar():
    > do something with arg1 and arg2
    > def zoo():
    > do something with arg2 and arg3 and
    > call helper
    > # oops; how do I return all these?
    > class bag(object):
    > pass
    > bag = bag()
    > bag.foo = foo
    > bag.bar = bar
    > bag.zoo = zoo
    > return bag
    >
    > which, I think, deserves no further comment...


    Could you please explain your reasoning behind why you opted to
    create the bag, fill it, and then return it? Usually I see
    something like

    return foo,bar,zoo

    and it would be helpful to understand the reasoning behind why
    one would choose one over the other. Some of the off-the-cuff
    thoughts in my trying to understand it:

    -defining a bag, instantiating a bag and filling it takes a few
    extra cycles for each call of foobar() so the returned tuple
    should be a hair faster (though perhaps defining a bag class
    outside the foobar() function might trim some of this?)

    -returning a tuple requires that any additional methods you might
    opt to add/return in the future involve adjusting all your code,
    whereas the bag method allows you to toss extra methods in the
    bag without adjusting every statement that calls foobar()

    Are either of these correct? Are there additional reasons I'm
    missing?

    Thanks,

    -tkc
     
    Tim Chase, Dec 6, 2006
    #11
  12. Karl Kofnarson

    Paul Boddie Guest

    Fredrik Lundh wrote:
    >
    > when doing some heavy optimization, I recently found myself writing:
    >
    > def foobar(arg1, arg2, arg3):
    > def helper(arg):
    > do something with arg1 and argument
    > def foo():
    > do something with arg1 and arg3 and
    > call helper
    > def bar():
    > do something with arg1 and arg2
    > def zoo():
    > do something with arg2 and arg3 and
    > call helper
    > # oops; how do I return all these?
    > class bag(object):
    > pass
    > bag = bag()
    > bag.foo = foo
    > bag.bar = bar
    > bag.zoo = zoo
    > return bag
    >
    > which, I think, deserves no further comment...


    Have I missed something deep here, or could you not have written the
    above as follows...?

    class foobar(object):
    def __init__(self, arg1, arg2, arg3):
    self.arg1, self.arg2, self.arg3 = arg1, arg2, arg3
    def helper(self, arg):
    do something with arg1 and argument
    def foo(self):
    do something with arg1 and arg3 and
    call helper
    def bar(self):
    do something with arg1 and arg2
    def zoo(self):
    do something with arg2 and arg3 and
    call helper

    There's certainly some boilerplate required (in both forms, though) and
    you'd have to use self in various places, but the above looks less
    contorted to me. I'd like to hear your further comment, however, since
    the principal inconvenience of making a class seems to be in the
    verbose initialisation and explicit self, the latter of which obviously
    being a feature that you won't find me complaining about, though. ;-)

    Paul
     
    Paul Boddie, Dec 6, 2006
    #12
  13. Paul Boddie wrote:
    > I'm not pointing the finger at you here, Karl, since you seem to be
    > experimenting with closures, but why are they suddenly so fashionable?
    > Haven't the features supporting them existed in Python for a few
    > versions now? Don't people want to write classes any more?
    >
    > Intrigued,
    >
    > Paul


    I believe decorators are in large part responsible for that. A callable
    object does not work
    as a method unless you define a custom __get__, so in decorator
    programming it is
    often easier to use a closure. OTOH closures a not optimal if you want
    persistency
    (you cannot pickle a closure) so in that case I use a callable object
    instead.

    Michele Simionato
     
    Michele Simionato, Dec 6, 2006
    #13
  14. Karl Kofnarson

    Klaas Guest

    Michele Simionato wrote:

    > I believe decorators are in large part responsible for that. A callable
    > object does not work
    > as a method unless you define a custom __get__, so in decorator
    > programming it is
    > often easier to use a closure. OTOH closures a not optimal if you want
    > persistency
    > (you cannot pickle a closure) so in that case I use a callable object
    > instead.


    Note that it isn't necessary to write the descriptor yourself. The
    'new' module takes care of it:

    In [1]: class A(object):
    ...: pass
    In [2]: a = A()
    In [3]: class Method(object):
    ...: def __call__(mself, oself):
    ...: print mself, oself
    In [4]: import new
    In [5]: a.method = new.instancemethod(Method(), a, A)
    In [6]: a.method()
    <__main__.Method object at 0xb7ab7f6c> <__main__.A object at
    0xb7ab79ec>

    -Mike
     
    Klaas, Dec 6, 2006
    #14
  15. Karl Kofnarson

    John Nagle Guest

    Paul Boddie wrote:
    > I know that everyone will say that Python is a "multi-paradigm"
    > language and that one should feel free to use whatever technique seems
    > appropriate to solve the problem at hand, but it seems to me that
    > there's been an explosion in nested function usage recently, with lots
    > of code snippets showing them off either in the context of a debugging
    > exercise or as a proposed solution to a problem, and yet in many cases
    > their usage seems frivolous in comparison to plain old object-oriented
    > techniques.


    Most of the examples given here are kind of silly, but closures have
    real uses. I used one today in Javascript because I was writing an
    AJAX application, and I was using an API, the standard XMLHttpRequestObject,
    which required a callback function with no arguments. A closure allowed
    the code to pass relevant information with the callback function, which
    would be called when a request to the server completed. A global
    variable wouldn't have worked, because multiple instances of the object
    making the callback are possible.

    It's a useful tool, but not one you need frequently. Don't get
    carried away.

    John Nagle
    Animats
     
    John Nagle, Dec 12, 2006
    #15
  16. John Nagle wrote:

    > Most of the examples given here are kind of silly, but closures have
    > real uses. I used one today in Javascript because I was writing an
    > AJAX application, and I was using an API, the standard XMLHttpRequestObject,
    > which required a callback function with no arguments. A closure allowed
    > the code to pass relevant information with the callback function, which
    > would be called when a request to the server completed. A global
    > variable wouldn't have worked, because multiple instances of the object
    > making the callback are possible.


    the usual way of handling multiple callback context instances in Python
    is of course to create multiple instances of the class that implements
    the behaviour, and use a bound method as the callback.

    </F>
     
    Fredrik Lundh, Dec 12, 2006
    #16
  17. Karl Kofnarson

    Roy Smith Guest

    John Nagle <> wrote:
    > Most of the examples given here are kind of silly, but closures have
    > real uses. I used one today in Javascript because I was writing an
    > AJAX application, and I was using an API, the standard XMLHttpRequestObject,
    > which required a callback function with no arguments. A closure allowed
    > the code to pass relevant information with the callback function, which
    > would be called when a request to the server completed. A global
    > variable wouldn't have worked, because multiple instances of the object
    > making the callback are possible.


    As another example, from a large C++ project I'm currently working on, we
    use closures in many places to push actions onto queues.

    One kind of queue we have is a timed queue. You push a closure and a time
    onto it, and it executes the closure at the appointed time. Another kind
    of queue is for inter-thread communication. You push a closure onto a
    thread's input queue when you want it to do something. The thread just
    keeps popping closures off the other end of the queue and executing them.

    Think of a closure as a way of saying "do this". You can say, "At 5:00, do
    this", or "Do this as soon as you get a chance".

    # pseudocode
    class husband (spouse):
    def pickUpGroceries():
    etc
    def addToQueue():
    etc

    c = makeClosure (husband.pickUpGroceries, [milk, eggs, bread])
    h = husband()
    h.addToQueue (c)
     
    Roy Smith, Dec 12, 2006
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mswc.net
    Replies:
    2
    Views:
    681
    Kevin Spencer
    Apr 26, 2005
  2. Justin Kadima

    Realy weird Atlas licence: Is it free or not?

    Justin Kadima, Oct 15, 2006, in forum: ASP .Net
    Replies:
    1
    Views:
    270
    Teemu Keiski
    Oct 15, 2006
  3. Patrick Kowalzick
    Replies:
    5
    Views:
    500
    Patrick Kowalzick
    Mar 14, 2006
  4. bogdan
    Replies:
    2
    Views:
    319
    bogdan
    Mar 31, 2008
  5. Bogdan

    Is asp.net menu realy that bad?

    Bogdan, Jun 14, 2009, in forum: ASP .Net
    Replies:
    3
    Views:
    475
    Gregory A. Beamer
    Jun 15, 2009
Loading...

Share This Page