Lambda forms and scoping

Discussion in 'Python' started by Márcio Faustino, Mar 19, 2009.

  1. Hi,

    Executing the example below doesn't produce the expected behavior, but
    using the commented code does. Is this normal, or is it a problem with
    Python? I've tested it with version 2.6.1 on Windows XP.

    Thanks,

    --

    from abc import *
    from types import *
    import re

    class Base (ObjectType):
    __metaclass__ = ABCMeta

    def __init__(self):
    for option in self.get_options().keys():
    method = 'get_%s_option' % re.sub(' ', '_', option.lower
    ())
    setattr(self.__class__, method, lambda self:
    self.get_option(option))

    #def create_method(option):
    # method = 'get_%s_option' % re.sub(' ', '_', option.lower
    ())
    # setattr(self.__class__, method, lambda self:
    self.get_option(option))
    #
    #map(create_method, self.get_options().keys())

    @abstractmethod
    def get_options(self):
    raise NotImplementedError()

    def get_option(self, option):
    return self.get_options()[option]

    class Derived (Base):
    def get_options(self):
    return {
    'Message': 'Hello world!',
    'Web site': 'http://www.example.com',
    }

    object = Derived()
    print object.get_message_option()
    print object.get_web_site_option()
    Márcio Faustino, Mar 19, 2009
    #1
    1. Advertising

  2. Márcio Faustino <m.faustino <at> gmail.com> writes:

    >
    > Hi,
    >
    > Executing the example below doesn't produce the expected behavior, but
    > using the commented code does. Is this normal, or is it a problem with
    > Python? I've tested it with version 2.6.1 on Windows XP.
    >
    > Thanks,
    >
    > --
    >
    > from abc import *
    > from types import *
    > import re
    >
    > class Base (ObjectType):
    > __metaclass__ = ABCMeta
    >
    > def __init__(self):
    > for option in self.get_options().keys():
    > method = 'get_%s_option' % re.sub(' ', '_', option.lower
    > ())
    > setattr(self.__class__, method, lambda self:
    > self.get_option(option))


    This is because the closure over option is changed when it is reassigned in the
    for loop. For example:

    >>> def f():

    .... return [lambda: num for num in xrange(2)]
    ....
    >>> f()

    [<function <lambda> at 0x83f30>, <function <lambda> at 0x83e70>]
    >>> f()[0]

    <function <lambda> at 0x83ef0>
    >>> g = f()
    >>> g[0]()

    1
    >>> g[1]()

    1
    Benjamin Peterson, Mar 19, 2009
    #2
    1. Advertising

  3. So simple :) thanks!
    Márcio Faustino, Mar 19, 2009
    #3
  4. Benjamin Peterson <> wrote:
    > Márcio Faustino <m.faustino <at> gmail.com> writes:
    > >
    > > Executing the example below doesn't produce the expected behavior, but
    > > using the commented code does. Is this normal, or is it a problem with
    > > Python? I've tested it with version 2.6.1 on Windows XP.
    > >
    > > Thanks,
    > >
    > > --
    > >
    > > from abc import *
    > > from types import *
    > > import re
    > >
    > > class Base (ObjectType):
    > > __metaclass__ = ABCMeta
    > >
    > > def __init__(self):
    > > for option in self.get_options().keys():
    > > method = 'get_%s_option' % re.sub(' ', '_', option.lower
    > > ())
    > > setattr(self.__class__, method, lambda self:
    > > self.get_option(option))

    >
    > This is because the closure over option is changed when it is reassigned in the
    > for loop. For example:
    >
    > >>> def f():

    > ... return [lambda: num for num in xrange(2)]
    > ...
    > >>> f()

    > [<function <lambda> at 0x83f30>, <function <lambda> at 0x83e70>]
    > >>> f()[0]

    > <function <lambda> at 0x83ef0>
    > >>> g = f()
    > >>> g[0]()

    > 1
    > >>> g[1]()

    > 1


    Here's the way I find it useful to think about this:

    When your lambda is created in your for loop inside your __init__ method,
    it acquires a pointer to __init__'s local namespace. (That's how I
    understand what "closure" means in this case, though I imagine "closure"
    probably means something slightly different in computer-science-ese :)

    So, when any of those lambdas is executed, they all have a pointer to
    the exact same namespace, that of __init__. And when they are called,
    __init__'s namespace is in whatever state it was left in when __init__
    ended. In this case, that means that 'option' is pointing to the value
    it had at the _end_ of the for loop.

    Hope this helps. I find that thinking in terms of namespaces helps
    me understand how Python works better than any other mental model
    I've come across.

    --
    R. David Murray http://www.bitdance.com
    R. David Murray, Mar 20, 2009
    #4
  5. On Mar 19, 10:52 pm, Márcio Faustino <> wrote:
    > Hi,
    >
    > Executing the example below doesn't produce the expected behavior, but
    > using the commented code does. Is this normal, or is it a problem with
    > Python?


    It is a common gotcha. Notice that it has nothing to do with lambda
    functions, you
    would get the same issue with regular functions.

    This post http://www.artima.com/forums/flat.jsp?forum=106&thread=251156
    and especially the comments
    below should share some light on why it is the way it is (short
    answer: for loops are
    implemented by mutating the loop variable at each iteration).
    Michele Simionato, Mar 20, 2009
    #5
  6. En Fri, 20 Mar 2009 09:28:08 -0300, R. David Murray
    <> escribió:
    > Benjamin Peterson <> wrote:
    >> Márcio Faustino <m.faustino <at> gmail.com> writes:
    >> >
    >> > Executing the example below doesn't produce the expected behavior, but
    >> > using the commented code does. Is this normal, or is it a problem with
    >> > Python? I've tested it with version 2.6.1 on Windows XP.


    > Here's the way I find it useful to think about this:
    >
    > When your lambda is created in your for loop inside your __init__ method,
    > it acquires a pointer to __init__'s local namespace. (That's how I
    > understand what "closure" means in this case, though I imagine "closure"
    > probably means something slightly different in computer-science-ese :)


    Well, some people would say that the function "closes over" part of its
    environment, but that's the idea.

    > So, when any of those lambdas is executed, they all have a pointer to
    > the exact same namespace, that of __init__. And when they are called,
    > __init__'s namespace is in whatever state it was left in when __init__
    > ended. In this case, that means that 'option' is pointing to the value
    > it had at the _end_ of the for loop.


    That's a pretty good explanation -- I would just substitute "pointer" by
    "reference".

    > Hope this helps. I find that thinking in terms of namespaces helps
    > me understand how Python works better than any other mental model
    > I've come across.


    Good to know! The Python execution model *is* based on namespaces - any
    other mental model won't be accurate past certain point (although some
    people insist...)

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 20, 2009
    #6
  7. On Mar 20, 12:28 pm, "R. David Murray" <> wrote:
    > Hope this helps.  I find that thinking in terms of namespaces helps
    > me understand how Python works better than any other mental model
    > I've come across.


    It does, thanks.

    On Mar 20, 12:41 pm, Michele Simionato <>
    wrote:
    > This post http://www.artima.com/forums/flat.jsp?forum=106&thread=251156
    > and especially the comments
    > below should share some light on why it is the way it is (short
    > answer: for loops are
    > implemented by mutating the loop variable at each iteration).


    Thanks for the link.
    Márcio Faustino, Mar 21, 2009
    #7
  8. En Fri, 20 Mar 2009 23:16:00 -0300, alex goretoy
    <> escribió:

    > i looks at lambdas as unbound functions(or super function), in the case
    > above we create the functions in a list places it in memory unboud, once
    > binding a call to the memory address space it returns the value
    >
    > it is basically same as doing this:
    > def f():
    > print "f"
    >
    > a=f #unbound function, same as rename function
    > a() #bind call to address space


    Mmm, I don't quite understand what you said. lambda creates functions that
    aren't different than functions created by def: apart from the name,
    they're really the same thing.

    And if you imply that *where* you call a function does matter, it does
    not. A function carries its own local namespace, its own closure, and its
    global namespace. At call time, no additional "binding" is done (except
    parameters -> arguments).

    (and the address space is always the one of the running process)

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 22, 2009
    #8
  9. "Gabriel Genellina" <> wrote:
    > En Fri, 20 Mar 2009 23:16:00 -0300, alex goretoy
    > <> escribió:
    >
    > > i looks at lambdas as unbound functions(or super function), in the case
    > > above we create the functions in a list places it in memory unboud, once
    > > binding a call to the memory address space it returns the value
    > >
    > > it is basically same as doing this:
    > > def f():
    > > print "f"
    > >
    > > a=f #unbound function, same as rename function
    > > a() #bind call to address space

    >
    > Mmm, I don't quite understand what you said. lambda creates functions that
    > aren't different than functions created by def: apart from the name,
    > they're really the same thing.


    Oh, good, I'm not the only one for whom the above didn't make sense :)
    I feel a little less dense now.

    > And if you imply that *where* you call a function does matter, it does
    > not. A function carries its own local namespace, its own closure, and its
    > global namespace. At call time, no additional "binding" is done (except
    > parameters -> arguments).
    >
    > (and the address space is always the one of the running process)


    I poked around in the API docs and experimented with func_closure and
    related attributes, and after bending my brain for a while I think I
    understand this. The actual implementation of the closure is a single
    list of 'cell' objects which represent namespace slots in the nested
    scopes in which the closed-over function is defined.

    But the fact that it is a single list is an implementation detail, and
    the implementation is in fact carefully designed so that conceptually
    we can think of the closure as giving the function access to those
    nested-scope namespaces in almost(*) the same sense that it has a
    reference to the global and local namespaces. That is, if what a name
    in _any_ of those namespaces points to is changed, then the closed-over
    function sees those changes.

    In this way, we understand the original example: when defining a
    lambda having a 'free variable' (that is, one not defined in either the
    local or global scope) that was a name in the surrounding function's
    local namespace, the lambda is going to see any changes made by the
    surrounding function with regards to what that name points to. Thus,
    the final value that the lambda uses is whatever the final value of the
    for loop variable was when the surrounding function finished executing.

    However, I think that a Python closure is not quite the same thing as a
    'computer science' closure, for the same reason that people coming from a
    language with variables-and-values as opposed to namespaces get confused
    when dealing with Python function call semantics. Consider:

    http://en.wikipedia.org/wiki/Closure_(computer_science)

    That says that a closure can be used to provide a function with a private
    set of variables that persist from one invocation to the next, so that
    a value established in one call can be accessed in the next. The last
    part of that sentence is not true in Python, since any assignment inside
    a function affects only the local (per-invocation) namespace or (given
    a global statement) the global namespace. A function cannot change the
    thing pointed to by a name in the closure. Only the outer function,
    for whom that name is in its local namespace, can do that.

    (*) That last sentence in the previous paragraph is why I said '_almost_
    the same sense' earlier: a function can modify what names point to in
    its local and global namespaces, but cannot modify what names point to
    in the closure namespace.

    Of course, we can produce the same _effect_ as a computer science closure
    in Python by using mutable objects...which is exactly parallel to the
    difference between passing mutable or immutable objects in a function
    call.

    --
    R. David Murray http://www.bitdance.com
    R. David Murray, Mar 22, 2009
    #9
  10. En Sun, 22 Mar 2009 20:43:02 -0300, alex goretoy
    <> escribió:

    > Sorry to have confused yall. What I meant was that you can do something
    > like
    > this, where the fucntion isn't called until it is bount to () with the
    > right
    > params
    >
    >>>> def a():

    > ... print "inside a"
    > ...
    >>>> def b():

    > ... print "inside b"
    > ...
    >>>> def c(a,b):

    > ... a()
    > ... b()
    > ...
    >>>> d={c:(a,b)}
    >>>> d[c][0]()

    > inside a
    >>>> d[c][1]()

    > inside b
    >>>> d[c(d[c][0],d[c][1])]

    > inside a
    > inside b
    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > KeyError: None
    >
    > where function a and b are bound in function c


    Ah, so this is a terminology issue. I'd say that a and b are *called* in
    function c, not *bound*. I've never seen "bind" used in this sense before,
    but as Humpty Dumpty said to Alice:

    - When I use a word, it means just what I choose it to mean -- neither
    more nor less.
    - The question is, whether you can make words mean so many different
    things.
    - The question is, which is to be master -- that's all.

    (Lewis Carroll, Through the Looking Glass, ch. VI)

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 23, 2009
    #10
  11. En Sun, 22 Mar 2009 16:42:21 -0300, R. David Murray
    <> escribió:
    > "Gabriel Genellina" <> wrote:


    >> And if you imply that *where* you call a function does matter, it does
    >> not. A function carries its own local namespace, its own closure, and
    >> its
    >> global namespace. At call time, no additional "binding" is done (except
    >> parameters -> arguments).

    >
    > I poked around in the API docs and experimented with func_closure and
    > related attributes, and after bending my brain for a while I think I
    > understand this. The actual implementation of the closure is a single
    > list of 'cell' objects which represent namespace slots in the nested
    > scopes in which the closed-over function is defined.
    >
    > But the fact that it is a single list is an implementation detail, and
    > the implementation is in fact carefully designed so that conceptually
    > we can think of the closure as giving the function access to those
    > nested-scope namespaces in almost(*) the same sense that it has a
    > reference to the global and local namespaces. That is, if what a name
    > in _any_ of those namespaces points to is changed, then the closed-over
    > function sees those changes.
    >
    > In this way, we understand the original example: when defining a
    > lambda having a 'free variable' (that is, one not defined in either the
    > local or global scope) that was a name in the surrounding function's
    > local namespace, the lambda is going to see any changes made by the
    > surrounding function with regards to what that name points to. Thus,
    > the final value that the lambda uses is whatever the final value of the
    > for loop variable was when the surrounding function finished executing.


    Exactly.

    > However, I think that a Python closure is not quite the same thing as a
    > 'computer science' closure, for the same reason that people coming from a
    > language with variables-and-values as opposed to namespaces get confused
    > when dealing with Python function call semantics. Consider:
    >
    > http://en.wikipedia.org/wiki/Closure_(computer_science)
    >
    > That says that a closure can be used to provide a function with a private
    > set of variables that persist from one invocation to the next, so that
    > a value established in one call can be accessed in the next. The last
    > part of that sentence is not true in Python, since any assignment inside
    > a function affects only the local (per-invocation) namespace or (given
    > a global statement) the global namespace. A function cannot change the
    > thing pointed to by a name in the closure. Only the outer function,
    > for whom that name is in its local namespace, can do that.


    That's true in Python 2.x, but 3.x has the "nonlocal" keyword - so you can
    modify variables in outer scopes too:

    p3> z = 1
    p3> def o():
    .... z = 2
    .... def i():
    .... nonlocal z
    .... print("z in i:", z)
    .... z = 5
    .... print("z in o:", z)
    .... i()
    .... print("z in o:", z)
    .... z=3
    .... print("z in o at exit:", z)
    .... return i
    ....
    p3> i=o()
    z in o: 2
    z in i: 2
    z in o: 5
    z in o at exit: 3
    p3> z
    1
    p3> i()
    z in i: 3
    p3> i()
    z in i: 5

    (Anyway I think the inability to "modify" a variable doesn't invalidate
    the "closure" concept...)

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 23, 2009
    #11
  12. "Gabriel Genellina" <> wrote:
    > > However, I think that a Python closure is not quite the same thing as a
    > > 'computer science' closure, for the same reason that people coming from a
    > > language with variables-and-values as opposed to namespaces get confused
    > > when dealing with Python function call semantics. Consider:
    > >
    > > http://en.wikipedia.org/wiki/Closure_(computer_science)
    > >
    > > That says that a closure can be used to provide a function with a private
    > > set of variables that persist from one invocation to the next, so that
    > > a value established in one call can be accessed in the next. The last
    > > part of that sentence is not true in Python, since any assignment inside
    > > a function affects only the local (per-invocation) namespace or (given
    > > a global statement) the global namespace. A function cannot change the
    > > thing pointed to by a name in the closure. Only the outer function,
    > > for whom that name is in its local namespace, can do that.

    >
    > That's true in Python 2.x, but 3.x has the "nonlocal" keyword - so you can
    > modify variables in outer scopes too:
    >
    > p3> z = 1
    > p3> def o():
    > ... z = 2
    > ... def i():
    > ... nonlocal z
    > ... print("z in i:", z)
    > ... z = 5
    > ... print("z in o:", z)
    > ... i()
    > ... print("z in o:", z)
    > ... z=3
    > ... print("z in o at exit:", z)
    > ... return i
    > ...
    > p3> i=o()
    > z in o: 2
    > z in i: 2
    > z in o: 5
    > z in o at exit: 3
    > p3> z
    > 1
    > p3> i()
    > z in i: 3
    > p3> i()
    > z in i: 5
    >
    > (Anyway I think the inability to "modify" a variable doesn't invalidate
    > the "closure" concept...)


    Invalidate, no, but it does mean that the word meant something slightly
    different to a Python 2.x programmer than to, say, a Scheme programmer.
    We could say that a Python 2.x closure is a "read-only closure".

    But now with Python 3.x we can really have fun (thank you for that info):

    >>> def g():

    .... def a(x):
    .... nonlocal z
    .... z = z + x
    .... def b(x):
    .... nonlocal z
    .... z = z - x
    .... def p():
    .... print(z)
    .... z = 1
    .... return a, b, p
    ....
    >>> add, sub, pr = g()
    >>> pr()

    1
    >>> add(10)
    >>> pr()

    11
    >>> sub(5)
    >>> pr()

    6

    So, as the wikipedia article says, we could, if we wanted to, use python 3
    closures to reimplement objects, in a very confusing fashion :)

    --
    R. David Murray http://www.bitdance.com
    R. David Murray, Mar 23, 2009
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matt S.
    Replies:
    0
    Views:
    399
    Matt S.
    May 4, 2004
  2. Ian McMeans

    scoping with lambda in loops

    Ian McMeans, Sep 16, 2003, in forum: Python
    Replies:
    8
    Views:
    344
    Jacek Generowicz
    Sep 17, 2003
  3. Roman Suzi
    Replies:
    13
    Views:
    607
    Bengt Richter
    Jan 7, 2005
  4. Steve Dogers

    lambda vs non-lambda proc

    Steve Dogers, Mar 30, 2009, in forum: Ruby
    Replies:
    1
    Views:
    177
    Sean O'Halpin
    Mar 30, 2009
  5. Haochen Xie
    Replies:
    4
    Views:
    242
    Haochen Xie
    Mar 17, 2013
Loading...

Share This Page