Feature request: subclassing FunctionType [Was: Some language proposals]

Discussion in 'Python' started by Michele Simionato, Feb 27, 2004.

  1. I have read with interest the recent thread about closures. The funny
    thing is that the authors are arguing one against the other but I
    actually agree with all of them and I have a proposal that may
    be of some interest. To refresh your memory, I report here some
    quotation from that thread.

    Jacek Generowicz:

    > I sumbit to you that read-only closures are rare in Python because
    > they are a recent addition to the language.
    >
    > I submit to you that read-write closures are so much rarer still,
    > because they require an ugly hack to work.


    Paul Prescod:

    > I disagree. Closures are rare in Python because Python is primarily an
    > OOP language.


    Micheal Hudson:

    > Using closures to fake objects sucks.


    Jacek Generowicz:

    > When I need "objects" I use objects, and when I need closures I want
    > to be able to use closures.
    > [...] I have some stateful methods of classes (which I create
    > dynamically and stick onto the class as the information about the
    > existence of the method becomes available). By implementing them as
    > closures I can just stick them on to the class. If I were to implement
    > them as instances then I'd have to reimplement all the descriptors
    > that take care of turning functions into bound or unbound methods.
    >
    > [At this point I'd love someone to step up and show me how to re-use
    > the existing FunctionType descriptors in my own classes.]


    > Another example. I make quite heavy use of a trivial memoizer. The
    > closure verison is much shorter, clearer and faster.


    Paul Prescod:

    > Before Python had nested scopes at all, it was VERY common to fake them
    > using a linguistic trick: default arguments. This idiom was very common
    > in Python's source base. Python didn't have a proper way to do the
    > nesting but people figured out how to do it anyway and did it a LOT.
    > (similarly people often emulate OO in C)


    > The "wrap your mutables" feature is by comparison _much less intrusive_
    > but I don't ever see it in real Python code. This suggests to me that
    > people don't need the feature that badly.


    There is some merit on all those points of view. Speaking for myself,
    I would certainly use more closures if they were better supported by
    the language. The fact that closures are read-only don't bothers me too
    much, since I also think that when you want write access to a closure
    you are really trying to fake an object, and you are better off using
    a real object. On the other hand, I am really bothered by the scoping
    rules. My first post on this mailing list some time ago was caused by
    problems I had with the surprising scope rules. Take for instance
    this example:

    def make_adders(n):
    return [(lambda x: x+i) for i in range(n)]

    add0,add1=make_adders(2)

    print add0(0) #=> 1
    print add1(0) #=> 1

    This does not work as expected, and the solution is an ugly hack
    with default arguments:

    def make_adders(n):
    return [(lambda x,i=i: x+i) for i in range(n)]

    add0,add1=make_adders(2)

    print add0(0) #=> 0
    print add1(0) #=> 1

    I don't like that, but I don't think that the current scope rules
    will change any soon. Alternatively, I can use a class:

    class adder(object):
    def __init__(self,i):
    self.i=i
    def __call__(self,x):
    return x+self.i

    def make_adders(n):
    return [adder(i) for i in range(n)]

    add0,add1=make_adders(2)

    print add0(0) #=> 0
    print add1(0) #=> 1

    However, this is not really a solution, since as Jacek pointed
    out, to really fake a function I need to reimplement the
    descriptor protocol and I cannot get it from FunctionType.
    I have experience with the examples he cites, memoizing
    functions and wrapping methods, and they are exactly the
    examples where I have used closures instead of callable objects.
    OTOH, Paul Prescod is right and Python is primarily an
    OOP language, so I would prefer to use real objects over
    closures.

    What I think would help is the ability to subclass FunctionType.
    I remember a thread of some time ago where Tim Peters said
    that it is quite doable, simply nobody bothered to implement
    it yet.

    At the moment, I can fake the built-in FunctionType with something like
    that:

    class function(object):
    def __init__(self,func):
    self.__func__=func
    def __call__(self,*args,**kw):
    return self.__func__(*args,**kw)
    def __get__(self,obj,cls=None):
    return self.__func__.__get__(obj,cls)

    Then I can subclass "function" to implement any kind of wrapped functions.
    For instance, to trace functions:

    class traced_function(function):
    def __init__(self,func):
    def traced_func(*args,**kw): # using an handy closure ...
    print "Calling %s with arguments %s %s ..." % (func.__name__,args,kw)
    result=func(*args,**kw)
    print "Called %s with result %s" % (func.__name__,result)
    return result
    self.__func__=traced_func

    Since "function" implements the descriptor protocol, I can wrap methods
    for free:

    class C(object):
    add1=traced_function(lambda self,x: x+1)

    C().add1(1)

    My previous example with the adder would read

    class adder(function):
    def __init__(self,i):
    self.__func__=lambda x: x+i

    I think this reads quite well. At the present, however, I cannot subclass
    FunctionType and I am a bit disturbed by that, also because I can subclass
    the other built-ins types and there no reason why functions must be so
    special: at they end they are just callable objects with a descriptor
    protocol. So, I submit it as a feature request for Python 2.4.
    Maybe, if enough people wants the feature, we may get it!

    Michele Simionato
    Michele Simionato, Feb 27, 2004
    #1
    1. Advertising

  2. Re: Feature request: subclassing FunctionType [Was: Some languageproposals]

    (Michele Simionato) writes:

    [...]

    > There is some merit on all those points of view. Speaking for myself,
    > I would certainly use more closures if they were better supported by
    > the language. The fact that closures are read-only don't bothers me too
    > much, since I also think that when you want write access to a closure
    > you are really trying to fake an object, and you are better off using
    > a real object. On the other hand, I am really bothered by the scoping
    > rules. My first post on this mailing list some time ago was caused by
    > problems I had with the surprising scope rules. Take for instance
    > this example:
    >
    > def make_adders(n):
    > return [(lambda x: x+i) for i in range(n)]
    >
    > add0,add1=make_adders(2)
    >
    > print add0(0) #=> 1
    > print add1(0) #=> 1
    >
    > This does not work as expected,


    However, you would probably like this to work:

    def make_mutual_rec():
    def y1(x):
    if x > 0: return y2(x/2)
    else: return x
    def y2(x):
    if x > 0: return y2(x-2)
    else: return x
    return y1

    make_mutual_rec()(3)

    It seems to be difficult to reconcile the two points of view.

    More reading:

    http://www.python.org/~jeremy/weblog/040204.html
    http://python.org/sf/872326

    > and the solution is an ugly hack with default arguments:


    Or an object...

    [...]
    > I don't like that, but I don't think that the current scope rules
    > will change any soon. Alternatively, I can use a class:


    Ah...

    ]...]
    > I think this reads quite well. At the present, however, I cannot subclass
    > FunctionType and I am a bit disturbed by that, also because I can subclass
    > the other built-ins types and there no reason why functions must be so
    > special: at they end they are just callable objects with a descriptor
    > protocol. So, I submit it as a feature request for Python 2.4.
    > Maybe, if enough people wants the feature, we may get it!


    I suspect that creating the patch wouldn't take much longer than
    writing that article.

    Cheers,
    mwh

    --
    In general, I'd recommend injecting LSD directly into your temples,
    Syd-Barret-style, before mucking with Motif's resource framework.
    The former has far lower odds of leading directly to terminal
    insanity. -- Dan Martinez
    Michael Hudson, Feb 27, 2004
    #2
    1. Advertising

  3. (Michele Simionato) writes:

    > def make_adders(n):
    > return [(lambda x: x+i) for i in range(n)]
    >
    > add0,add1=make_adders(2)
    >
    > print add0(0) #=> 1
    > print add1(0) #=> 1
    >
    > This does not work as expected,


    Hmm, I think that what is expected is not an invariant between
    different people and even between more careful or less careful
    readings by the same person.

    On careful reading, it does exatly what I expect it to do.

    All the closures in the list share the same free-variable binding[*];
    there is only one binding of i, so they all find that single binding.

    I don't think that Python's behaviour differs from other languages, in
    this respect. For example in Common Lisp:

    * (defun make-adders (n)
    (loop for i below n collect
    (lambda (x) (+ x i))))
    MAKE-ADDERS
    * (destructuring-bind (add0 add1) (make-adders 2)
    (list (funcall add0 0)
    (funcall add1 0)))
    (2 2)

    [The important point is that the two resulting numbers are the same,
    the fact that they are both twos rather than ones is a feature of the
    looping construct and is irrelevant to the scoping issues.]

    > and the solution is an ugly hack
    > with default arguments:
    >
    > def make_adders(n):
    > return [(lambda x,i=i: x+i) for i in range(n)]
    >
    > add0,add1=make_adders(2)
    >
    > print add0(0) #=> 0
    > print add1(0) #=> 1


    Yes, if you want the free variables to be bound differently, then
    you need to bind i in a seperate "more inner" scope for each closure:

    (defun make-adders (n)
    (loop for i below n collect
    (let ((i i))
    (lambda (x) (+ x i)))))
    MAKE-ADDERS
    *
    (destructuring-bind (add0 add1) (make-adders 2)
    (list (funcall add0 0)
    (funcall add1 0)))
    (0 1)

    > I don't like that, but I don't think that the current scope rules
    > will change any soon.


    I'm not sure how you would like them to change.

    > OTOH, Paul Prescod is right and Python is primarily an
    > OOP language, so I would prefer to use real objects over
    > closures.


    Bah! Humbug! It's multi-paradigm, I tell you :)

    > So, I submit it as a feature request for Python 2.4. Maybe, if
    > enough people wants the feature, we may get it!


    In complete ignorance of the technical difficulties and second order
    consequences of allowing this, is second your request.


    [*] I have a feeling that "free variable binding" is not a correct way
    of expressing this, but I can't think of anything better just now.
    Jacek Generowicz, Feb 27, 2004
    #3
  4. Jacek Generowicz <> wrote in message news:<>....
    >
    > I don't think that Python's behaviour differs from other languages, in
    > this respect. For example in Common Lisp:
    >
    > * (defun make-adders (n)
    > (loop for i below n collect
    > (lambda (x) (+ x i))))
    > MAKE-ADDERS
    > * (destructuring-bind (add0 add1) (make-adders 2)
    > (list (funcall add0 0)
    > (funcall add1 0)))
    > (2 2)


    I don't know Common Lisp, but Scheme scope rules do what I expect:

    (define (make-adders n)
    (unfold (cut = <> n) (lambda (i) (lambda (x) (+ x i))) add1 0))

    (define-values (add0 add1) (apply values (make-adders 2)))

    (print (add0 0)) ;=> 0
    (print (add1 0)) ;=> 1

    My feeling about Python scope rules are mixed: in the beginning, I thought
    they were just broken, since they were not following my expectations
    (I don't know where those expectations were coming from, maybe from
    Pascal? I didn't know any functional language at that time, mah!).
    Later on, when I learned about the default argument tricks, I thought
    there was a logic in there, i.e. I may choose if I want to use the value
    of the "i" argument at the definition time or at the usage time
    explicitly and "explicit is better than implicit". However, now that
    I see that there are languages that implement the scope rule I
    would expect, and since I don't see a real life case where Python
    default of using the value of "i" as the usage time is useful, I am
    starting questioning them again.
    The only reason I see for the present state of the scope rules, is
    to discourage people from using closures and using objects instead.
    Which is not necessarely a bad thing, yes, but then I would need a better
    support from the language, and the ability to subclass FunctionType
    to be happy again ;)

    Michele Simionato
    Michele Simionato, Feb 28, 2004
    #4
  5. On 26 Feb 2004 21:58:41 -0800, (Michele Simionato) wrote:
    [...]
    >a real object. On the other hand, I am really bothered by the scoping
    >rules. My first post on this mailing list some time ago was caused by
    >problems I had with the surprising scope rules. Take for instance
    >this example:
    >
    >def make_adders(n):
    > return [(lambda x: x+i) for i in range(n)]
    >
    >add0,add1=make_adders(2)
    >
    >print add0(0) #=> 1
    >print add1(0) #=> 1
    >
    >This does not work as expected, and the solution is an ugly hack
    >with default arguments:


    I got misled by those near-each-other i's too, but though they both mean use-the-x-binding-in-
    a-particular-environment, the use is not at the same time. The code from the lambda
    expression evaluation does not look for i until it gets executed at some future time that could be
    arbitrarily long after the list comprehension is done with its i-value side effect in the local
    namespace of make_adders.

    I agree it is easy to make the wrong assumption, with those i's seemingly in the
    same expression context, but what should be the expectation? I think the problem is that
    lambda x: x+i is a lambda expression, that results in a function, but the i is a name,
    not a value in the context of the lambda expression per se. Thus the result is
    code that looks for the value of a name someplace, not code that refers to a
    specific value. If you want the name lookup to find different values, you must
    provide different name bindings, which you don't get in the single closure
    reflecting the environment of the list comprehension in make_adders above.

    But you can generate multiple closures with separate values
    (which maybe your scheme code really is doing?):

    >>> def make_separate_adders(n):

    ... return [(lambda sepval: lambda x: x + sepval)(i) for i in range(n)]
    ...
    >>> add0, add1 = make_separate_adders(2)
    >>> print add0(0)

    0
    >>> print add1(0)

    1

    Here the lambda being _executed_ as well as _evaluated_ in the list comprehension provides
    the closure environment for the lambda expression only being _evaluated_ (inside the one
    being executed, to generate code looking for sepval). UIAM ;-)

    [...]

    I think having function instances be first-class instances of a first-class function class would
    be a unifying move. (First-class class sounds funny) I wonder if we can integrate the creation
    of generators and coroutines (and even threads?) nicely in that, in some hierarchy. I'd like to
    think about that some, but no time for now. Ciao ;-)

    Regards,
    Bengt Richter
    Bengt Richter, Feb 28, 2004
    #5
  6. (Bengt Richter) wrote in message news:<c1qo74$s9j$0@216.39.172.122>...
    > But you can generate multiple closures with separate values
    > (which maybe your scheme code really is doing?):
    >
    > >>> def make_separate_adders(n):

    > ... return [(lambda sepval: lambda x: x + sepval)(i) for i in range(n)]
    > ...
    > >>> add0, add1 = make_separate_adders(2)
    > >>> print add0(0) 0
    > >>> print add1(0)

    > 1
    >
    > Here the lambda being _executed_ as well as _evaluated_ in the list comprehension provides
    > the closure environment for the lambda expression only being _evaluated_ (inside the one
    > being executed, to generate code looking for sepval). UIAM ;-)


    Hi Bengt! You reduced your contribution to this list in the
    last few months, it is nice to see you back :)

    For what concerns the trick you are talking about, it is the workaround
    I often use in this situation. Of course, I don't nest the two lambdas
    together and I write something more like

    def adder(sepval):
    return lambda x : x + sepval

    def make_separate_adders(n):
    return [adder(i) for i in range(n)]

    which does the right thing and is readable too. But still it is
    easy to forget that the helper function is needed and that using
    directly the lambda would not work. It is kind of error prone.
    Scheme is less error prone since it uses local loop variables and
    automatically generates separate closures:

    (define list-of-adders '())

    (let loop ((i 0))
    (set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
    (if (= i 1) 'exit (loop (add1 i))))

    (define add1 (first list-of-adders))
    (define add0 (second list-of-adders))

    (print (add0 0)) ;=> 0
    (print (add1 0)) ;=> 1

    You can get Python behavior if you explicitely use a global variable:

    (define list-of-adders '())
    (define i 0)

    (let loop ()
    (set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
    (if (= i 1) 'exit (begin (set! i (add1 i)) (loop))))

    (define add1 (first list-of-adders))
    (define add0 (second list-of-adders))

    (print (add0 0)) ;=> 1
    (print (add1 0)) ;=> 1

    So, in a sense, Python and Scheme have the same scope
    rules here, but in Scheme you don't see the issue
    because you just use a local variable, whereas
    Python by default use a global loop variable.


    Michele Simionato
    Michele Simionato, Feb 29, 2004
    #6
  7. (Michele Simionato) writes:

    > I don't know Common Lisp, but Scheme scope rules do what I expect:
    >
    > (define (make-adders n)
    > (unfold (cut = <> n) (lambda (i) (lambda (x) (+ x i))) add1 0))
    >
    > (define-values (add0 add1) (apply values (make-adders 2)))
    >
    > (print (add0 0)) ;=> 0
    > (print (add1 0)) ;=> 1


    I think that you are misleading yourself. You are comparing apples to
    quinces.

    Let's try to reproduce your Scheme code in Python. (I'm no Schemer,
    but here goes ...). Googling for "scheme unfold" comes up with
    something which I can fake in Python more or less like this:

    def unfold(p, f, g, seed):
    if p(seed):
    return []
    return [f(seed)] + unfold(p, f, g, g(seed))

    I'm guessing that add1 does this:

    def scheme_add1(n): # don't name-clash with the add1 we're about to
    make
    return n+1

    Allow me to simplify cut to the following, which I hope you will agree
    does what you want in this particular case:

    def cut(limit): # [*]
    def cut(x):
    return not x<limit
    return cut

    Now our user code becomes

    def make_adders(n):
    return unfold(cut(n), lambda i: lambda x:x+i, scheme_add1, 0)

    add0, add1 = make_adders(2)

    print add0(0) # => 0
    print add1(0) # => 1

    Note: Python gives the same results as Scheme.

    So, what's going on? The original problem was that all the closures
    that you generated in your Python code shared a single binding for
    their free variable. The solution is to create an inner binding unique
    to each closure, which shadows the original shared binding. In your
    Python solution you achieve this by creating a local variable i, which
    shadows the outer i (so the i in the lambda is no longer free, and the
    lambda is no longer a closure); in my CL solution I achieve this by
    creating a new binding of i using let (which is still outside of the
    scope of the lambda, so lambda's i remains free, and therefore it
    remains a closure). In your Scheme solution, and my Python translation
    of it, you'll see that we have:

    lambda i: lambda x:x+i

    which gets called by unfold to create the closure you want. Note that
    you are rebinding i in the outer lambda, each time unfold calls it to
    create a closure over the next number in the sequence, and that
    binding is visible only to the single closure being returned from that
    invocation of the outer lambda. So, to compare like to like, you
    should do the same in your original Python attempt.

    def make_adders(n):
    return [ (lambda i: lambda x:x+i)(i) for i in range(n) ]

    add0, add1 = make_adders(2)
    print add0(0) # => 0
    print add1(0) # => 1

    So, you see, Python's scoping rules are just like Scheme's, in this
    sense. Unfortunately, I don't think that it's so easy (or even
    possible?) to demonstrate it the other way around (ie show a Scheme
    snippet in which add0(0) == add1(0) == 1), because of Scheme's
    insistence on using recursion to implement iteration. You see, in both
    the Python and the Common Lisp versions of the original, there is a
    single biding of i in the loop, which gets modified as the loop goes
    about its business. In Scheme, the "loop variable" gets rebound each
    time the looping function (eg unfold) calls itself, so for each and
    every number in the sequence there is a seperate binding of seed, as
    well as a separate binding of i.

    Looking at it pictorially (variables with a * next to them are free,
    those with a + are bound; where the value of the binding does not
    change, the value is shown in parentheses; arrow up means binding
    found by free variable, arrow down means binding by argument passing):

    The Python case

    list comprehension: i+
    ^
    |
    ________ / \___
    / \
    add0: i* add1: i*


    Scheme style case

    unfold: seed+ (0)
    |
    |
    / \
    | ---(via scheme_add1)--
    | \
    V V
    lambda: i+ (0) unfold: seed+ (1)
    ^ |
    | V
    add0: i* lambda: i+ (1)
    ^
    |
    add1: i*

    The scoping rules in the two languages are pretty much the same, but
    the scheme code throws in lots of extra scopes which are absent from
    the (original) Python code.

    If you can find a way of making a genuine loop in Scheme, you should
    observe similar behaviour to that of the original Python example.

    > However, now that I see that there are languages that implement the
    > scope rule I would expect


    I hope that you agree, by now, that Scheme is _not_ an example of such
    a language.

    If you need more convincing, how about the following?

    guile> (define (make-adder i)
    (list (lambda (x) (+ x i))
    (lambda (n) (set! i (+ i n)))))
    guile> (define tmp (make-adder 5))
    guile> (define add5 (car tmp))
    guile> (define hehe (cadr tmp))
    guile> (add5 3)
    8
    guile> (hehe 1000)
    guile> (add5 3)
    1008

    Conclusion: the value of i that add5 sees, is definitely _not_ fixed
    at the time of creation of add5.

    > and since I don't see a real life case where Python default of using
    > the value of "i" as the usage time is useful,


    You are effectively saying that you don't see the point of using the
    run-time value of a variable, and that you'd rather be stuck with the
    first value that was ever assigned to the variable. You'd never claim
    this was the right thing for bound variables (including globals
    referenced in a local scope), would you? And I don't believe that you
    _really_ think it is appropriate for free variables either.

    Try the following:

    x = 3
    def show(): global x; print x
    show() # => 3
    x += 1
    show() # => 4

    def outer():
    x = 3
    def show(): print x
    show() # => 3
    x += 1
    show() # => 4
    outer()

    .... which prints out the numbers 3,4,3 and 4.

    You don't really believe that the last number should be 3, do you ?

    [In case anyone is in any doubt, the following is essentially
    identical

    def outer(x):
    def show(): print x
    show() # => 3
    x += 1
    show() # => 4
    outer(3)
    ]

    [*] def cut(limit):
    def cut(x):
    return not x<limit
    return cut

    Sorry, I should have written cut as follows, which is much clearer and
    more Pythonic, of course

    class cut:

    def __init__(self,limit):
    self.limit = limit

    def __call__(self,x):
    return not x<limit

    :)
    Jacek Generowicz, Mar 1, 2004
    #7
  8. Jacek Generowicz <> wrote in message news:<>...
    > I think that you are misleading yourself. You are comparing apples to
    > quinces.


    I thank you for the long post and the time you spent in clarifying the
    issue I had; however, you are speaking to a converted, since I already
    came to the same conclusion on my own (see my reply to Bengt Richter).

    > So, you see, Python's scoping rules are just like Scheme's, in this
    > sense. Unfortunately, I don't think that it's so easy (or even
    > possible?) to demonstrate it the other way around (ie show a Scheme
    > snippet in which add0(0) == add1(0) == 1)


    See my reply to Bengt.

    > Try the following:
    >
    > x = 3
    > def show(): global x; print x
    > show() # => 3
    > x += 1
    > show() # => 4
    >
    > def outer():
    > x = 3
    > def show(): print x
    > show() # => 3
    > x += 1
    > show() # => 4
    > outer()
    >
    > ... which prints out the numbers 3,4,3 and 4.
    >
    > You don't really believe that the last number should be 3, do you ?


    No, I agree with you on every respect. It is just that Scheme hides
    the issue; but it hides it so well that in practice you never do
    the mistake.
    Still, I maintain that the ability to subclass FunctionType would be nice ;)

    Michele Simionato
    Michele Simionato, Mar 1, 2004
    #8
  9. Thinking a bit more, the issue is not about the scope rules, it is about
    how the "for" loop is interpreted. Currently

    for i in iterable:
    <do_something i>

    is interpreted as

    try:
    it=iter(iterable)
    while True:
    i=it.next()
    <do_something i>
    except StopIteration:
    pass

    The issue would disappear if the "for" loop included an hidden function
    call and was interpreted as follows:

    try:
    it=iter(iterable)
    def helper(i):
    <do_something i>
    while True:
    helper(it.next())
    except StopIteration:
    pass

    For instance, in the example I am talking about,

    def make_adders(n):
    return [lambda x: x+i for i in range(n)]

    would be interpreted as

    def make_adders(n):
    try:
    adders=[]
    it=iter(range(2))
    def helper(i):
    adders.append(lambda x: x+i)
    while True:
    helper(it.next())
    except StopIteration:
    return adders

    Essentially, the "i" variable would be passed via the helper function
    call and at each iteration the lambda function would see a different
    value of it.

    I am proposing nothing here, just asking if it would make sense to
    have a looping construct acting this way (my guess is that this
    has already been proposed and discussed). This would have
    the added benefit of avoiding a non-local loop variable (i.e. a
    loop variable which exists even outside the loop) which is the
    actual unfortunate behavior.

    Michele Simionato
    Michele Simionato, Mar 1, 2004
    #9
  10. (Michele Simionato) writes:

    > Jacek Generowicz <> wrote in message news:<>...
    > > I think that you are misleading yourself. You are comparing apples to
    > > quinces.

    >
    > I thank you for the long post and the time you spent in clarifying the
    > issue I had; however, you are speaking to a converted, since I already
    > came to the same conclusion on my own (see my reply to Bengt Richter).


    Ah, OK, sorry. Didn't see that one at the time I was writing.

    > Still, I maintain that the ability to subclass FunctionType would be nice ;)


    I agreed with you originally on that, and I still do :)

    Just one thing bothers me about your conclusions in your reply to
    Bengt ...

    [As my discussion turned out longer than I originally intended, let me
    pull the summary up here to the top:

    In short, the reason Scheme hides the "feature" is that there is no
    iteration in Scheme, only recursion.
    ]

    (Michele Simionato) writes:

    > But still it is easy to forget that the helper function is needed
    > and that using directly the lambda would not work. It is kind of
    > error prone.


    Don't think about helper functions, think about the scopes that are
    intoduced. It's not the function that matters, it's the scope (the
    inner binding), which in Python can only be introduced by nesting a
    function, while in Scheme and CL you can use let.

    > Scheme is less error prone since it uses local loop variables


    The behaviour has nothing to do with globality or locality of the
    variable which is used. It depends on whether the same binding is
    being updated, or whether a new binding is being created each time
    around the loop.

    > and automatically generates separate closures:


    Python also greates separate closures. Scheme (and Python when you
    instruct it to do so) creates separate bindings of the same name so
    that the binding found by the free variables in the closure is
    different for each closure.

    > (define list-of-adders '())
    >
    > (let loop ((i 0))
    > (set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
    > (if (= i 1) 'exit (loop (add1 i))))
    >
    > (define add1 (first list-of-adders))
    > (define add0 (second list-of-adders))


    [...]

    > You can get Python behavior if you explicitely use a global variable:
    >
    > (define list-of-adders '())
    > (define i 0)
    >
    > (let loop ()
    > (set! list-of-adders (cons (lambda (x) (+ x i)) list-of-adders))
    > (if (= i 1) 'exit (begin (set! i (add1 i)) (loop))))


    In the second case you are updating the _single_ (global as it
    happens) binding [(set! i (add1 i))]. In the first case you do (loop
    (add1 i)) which certainly looks like it's passing the new value to a
    function which will create a new binding (though I suspect that
    there's some macrology behind the scenes). If, in the first case, you
    were to introduce a local variable to the loop and use (set! i (add1
    i)) to update it, just like you do to the global, you'd see the
    closures sharing the same value again.

    As I said before, iteration in Scheme is usually (always?) implemented
    via recursion, so it's difficult create a loop with a genuine looping
    variable.

    If you used recursion to implement iteration in Python, you'd get the
    same as in Scheme:

    def make_adders(i):
    if i < 0: return []
    return make_adders(i-1)+[lambda x:x+i]

    add0, add1 = make_adders(1)

    print add0(0) # => 0
    print add1(0) # => 1



    In short, the reason Scheme hides the "feature" is that there is no
    iteration in Scheme, only recursion.


    Disclaimer:

    I am no Scheme expert. There may well be Schemes which support real
    iteration, but I believe that the usual and even encouraged means of
    iterating in Scheme is via recursion. (Note, this is _not_ the case
    in Common Lisp.) Please correct me if you know this to be wrong.
    Jacek Generowicz, Mar 1, 2004
    #10
  11. (Michele Simionato) writes:

    > Thinking a bit more, the issue is not about the scope rules, it is about
    > how the "for" loop is interpreted. Currently


    Heh, this hadn't yet appeared on my server at the time I started
    writing my last followup to you. It seems that you have spotted for
    yourself, that which my article attempts to point out ... again :)

    > Essentially, the "i" variable would be passed via the helper function
    > call and at each iteration the lambda function would see a different
    > value of it.


    The important point is that it would see a different _binding_ of that
    name.

    > I am proposing nothing here, just asking if it would make sense to
    > have a looping construct acting this way


    Scheme is over there =>

    :)
    Jacek Generowicz, Mar 1, 2004
    #11
  12. Jacek Generowicz <> wrote in message news:<>...
    > (Michele Simionato) writes:
    >
    > > Thinking a bit more, the issue is not about the scope rules, it is about
    > > how the "for" loop is interpreted. Currently

    >
    > Heh, this hadn't yet appeared on my server at the time I started
    > writing my last followup to you. It seems that you have spotted for
    > yourself, that which my article attempts to point out ... again :)


    Yes, but still your thoughtful explanations may be useful to others
    who got catched by the same issues. Personally, I did understood
    quite well how Python worked, but I didn't realized until this thread
    that Scheme actually works the same, since the recursive loop hides
    the effect. I also thank you for correcting my sloppy and/or
    incorrect usage of the terminology. Nice discussion, anyway! :)


    Michele
    Michele Simionato, Mar 1, 2004
    #12
  13. (Michele Simionato) wrote in message news:<>...
    > Thinking a bit more, the issue is not about the scope rules, it is about
    > how the "for" loop is interpreted. Currently
    >
    > for i in iterable:
    > <do_something i>
    >
    > is interpreted as
    >
    > try:
    > it=iter(iterable)
    > while True:
    > i=it.next()
    > <do_something i>
    > except StopIteration:
    > pass
    >
    > The issue would disappear if the "for" loop included an hidden function
    > call and was interpreted as follows:
    >
    > try:
    > it=iter(iterable)
    > def helper(i):
    > <do_something i>
    > while True:
    > helper(it.next())
    > except StopIteration:
    > pass
    >
    > For instance, in the example I am talking about,
    >
    > def make_adders(n):
    > return [lambda x: x+i for i in range(n)]
    >
    > would be interpreted as
    >
    > def make_adders(n):
    > try:
    > adders=[]
    > it=iter(range(2))
    > def helper(i):
    > adders.append(lambda x: x+i)
    > while True:
    > helper(it.next())
    > except StopIteration:
    > return adders
    >
    > Essentially, the "i" variable would be passed via the helper function
    > call and at each iteration the lambda function would see a different
    > value of it.
    >
    > I am proposing nothing here, just asking if it would make sense to
    > have a looping construct acting this way (my guess is that this
    > has already been proposed and discussed). This would have
    > the added benefit of avoiding a non-local loop variable (i.e. a
    > loop variable which exists even outside the loop) which is the
    > actual unfortunate behavior.
    >
    > Michele Simionato


    Just to add another data point, I asked on comp.lang.functional and
    discovered that Haskell list comprehension works as I would expect,
    i.e. differently from Python:

    Prelude> let make-adders n = [ \x -> x + i | i <- [0..n-1] ]
    Prelude> let [add0,add1] = make_adders 2
    Prelude> add0 0
    0
    Prelude> add1 0
    1

    So, it looks like as if Haskell implements the idea I exposed.
    Still, it is to be discussed if the idea makes sense in Python.

    1. Assuming all "for" loops and list comprehensions are replaced
    with Haskell bindings rules, what about backward compatibility
    problems? For instance: is there enough code relying on the loop
    variable being available outside the loop?

    2. If, due to compatibility issues, it is not possible to change the
    binding rules of existing loops, would it make sense to propose
    these binding rules for Python 2.4 generator comprehension?
    This would not be backward incompatible, but would it generate
    confusion?

    3. Assuming the rules can be changed without real trouble, is the feature
    worth enough? Do people care about it, and is there somebody willing
    to take the job?


    Michele Simionato
    Michele Simionato, Mar 2, 2004
    #13
  14. Re: Feature request: subclassing FunctionType [Was: Some languageproposals]

    > 1. Assuming all "for" loops and list comprehensions are replaced
    > with Haskell bindings rules, what about backward compatibility
    > problems? For instance: is there enough code relying on the loop
    > variable being available outside the loop?


    Having a variable available outside of a loop can be both a convenience
    and inconvenience, depending on the situation. I have used it before,
    but acknowledge that it is poor form.


    > 2. If, due to compatibility issues, it is not possible to change the
    > binding rules of existing loops, would it make sense to propose
    > these binding rules for Python 2.4 generator comprehension?
    > This would not be backward incompatible, but would it generate
    > confusion?


    From what I can gather from the python-dev mailing list, generator
    expressions seem to work the same as list comprehensions, without the
    list. If I am incorrect, feel free to correct me.


    > 3. Assuming the rules can be changed without real trouble, is the feature
    > worth enough? Do people care about it, and is there somebody willing
    > to take the job?


    Changing the rules can cause real trouble. If loop variables were an
    unintended consequence, were not desireable by a large portion of Python
    users, and were used rarely, if at all, then I believe it is likely that
    would have been removed already.

    - Josiah
    Josiah Carlson, Mar 2, 2004
    #14
  15. Michele Simionato

    Bob Ippolito Guest

    On 2004-03-02 01:24:04 -0500, (Michele
    Simionato) said:

    > (Michele Simionato) wrote in message
    > news:<>...
    >> Thinking a bit more, the issue is not about the scope rules, it is about
    >> how the "for" loop is interpreted. Currently
    >>
    >> for i in iterable:
    >> <do_something i>
    >>
    >> is interpreted as
    >>
    >> try:
    >> it=iter(iterable)
    >> while True:
    >> i=it.next()
    >> <do_something i>
    >> except StopIteration:
    >> pass
    >>
    >> The issue would disappear if the "for" loop included an hidden function
    >> call and was interpreted as follows:
    >>
    >> try: it=iter(iterable)
    >> def helper(i):
    >> <do_something i>
    >> while True:
    >> helper(it.next())
    >> except StopIteration:
    >> pass
    >>
    >> For instance, in the example I am talking about,
    >>
    >> def make_adders(n):
    >> return [lambda x: x+i for i in range(n)]
    >>
    >> would be interpreted as
    >>
    >> def make_adders(n):
    >> try:
    >> adders=[]
    >> it=iter(range(2))
    >> def helper(i):
    >> adders.append(lambda x: x+i)
    >> while True:
    >> helper(it.next())
    >> except StopIteration:
    >> return adders
    >>
    >> Essentially, the "i" variable would be passed via the helper function
    >> call and at each iteration the lambda function would see a different
    >> value of it.
    >>
    >> I am proposing nothing here, just asking if it would make sense to
    >> have a looping construct acting this way (my guess is that this
    >> has already been proposed and discussed). This would have
    >> the added benefit of avoiding a non-local loop variable (i.e. a
    >> loop variable which exists even outside the loop) which is the
    >> actual unfortunate behavior.
    >>
    >> Michele Simionato

    >
    > Just to add another data point, I asked on comp.lang.functional and
    > discovered that Haskell list comprehension works as I would expect,
    > i.e. differently from Python:
    >
    > Prelude> let make-adders n = [ \x -> x + i | i <- [0..n-1] ]
    > Prelude> let [add0,add1] = make_adders 2
    > Prelude> add0 0
    > 0
    > Prelude> add1 0
    > 1
    >
    > So, it looks like as if Haskell implements the idea I exposed.
    > Still, it is to be discussed if the idea makes sense in Python.
    >
    > 1. Assuming all "for" loops and list comprehensions are replaced
    > with Haskell bindings rules, what about backward compatibility
    > problems? For instance: is there enough code relying on the loop
    > variable being available outside the loop?
    >
    > 2. If, due to compatibility issues, it is not possible to change the
    > binding rules of existing loops, would it make sense to propose
    > these binding rules for Python 2.4 generator comprehension?
    > This would not be backward incompatible, but would it generate
    > confusion?


    It could always be a __future__

    > 3. Assuming the rules can be changed without real trouble, is the feature
    > worth enough? Do people care about it, and is there somebody willing
    > to take the job?


    I know that I've been bitten by this before, but after doing it once or
    twice you learn how to get around it.. the biggest problem I have with
    list comprehensions is that I think they should have the same scoping
    rules as an inner function would, as opposed to a for loop (i.e.,
    shouldn't pollute namespace).

    -bob
    Bob Ippolito, Mar 2, 2004
    #15
  16. Michele Simionato

    Aahz Guest

    Scoping loops (was Re: Feature request: subclassing FunctionType)

    In article <>,
    Michele Simionato <> wrote:
    >
    >1. Assuming all "for" loops and list comprehensions are replaced
    > with Haskell bindings rules, what about backward compatibility
    > problems? For instance: is there enough code relying on the loop
    > variable being available outside the loop?


    Yes. Guido has Pronounced that this will stay, but...

    >2. If, due to compatibility issues, it is not possible to change the
    > binding rules of existing loops, would it make sense to propose
    > these binding rules for Python 2.4 generator comprehension?
    > This would not be backward incompatible, but would it generate
    > confusion?


    .....while the rules for generator expressions have not been finalized,
    Guido has agreed that they will have their own scope and that listcomps
    will be changed in Python 3 to have their own scope. You should read the
    python-dev thread on generator expressions yourself to understand what's
    being proposed.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Do not taunt happy fun for loops. Do not change lists you are looping over."
    --Remco Gerlich, comp.lang.python
    Aahz, Mar 2, 2004
    #16
  17. Michele Simionato

    Terry Reedy Guest

    Re: Feature request: subclassing FunctionType [Was: Somelanguageproposals]

    "Michele Simionato" <> wrote in message
    news:...
    > 1. Assuming all "for" loops and list comprehensions are replaced
    > with Haskell bindings rules, what about backward compatibility
    > problems? For instance: is there enough code relying on the loop
    > variable being available outside the loop?


    Guido has given things like the following as an intentional reason for the
    current rule (and for for...else):

    for item in seq:
    if acceptable(item): break
    else:
    item = None
    # item is now either the first acceptible item in seq or None

    If you do not like the iteration var remaining bound, you can explicitly
    delete it. If it is automatically deleted, constructions like the above
    are rendered more difficult.

    A list comp encapulates a specialized for loop whose specific job is to
    produce a list. It makes more sense to delete the binding automatically
    there since a) there is no way, when seq is empty, to force a binding to
    *something*, with else and b) the last item of the list is readily
    available anyway as result[-1].

    Terry J. Reedy
    Terry Reedy, Mar 2, 2004
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Antoon Pardon

    Some language proposals.

    Antoon Pardon, Feb 24, 2004, in forum: Python
    Replies:
    38
    Views:
    715
    Jacek Generowicz
    Mar 5, 2004
  2. Samuele Pedroni
    Replies:
    0
    Views:
    278
    Samuele Pedroni
    May 18, 2006
  3. Grizlyk
    Replies:
    1
    Views:
    331
    =?iso-8859-1?q?Erik_Wikstr=F6m?=
    Feb 19, 2007
  4. Alex Popescu

    MethodType/FunctionType and decorators

    Alex Popescu, Jul 4, 2007, in forum: Python
    Replies:
    12
    Views:
    474
    Alex Popescu
    Jul 6, 2007
  5. Dmitry Groshev

    Some syntactic sugar proposals

    Dmitry Groshev, Nov 15, 2010, in forum: Python
    Replies:
    30
    Views:
    687
    Tim Chase
    Dec 2, 2010
Loading...

Share This Page