Some language proposals.

[Antoon Pardon]

I'm rather new at this, so I don't know how I should
introduce this, nor whether these ideas are worth much
but here goes.

What I would like to change is access to variables on
an intermediate scope. Given the following example

def fun_1()

a = some_value

def fun_2()

# a is never the one above.
# on the left side of a statement.


The solution that I heard proposed was to use
a mutable variable. Something like


def fun_1():

a = [some_value]

def fun_2();

a[0] = new_value



And of course if you had a number of such variables
you could group then in an object


def fun_1():

class var_1:
pass

var_1.a = some_value
var_1.b = a_value_too

def fun_2():

var_1.a = new_value
var_1.b = next_value


Now my idea would be to look at the local variables in
a function like member variables of an object so that
you could write


def fun_1():

a = some_value
b = a_value_too

def fun_2():

fun_1.a = new_value
fun_1.b = next_value


This could maybe even extended further is seeing functions
like some kind of module which would allow something like
the following


def fun_1():

a = some_value
b = a_value_too

def fun_2():

from fun_1 import a, b

a = new_value
b = next_value


What do people think about this?
As far as I know the proposal doesn't break existing
code and seems in the spirit of python.


On a side note, what would people think about the idea of
a from ... import ... statement usable on any object? So
that we could do:

class A:
pass

A.a = ...
A.b = ...

from A import a, b

# from here a and be refer to A.a and A.b

 

[Jeff Epler]

Now my idea would be to look at the local variables in
a function like member variables of an object so that
you could write


def fun_1():

a = some_value
b = a_value_too

def fun_2():

fun_1.a = new_value
fun_1.b = next_value

This is not compatible with existing code. 'fun_1.a = new_value'
already has a meaning (it creates or modifies an attribute on the
function object), so it can't be used to modify the locals() of a
particular invocation of fun_1.

def fun_1():

a = some_value
b = a_value_too

def fun_2():

from fun_1 import a, b

a = new_value
b = next_value


What do people think about this?
As far as I know the proposal doesn't break existing
code and seems in the spirit of python.

This is not in the sprit of Python.

This already has a meaning:
from m import a
a = 3
.... it almost is the same as
import m as _m
a = m.a
del _m
a = 3
.... which basically just sets a to 3 (it doesn't modify the module m)

On a side note, what would people think about the idea of
a from ... import ... statement usable on any object? So
that we could do:

No, that won't do. Consider the following Python program:
sys = 64738
from sys import exit
exit()
The module name in 'import' and 'from ... import' doesn't refer to
objects in the current module's namespace, but in the namespace of
modules.

Anyway, because 'from .. import' is not magic, but just assignment,
there's no reason to prefer your proposed
from A import a, b
to
a, b = A.a, A.b
... for instance, it can't make 'a = 3' change A. It has a small
advantage that you don't need to type 'a' once on the left and once on
the right of '=', but that's about it.

Jeff

 

[Anton Vredegoor]

I'm rather new at this, so I don't know how I should
introduce this, nor whether these ideas are worth much
but here goes.

I'm rather new at criticizing new language proposals so forgive me for
not fighting new ideas ferociously enough But seriously, I think
it's better to keep an eye open to new proposals. Even if things have
been discussed before, the fact that the same ideas come up again and
again should mean something in itself. Also a lot of proposals come in
the company of new ideas that have not yet been evaluated.

Now my idea would be to look at the local variables in
a function like member variables of an object so that
you could write


def fun_1():

a = some_value
b = a_value_too

def fun_2():

fun_1.a = new_value
fun_1.b = next_value


This could maybe even extended further is seeing functions
like some kind of module which would allow something like
the following


def fun_1():

a = some_value
b = a_value_too

def fun_2():

from fun_1 import a, b

a = new_value
b = next_value


What do people think about this?
As far as I know the proposal doesn't break existing
code and seems in the spirit of python.

I don't see much merit in this, since functions are designed to go out
of scope once they are executed. Having sub-scopes change names higher
up is considered a no-no in Python, although new developments like
"properties" are already nibbling little chunks away from this
concept.

However, in the case of *generator functions* this situation changes
dramatically, so you might score some points there. The road to take
would be something along the lines of "practicality beats purity".

On a side note, what would people think about the idea of
a from ... import ... statement usable on any object? So
that we could do:

class A:
pass

A.a = ...
A.b = ...

from A import a, b

# from here a and be refer to A.a and A.b

This raises the question of why not to use a class to begin with even
for your earlier proposals. Anyway, this can be accomplished easily
with:

a,b = A.a,A.b

If you still want to go through with this, have a look at this link:

http://us.st5.yimg.com/store4.yimg.com/I/demotivators_1780_2263455

Anton

 

[Terry Reedy]

What I would like to change is access to variables on
an intermediate scope.

As I understand, you want write (rebind) access in addition to read
access -- without having to wrap the variable in a mutable, which one can
do at present. Some proposals to do this have been discussed on the PyDev
list. They should be listed in Brett Cannon's summaries, which are
archived on Python.org.

Bottom line so far: given that write access is only clumsy (sometimes,
through wrapping) rather than impossible, no proposal struck enough people
as worth the bother of a syntax change. Nested functions are much rarer
than unnested. Needing to rebind an outer local, rather than merely read
or mutate, is rarer still.

Terry J. Reedy

 

[Jacek Generowicz]

The solution that I heard proposed was to use
a mutable variable. Something like


def fun_1():

a = [some_value]

def fun_2();

a[0] = new_value

Could somebody point me to an explanation of why closures are broken
in this way in the first place, please ?

 

[Jacek Generowicz]

Nested functions are much rarer than unnested. Needing to rebind an
outer local, rather than merely read or mutate, is rarer still.

Chicken and Egg.

Sapir-Whorf.

I sumbit to you that read-only closures are rare in Python because
they are a recent addition to the language.

I sumbit to you that read-write closures are so much rarer still,
because they require an ugly hack to work.

 

[Antoon Pardon]

[ ... ]

What do people think about this?
As far as I know the proposal doesn't break existing
code and seems in the spirit of python.

I don't see much merit in this, since functions are designed to go out
of scope once they are executed. Having sub-scopes change names higher
up is considered a no-no in Python,

I don't understand why. It already is possible with global variables.
I don't see much difference between a function changing a higher up
variable that is global or just in between.

This raises the question of why not to use a class to begin with even
for your earlier proposals. Anyway, this can be accomplished easily
with:

a,b = A.a,A.b

Well maybe my wording was off, but this doesn't accomplisch what I
want since after your statement followed by: a = new_value; A.a will
not be changed.

 

[Paul Prescod]

...

Chicken and Egg.

Sapir-Whorf.

I sumbit to you that read-only closures are rare in Python because
they are a recent addition to the language.

I sumbit to you that read-write closures are so much rarer still,
because they require an ugly hack to work.

I disagree. Closures are rare in Python because Python is primarily an
OOP language.

http://www.kimbly.com/blog/000063.html

"""The venerable master Qc Na was walking with his student, Anton.
Hoping to prompt the master into a discussion, Anton said "Master, I
have heard that objects are a very good thing - is this true?" Qc Na
looked pityingly at his student and replied, "Foolish pupil - objects
are merely a poor man's closures."

Chastised, Anton took his leave from his master and returned to his
cell, intent on studying closures. He carefully read the entire
"Lambda: The Ultimate..." series of papers and its cousins, and
implemented a small Scheme interpreter with a closure-based object
system. He learned much, and looked forward to informing his master of
his progress.

On his next walk with Qc Na, Anton attempted to impress his master by
saying "Master, I have diligently studied the matter, and now understand
that objects are truly a poor man's closures." Qc Na responded by
hitting Anton with his stick, saying "When will you learn? Closures are
a poor man's object." At that moment, Anton became enlightened. """

Paul Prescod

 

[Paul Prescod]

Could somebody point me to an explanation of why closures are broken
in this way in the first place, please ?

a = 5
def b():
a = 6
print a
b()
print a

What will this print?

5
6

Okay, then.

a = 5
def b():
a = 6
def c():
a = 7
print a
print a
c()

print b()()
print a

What does this print?

7
6
5

Python uses a single syntax for declarating a variable and assigning to
it. This causes a problem for Lisp purists but is otherwise helpful.

Paul Prescod

 

[Jacek Generowicz]

a = 5
def b():
a = 6
def c():
a = 7
print a
print a
c()

print b()()

Methinks you forgot to return something somewhere.

Python uses a single syntax for declarating a variable and assigning
to it.

Ah, thank you, _this_ is the salient point ( ... though I'm not sure
what "declarating" means, but I'm pretty sure that Python does not do
it to variables .

But what happens in other languages which support closures, but have
no syntactic distinction between binding and re-binding ?

This causes a problem for Lisp purists but is otherwise helpful.

Lisp causes problems for the simple-minded, but is otherwise helpful.

(Infer as many winks/smileys as you implied in your original.)

 

[Jacek Generowicz]

I disagree. Closures are rare in Python because Python is primarily an
OOP language.

I disagree, Python is a multi-paradigm language ... I fail to see how
this has any bearing on the use of closures ... all of which is
irrelevant to my original point, which was to note out that saying
"people don't use it much" is not a very convincing argument for not
fixing something that is broken ... because the very fact that it is
broken probably contributes to people not using it much.

 

[Michael Hudson]

Chicken and Egg.

Sapir-Whorf.

Possibly. Also see Paul's response.

I sumbit to you that read-only closures are rare in Python because
they are a recent addition to the language.

Actually, I think uses of nested scopes are fairly common -- in test
suites! It was writing test cases that motivated us to implement them
for PyPy...

Cheers,
mwh

 

[Michael Hudson]

I disagree, Python is a multi-paradigm language ... I fail to see how
this has any bearing on the use of closures ...

Really? Paul is saying "if you are in a situation in Python where you
want to use a read-write closure, you'll probably be happier if you
use an object-based solution instead".

Now you may *disagree* with this point, but I find it hard to believe
you don't *see* it.

I agree with him, FWIW. Using closures to fake objects sucks (in
scheme or CL as much as in Python).

Can you post an example of using a read-write closure that you think
wouldn't be better off as an object (invent syntax as necessary...).
No-one was very convincing at this last time it went around on
python-dev.

all of which is irrelevant to my original point, which was to note
out that saying "people don't use it much" is not a very convincing
argument for not fixing something that is broken ...

OK, how about the argument above?

because the very fact that it is broken probably contributes to
people not using it much.

Well, true as this may be, I hypothesise that it wouldn't be used much
if it wasn't "broken" and if it was used most of the time you'd wish
it wasn't. Not a particularly verifiable statement, I'll grant, but
things like this is what Guido is for

Cheers,
mwh

 

[Jacek Generowicz]

Really? Paul is saying "if you are in a situation in Python where you
want to use a read-write closure, you'll probably be happier if you
use an object-based solution instead".

Now you may *disagree* with this point, but I find it hard to believe
you don't *see* it.

Aaah, OK, _if_ you think that closures are _only_ a way of faking
objects, _then_ claiming that Python is primarily OO does have a
bearing on the use of closures. Forgive me for not pursuing
this tangent any further.

Using closures to fake objects sucks (in scheme or CL as much as in
Python).

And using objects (err ... instances of classes) to fake stateful
functions sucks just as much.

When I need "objects" I use objects, and when I need closures I want
to be able to use closures.

Can you post an example of using a read-write closure that you think
wouldn't be better off as an object (invent syntax as necessary...).
No-one was very convincing at this last time it went around on
python-dev.

Sure. I have some stateful methods of classes (which I create
dynamically and stick onto the class as the information about the
existence of the method becomes available). By implementing them as
closures I can just stick them on to the class. If I were to implement
them as instances then I'd have to reimplement all the descriptors
that take care of turning functions into bound or unbound methods.

[At this point I'd love someone to step up and show me how to re-use
the existing FunctionType descriptors in my own classes.]

Another example. I make quite heavy use of a trivial memoizer. The
closure verison is much shorter, clearer and faster. (I want it to be
writeable because I sometimes want to 'prime' the cache, or do other
things to it ... so I'm giving myself access to the cache by binding
the cache as a function attribute of the closure, before returning it
from the memoizer)

In short, whenever I want a _function_ which happens to have some
internal state, I'd much rather use a closure than an instance of a class.

OK, how about the argument above?

My original point, although prompted in a context which discussed
closures, in independent of closures.

[Yes, I happen to wish that closures were writeable as god intended
them to be, dammit , but that is completely independent of my
objection to the self-fulfilling prophecy argument.]

 

[Jacek Generowicz]

Actually, I think uses of nested scopes are fairly common

While I don't pretend to have any idea how much the "typical" Python
programmer uses nested scopes, I am tempted to agree with you that
their use is quite common. However, I did not want to start a debate
(with Terry or anyone else) about the _absolute_ frequency of nested
functions, and so decided to phrase my statement in terms of an
agremment on the _relative_ infrequency of nested functions as
compared to unnested ones.

 

[Anton Vredegoor]

I don't see much difference between a function changing a higher up
variable that is global or just in between.

If you want a function with its own namespace, that is already
possible as you showed yourself in an earlier post. On the other hand,
if a function has its own namespace what is the difference between a
function and a class? Why not use a class in the first place?

Remember that "global" is just a way for a function to tell the
interpreter that this variable belongs to the global namespace.

There is some inconsistency if one has a function inside a function
and that function has a global 'x' but the outer function doesn't have
a global 'x'. Modifications of 'x' from the deepest function do not
reach the global namespace, in fact they do not even reach the
namespace of the containing function.

Only if both functions have a global 'x' the name is visible at the
global and intermediate level.

However there is already a mechanism to solve these kind of problems:
namespaces! Why not use them?

Well maybe my wording was off, but this doesn't accomplisch what I
want since after your statement followed by: a = new_value; A.a will
not be changed.

For that to work you'd have to imbue names with new magical powers.
They would have to have some way of knowing who their fellow holders
are with whom they share their objects and inform them about their new
responsibilities.

The fellow objects would need some instructions as to how to handle
such things. Should they hold on to their previous bindings or let go
and use the new objects? Maybe some automatic message passing between
objects "behind the covers" would be necessary. Maybe it's a great
idea or maybe we would get hit over the head with a stick. I don't
know.

By the way, would it then be really, really black magic to be able to
call an object by its *real* name and force all holders to obey? One
ring to rule them all!

.... much, much, later ...

Message from your Atoned Governor.

Ok, after finally having relinquished the ring of power and freely
dropping it into the crater, I would like to make a public
announcement.

It is a really bad idea to indiscriminately rip the structure and
inner cohesion and the "whole is more than the sum of the parts"
quality of existing cooperations of objects apart in order to gain
access to a little trinket, and from now on I will desist from such
actions until there is a way to replace the lost objects with
functionality that is the same or better, according to the judgement
of said cooperations, in order not to disrupt working ecosystems.

Anton

 

[Paul Prescod]

I disagree, Python is a multi-paradigm language ...

In Python functions are objects but objects are not functions. In (e.g.)
Scheme the opposite is true.

... I fail to see how
this has any bearing on the use of closures ... all of which is
irrelevant to my original point, which was to note out that saying
"people don't use it much" is not a very convincing argument for not
fixing something that is broken ... because the very fact that it is
broken probably contributes to people not using it much.

But the more important point is that people do not NEED it much. Guido
writes hundreds of lines of Python code per week. If he often ran into
situations where a mutable closure would make a big difference then he
would presumably find some way of doing it. The people who want this
seem most often to be people trying to import their coding styles from
another language.

Paul Prescod

 

[Paul Prescod]

While I'm thinking about it, let me refute your Sapir-Wharf claim (BTW,
its been pretty widly debunked in linguistics)

Before Python had nested scopes at all, it was VERY common to fake them
using a linguistic trick: default arguments. This idiom was very common
in Python's source base. Python didn't have a proper way to do the
nesting but people figured out how to do it anyway and did it a LOT.
(similarly people often emulate OO in C)

The "wrap your mutables" feature is by comparison _much less intrusive_
but I don't ever see it in real Python code. This suggests to me that
people don't need the feature that badly.

...
And using objects (err ... instances of classes) to fake stateful
functions sucks just as much.

Let's step above the words "function" and "object" for a second.
Sometimes you need to combine mutable state and code into a single unit.
Python has a first-class feature for doing that: the object. That's the
implementation of the requirement.

But you are expressing an _implementation_ as a _requirement_. "I need a
combination of state and data and it must be a function." Why?

Sure. I have some stateful methods of classes (which I create
dynamically and stick onto the class as the information about the
existence of the method becomes available).

Why are the METHODS stateful rather than keeping their state in the
class? If I had to maintain your code I would much rather that you keep
state in the same place all the other Python programmers do.

But if you must do this then use the well-known mutable closure hack.
It'll take you just a few extra characters of code. When we start seeing
dozens or hundreds of people making use of it then we'll know the
community really needs it (and isn't just trying to turn Python into
Lisp for its own sake).

... By implementing them as
closures I can just stick them on to the class. If I were to implement
them as instances then I'd have to reimplement all the descriptors
that take care of turning functions into bound or unbound methods.

Or you could wrap them in functions.

inst = Class()

someclass.method = lambda x, *args, **kwargs: inst(*args, **kwargs)

You've cleverly found one of the very few places where Python
distinguishes between "true" functions and callable objects. If one is
looking at the overall Python system, the bug is in the way functions
are bound. In general, Python very seldom distinguishes between objects
based on type (rather than behaviour) and in particular, seldom
distinguishes between functions and callable objects.

I would argue that classes should distinguish between
attributes-to-be-treated-as-methods and
attributes-to-be-treated-as-just-attributes using something other than
type. Consider the following program:

class a:
pass

def func():
pass

lst = [1, 2, 3, "a", "b", "c", object(), lambda x:x ]

for x in lst:
a.b = x
assert a.b is x, "What???? %s"% x
print "Okay", x

Frankly I think it is unpythonic that a second after assignment to a.b
the thing I get back is different than the thing I put in.

But now we're talking about a pretty subtle bug that has been in Python
for over ten years and it so seldom bites anyone that it is probably nto
worth fixing. (if I were to fix it I would add an add_method method to
classes)

Another example. I make quite heavy use of a trivial memoizer. The
closure verison is much shorter, clearer and faster.

Well it boils down to a question of who would believe it is clearer.
Lisp programmers think mutable closures are clearer. Perl programmers
think having a variety of different ways to spell "if" is clearer than
having just one. Part of the Zen of Python is try to encourage different
camps towards common spellings of idioms.

bash-2.05a$ python -c "import this" | grep way
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

In short, whenever I want a _function_ which happens to have some
internal state, I'd much rather use a closure than an instance of a class.

Barring the one ancient design misfeature above, callable instances are
almost always good replacements for functions.

> ...

OK, how about the argument at the top? If people needed it they would
fake it as they do many other missing feature (in Python and other
languages). That they don't fake it despite how easy it is to do so
indicates that they don't need it.

Paul Prescod

 

[Michael Hudson]

Aaah, OK, _if_ you think that closures are _only_ a way of faking
objects, _then_ claiming that Python is primarily OO does have a
bearing on the use of closures. Forgive me for not pursuing
this tangent any further.

OK, you really hadn't seen this argument before. You must be new here
(or at least, new to this particular dispute).

And using objects (err ... instances of classes) to fake stateful
functions sucks just as much.

Hmm. Really? I'm not sure that I often find myself wanting a
"stateful function". I think I probably usually use bound methods.
This could be the tail wagging the dog, of course.

When I need "objects" I use objects, and when I need closures I want
to be able to use closures.

So could this

(trivially, you never *need* closures, just the same as you don't
*need* objects, there's a rather dull and unhelpful isomorphism
between the two concepts).

Sure. I have some stateful methods of classes (which I create
dynamically and stick onto the class as the information about the
existence of the method becomes available). By implementing them as
closures I can just stick them on to the class. If I were to implement
them as instances then I'd have to reimplement all the descriptors
that take care of turning functions into bound or unbound methods.

Which isn't that hard... I think a code example might lend a bit more
clarity.

[At this point I'd love someone to step up and show me how to re-use
the existing FunctionType descriptors in my own classes.]

Post some code, and I'll have a go.

Another example. I make quite heavy use of a trivial memoizer. The
closure verison is much shorter, clearer and faster.

I'll grant you this: I've done the same.

(I want it to be writeable because I sometimes want to 'prime' the
cache, or do other things to it ... so I'm giving myself access to
the cache by binding the cache as a function attribute of the
closure, before returning it from the memoizer)

Um, here, unless I misunderstand you, you don't want to mutate the
closed over binding (which is what Python currently doesn't let you
do) but mutate the object the closed over binding is bound to. In
this respect, Python is no different from anything else, you need to
arrange some way to expose said object outside the closure.

What am I missing?

In short, whenever I want a _function_ which happens to have some
internal state, I'd much rather use a closure than an instance of a
class.

Maybe, once you've been assimilated into the Python Borg mindset, you
don't, actually.

There are sort of two extreme possibilities here:

1) I am so blinded by the fact that I know Python as of 2.3 FAR better
than any other language that I don't see how wonderful Python would
be if "full closures" were added.

2) Full closures just aren't that appropriate to Pythonic programming.

The truth, almost certainly, is somewhere in between these points.

My original point, although prompted in a context which discussed
closures, in independent of closures.

Fine. Let's have a real argument instead

Cheers,
mwh

 

[Aahz]

Sure. I have some stateful methods of classes (which I create
dynamically and stick onto the class as the information about the
existence of the method becomes available). By implementing them as
closures I can just stick them on to the class. If I were to implement
them as instances then I'd have to reimplement all the descriptors
that take care of turning functions into bound or unbound methods.

Why not use callable instances?