pre-PEP: The create statement

[Steven Bethard]

Could this still make it in Python 2.5 even? If it's pushed hard
enough? I don't know if this has been discussed on the python-dev
mailing lists and what the reactions of python-devs and GvR was?

Unlikely. I haven't posted it to python-dev yet, and they've basically
said that anything that isn't yet in the release schedule (`PEP 356`_)
won't make it into Python 2.5. I will try to post it there soon though
as it seems like it's gotten pretty good feedback.

... _PEP 356: http://www.python.org/dev/peps/pep-0356/

STeVe

 

[Sion Arrowsmith]

This agrees with my scan (except I also found an occurrence of 'create'
in Tkinter).
BTW, I would be curious to see the script you are using for the
scanning.

Old-fashioned egrep '\Wcreate\W' and eyeball out the false positives
. I missed the Tkinter one because I don't appear to have it
installed.

 

[Steven Bethard]

This PEP proposes a generalization of the class-declaration syntax,
the ``create`` statement. The proposed syntax and semantics parallel
the syntax for class definition, and so::

create <callable> <name> <tuple>:
<block>

is translated into the assignment::

<name> = <callable>("<name>", <tuple>, <namespace>)

where ``<namespace>`` is the dict created by executing ``<block>``.
The PEP is based on a suggestion [1]_ from Michele Simionato on the
python-dev list.

And who needs a class statement after that?

create type A:
<block>

That's probably even more readable than class A, if not as familiar.
My biggest concern with this is the special arguments of the caller.
It breaks my heart that we couldn't do something like this:

create dict keymap:
A = 1
B = 2

And it'll probably confuse people as well. We ought to keep that in
mind.

This is definitely an important point. I've thought about this a bit
and I'm still not sure which way is the right way to go on this. If the
tuple and dict arguments actually get passed as *args and **kwargs
arguments, the create statement could be used with a much wider variety
of existing callables. I've updated the open issues section of the PEP
to point this out.

STeVe

 

[Azolex]

Steven Bethard wrote:
....

Optional Extensions
===================

Remove the create keyword
-------------------------

It might be possible to remove the create keyword so that such
statements would begin with the callable being called, e.g.:

module mod:
def f1():
...
def f2():
...

interface C(...):

as someone else noted, this is not the ideal example

...

However, this would probably add some complexity in the grammar and
so far I (Steven Bethard) have not been able to implement the feature
without the keyword.

Well, I can't pronounce myself on the technical difficulties, but I'd
like to point out that the proposal of the pep (especially without
keyword I'd say) offers part of what would be needed to endow python
with a standard bidirectional source-to-source transform to/fro xml. But
it might as well miss the opportunity...

I feel the python grammar offers more readable solutions (or
near-solutions) to the problems solved by the cumbersome grammar of xml.
So whenever I need to work with xml source or templates, I dream of a
reversible transformer that would allow me to work with a transparent
"mock-python" rendering of my xml file - it would replace closing tags
with indentations, attributes with keyword parameters syntax, cdatas
with triple quotes and perhaps xmlns with import statements - doing away
with all the unnecessary <xml/> cruft.

Symmetrically, I feel a (family of) standard xml rendition(s) of python
source would be quite useful. For one, it would facilitate access of
python source to vehicles designed for xml. For two, it would provide
python with an equivalent to lisp s-expression syntax. To illustrate :
imagine processing python source code using xslt over an xml
representation of python source with the relevant xslt itself expressed
in transparent python source.

So what I'd ideally want (and also to attract "foreign" programmers to
python) is to modify python syntax and semantics conservatively to eg
"xpython" that would make the following possible :

(a) given arbitrary "xpython" source code, there is a clearly obvious
way to express in xml a superficial parse of the code (ie a parse of
statement-level syntax).

(b) given arbitrary xml, there is a clearly obvious way to transform it
to mimetic "xpython" source code.

(c) the transforms are mutual inverses.

(d) running any "xpython" obtained from arbitrary xml will reconstruct
the original xml.

Similar wish, anyone ?

As relates to the pre-pep :

+1 for adding this general style of statement,

-0.5 on requiring a keyword,

-1 on choosing a *verb* keyword that forces to the imperative
interpretation of the code (since I'd want "xpython" to function as well
as a declarative language equivalent to xml)

-0.8 on the parameter syntax that restricts to positional parameters
while leaving out keyword arguments

Cheers.

 

[Michele Simionato]

Old-fashioned egrep '\Wcreate\W' and eyeball out the false positives
. I missed the Tkinter one because I don't appear to have it
installed.

I am impressed by your eyeball ability! I was unable to disentangle the
false
positive, so I was forced to write the script

 

[Michele Simionato]

My biggest concern with this is the special arguments of the caller.
It breaks my heart that we couldn't do something like this:

create dict keymap:
A = 1
B = 2

And it'll probably confuse people as well. We ought to keep that in
mind.



Let's not do this, really. A module should be one-to-one with a file,
and you should be able to import any module. Having in-line modules
complicates everything. And it becomes a misnomer. So, please, let's
get a better example. If you must, call it a scope or namespace.

You are right, I think I am responsible for the wrong choice of the
name.
What I had in mind was a namespace or, if you wish, a named dictionary.
An simple-minded implementation could be

class Namespace(object):
def __init__(self, name, args, dic):
self.__name__ = name
self.__dict__.update(dic)

create Namespace keymap:
A = 1
B = 2

and nothing forbids to have the Namespace class in some module of the
standard
library

Michele Simionato

 

[Steven Bethard]

You are right, I think I am responsible for the wrong choice of the
name.
What I had in mind was a namespace or, if you wish, a named dictionary.
An simple-minded implementation could be

class Namespace(object):
def __init__(self, name, args, dic):
self.__name__ = name
self.__dict__.update(dic)

create Namespace keymap:
A = 1
B = 2

I changed the PEP to use "namespace" instead of "module":

http://ucsu.colorado.edu/~bethard/py/pep_create_statement.txt
http://ucsu.colorado.edu/~bethard/py/pep_create_statement.html


STeVe

 

[Terry Reedy]

I'd advocate for the shortest possible keyword, and - maybe because
English is not my native language - I'd thus prefer "make" if any is
really required.

While currently +0 on the proposal, I am +1 on 'make' versus 'create'.
Aside from the above, factories don't create things, they make them.

Terry Jan Reedy

 

[Terry Reedy]

This is a very relevant question. I would expect the new keyword would
break lots
of modules. However measuring is better than speculating.

Please run also with alternatives, such as 'make'.

tjr

 

[Steven Bethard]

Please run also with alternatives, such as 'make'.

I ran a check on my stdlib with 'make' and all I got was:

$ count_names.py make "C:\Program Files\Python\Lib"
*** C:\Program Files\Python\Lib\site-packages\win32\test\handles.py ***
line 73: def make():
line 77: make()
Found 2 occurrences of 'make'

So it only happens in testing code in the stdlib, which isn't bad
breakage-wise as far as I'm concerned. So it looks like 'make' is a
much better option. I'll probably update the PEP and the patch to use
'make' soon unless I hear good reasons not to.


FWIW, here's what I got with 'create':

$ count_names.py create "C:\Program Files\Python\Lib"
*** C:\Program Files\Python\Lib\imaplib.py ***
line 379: def create(self, mailbox):
*** C:\Program Files\Python\Lib\bsddb\dbtables.py ***
line 132: def __init__(self, filename, dbhome, create=0, truncate=0,
mode=0600,
line 140: if create:
*** C:\Program Files\Python\Lib\bsddb\test\test_dbtables.py ***
line 54: filename='tabletest.db', dbhome=homeDir, create=1)
*** C:\Program Files\Python\Lib\distutils\cmd.py ***
line 312: def get_finalized_command (self, command, create=1):
line 318: cmd_obj = self.distribution.get_command_obj(command,
create)
*** C:\Program Files\Python\Lib\distutils\dist.py ***
line 828: def get_command_obj (self, command, create=1):
line 835: if not cmd_obj and create:
*** C:\Program Files\Python\Lib\lib-tk\Tkinter.py ***
line 1569: self.tk = _tkinter.create(screenName, baseName,
className, interactive, wantobjects, useTk, sync, use)
*** C:\Program Files\Python\Lib\test\test_mhlib.py ***
line 264: def create(n):
line 268: create(7)
line 269: create(8)
line 270: create(9)
line 285: create(10)
line 286: create(11)
line 287: create(12)
*** C:\Program Files\Python\Lib\test\test_site.py ***
line 75: pth_file.create()
line 88: pth_file.create()
line 109: def create(self):
Found 19 occurrences of 'create'


And here's my slightly modified version of Michele's script:

import glob
import optparse
import os
import sys
import token
import tokenize

def count_name(name, filename):
"Count the occurrences of a Python name in a script"
counter = 0
tokens_gen = tokenize.generate_tokens(open(filename).readline)
for tok_code, tok_value, (srow, scol), _, line in tokens_gen:
if tok_code == token.NAME and tok_value == name:
if not counter:
print '*** %s ***' % filename
counter += 1
print 'line %s: %s' %(srow, line),
return counter

if __name__ == '__main__':
parser = optparse.OptionParser(usage='%prog name dir [dir ...]')
options, args = parser.parse_args()
try:
name = args[0]
args[1] # trigger an IndexError if no dirs are provided
dirnames = args[1:]
except IndexError:
parser.error('wrong number of arguments')
total = sum(count_name(name, os.path.join(dirpath, filename))
for dirname in dirnames
for dirpath, _, filenames in os.walk(dirname)
for filename in filenames
if filename.endswith('.py'))
print 'Found %d occurrences of %r' % (total, name)


STeVe

 

[Ben Cartwright]

Is there a natural way
to extend this to other things, so that function creation can be
modified? For example:

create tracer fib(x):
# Return appropriate data here
pass

tracer could create a function that logs its entry and exit; behavior
could be modifiable at run time so that tracer can go away into oblivion.

Given the current semantics of create, this wouldn't work. What would be
reasonable syntax and semantics to make something like this possible?

The standard idiom is to use a function wrapper, e.g.

def tracer(f):
def wrapper(*args):
print 'call', f, args
result = f(*args)
print f, args, '=', result
return result
return wrapper

def fact(x):
if not x: return 1
return x * fact(x-1)
fact = tracer(fact) # wrap it

The decorator syntax was added in Python 2.4 to make the wrapper
application clearer:

@tracer
def fact(x):
if not x: return 1
return x * fact(x-1)

http://www.python.org/dev/peps/pep-0318

--Ben

 

[Michele Simionato]

I think looking at the occurrences in the standard library only is
unfair. I have a large "Zope +Plone+my stuff" installation and I get
154 occurrences of 'create' but nearly 0 occurrences
of 'make' (only a few in Zope/lib/python/BTrees/tests/testSetOps.py). I
guess 'make' sounds
too Lispy, this is why it is never used in Pythonland.

Michele Simionato

 

[jantod]

The PEP below should be mostly self explanatory. I'll try to keep the
most updated versions available at:

[snip]


Seems mostly clean. +1.

That's what Trojans said when they saw a wooden horse at the gates of
Troy

Serge.

QOTW!

 

[skip]

Michele> I think looking at the occurrences in the standard library only
Michele> is unfair.

In addition, when considering the standard library you need to search the
source repository, not just what's installed on your platform. I noticed in
your earlier post that you pointed your count_names script (very handy, btw)
at your /usr/lib/pythonN.M directory. Pointing it at the source repository
picks up stuff in the platform-dependent sections as well as searching
through other less-used code.

Michele> I guess 'make' sounds too Lispy, this is why it is never used
Michele> in Pythonland.

It doesn't seem lispish to me. OTOH, "make" seems more utilitarian than
"create". For instance, cooks make things while chefs create things.
Perhaps "make" is better anyway. The proposed statement is essentially a
namespace factory. I think of object factories as very utilitarian beasts.

Skip

 

[John J. Lee]

This agrees with my scan (except I also found an occurrence of 'create'
in Tkinter).
BTW, I would be curious to see the script you are using for the
scanning. Are you
using tokenize too? In am quite fond of the tokenize module

Having seen Sion's (non-existent) script, how about posting yours? I
regularly grep for things in Python code, and it hadn't occurred to me
that I could instead use module tokenize.

Module tokenize looks pretty simple, but would be nice to see some
code that has already been used.


John

 

[Steven Bethard]

Having seen Sion's (non-existent) script, how about posting yours? I
regularly grep for things in Python code, and it hadn't occurred to me
that I could instead use module tokenize.

Module tokenize looks pretty simple, but would be nice to see some
code that has already been used.

Michele's code:
http://groups.google.com/group/comp.lang.python/msg/e84e36ad10d78303

My derivative:
http://groups.google.com/group/comp.lang.python/msg/ed6dd53c267c7c77

STeVe

 

[BBands]

In the invaluable 'Dr. Dobb's Python-URL! - weekly Python news and
links' of April 17 Peter Otten writes: "Michele Simionato's little
script lets you search for a name in Python scripts, avoiding false
positives that a standard tool like grep would yield." Can someone
explain why this is so? I have attached the script below.

import sys, tokenize, token

def count_name(name, script):
"Count the occurrences of a Python name in a script"
counter = 0
for tok_code, tok_value, (srow, scol), (erow, ecol), line in \
tokenize.generate_tokens(file(script).readline):
if tok_code == token.NAME and tok_value == name:
counter += 1
print 'line %s: %s' %(srow, line),
if counter: print '*** %s ***\n' % script
return counter

if __name__ == '__main__':
name = sys.argv[1]
scripts = sys.argv[2:]
total = sum(count_name(name, script) for script in scripts)
print 'Found %d occurrences of %r' % (total, name)

jab--who laments the day that Doctor Dobbs' Journal of Computer
Callisthenics and Orthodontics changed its name.

 

[Fredrik Lundh]

In the invaluable 'Dr. Dobb's Python-URL! - weekly Python news and
links' of April 17 Peter Otten writes: "Michele Simionato's little
script lets you search for a name in Python scripts, avoiding false
positives that a standard tool like grep would yield." Can someone
explain why this is so?

consider grepping for "main" in this script:

def main():
# this is the main function
print "hello, main"

if __name__ == "__main__":
main()

a standard grep will find five instances. a word grep will find four. a name
token grep will find two.

</F>