PEP 289: Generator Expressions (please comment)

R

Raymond Hettinger

Peter Norvig's creative thinking triggered renewed interest in PEP 289.
That led to a number of contributors helping to re-work the pep details
into a form that has been well received on the python-dev list:

http://www.python.org/peps/pep-0289.html

In brief, the PEP proposes a list comprehension style syntax for
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)
min(d.temperature()*9/5 + 32 for d in days)
Set(word.lower() for word in text.split() if len(word) < 5)
dict( (k, somefunc(k)) for k in keylist )
dotproduct = sum(x*y for x,y in itertools.izip(xvec, yvec))
bestplayer, bestscore = max( (p.score, p.name) for p in players )

Each of the above runs without creating a full list in memory,
which saves allocation time, conserves resources, and exploits
cache locality.

The new form is highly expressive and greatly enhances the utility
of the many python functions and methods that accept iterable arguments.


Raymond Hettinger
 
D

Daniel Dittmar

Raymond said:
bestplayer, bestscore = max( (p.score, p.name) for p in players )

Each of the above runs without creating a full list in memory,
which saves allocation time, conserves resources, and exploits
cache locality.

Why should generator comprehension be useful only for arguments?

counter proposal:
g[(p.score, p.name) for p in players ]

This works similar to r'\' for raw strings and u'x' for unicode strings.

This has the advantage that it can be more easily extended:
d[(p.name, p.score) for p in players] creates a dictionary
s[(p.score, p.name) for p in players] creates a sorted list
and
p[$*-=%/sd/!] allows to embed perl code

Daniel
 
P

Peter Hansen

Daniel said:
Raymond said:
bestplayer, bestscore = max( (p.score, p.name) for p in players )

Each of the above runs without creating a full list in memory,
which saves allocation time, conserves resources, and exploits
cache locality.

Why should generator comprehension be useful only for arguments?

counter proposal:
g[(p.score, p.name) for p in players ]

This works similar to r'\' for raw strings and u'x' for unicode strings.

Except that r'' can mean nothing else, syntactically, while g[something]
could be a dictionary lookup using <something> as the key.
 
R

Raymond Hettinger

Daniel Dittmar said:
Why should generator comprehension be useful only for arguments?

It is not limited to just function arguments.
It is a first class citizen. You can use it for anything you want.


Raymond Hettinger
 
S

Stephen Horne

Peter Norvig's creative thinking triggered renewed interest in PEP 289.
That led to a number of contributors helping to re-work the pep details
into a form that has been well received on the python-dev list:

I like it. Simple and effective.
 
R

Rainer Deyke

Raymond said:
It is not limited to just function arguments.
It is a first class citizen. You can use it for anything you want.

At the very least, it would require parentheses if you want to create a list
containing exactly one such generator.

[(x for x in y)] # List containing generator
[x for x in y] # List comprehension
 
R

Raymond Hettinger

[Rainer Deyke]
At the very least, it would require parentheses if you want to create a list
containing exactly one such generator.

[(x for x in y)] # List containing generator
[x for x in y] # List comprehension

Yes, that's what the pep call for.


Raymond Hettinger
 
J

John Roth

Raymond Hettinger said:
Peter Norvig's creative thinking triggered renewed interest in PEP 289.
That led to a number of contributors helping to re-work the pep details
into a form that has been well received on the python-dev list:

http://www.python.org/peps/pep-0289.html

In brief, the PEP proposes a list comprehension style syntax for
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)
min(d.temperature()*9/5 + 32 for d in days)
Set(word.lower() for word in text.split() if len(word) < 5)
dict( (k, somefunc(k)) for k in keylist )
dotproduct = sum(x*y for x,y in itertools.izip(xvec, yvec))
bestplayer, bestscore = max( (p.score, p.name) for p in players )

Each of the above runs without creating a full list in memory,
which saves allocation time, conserves resources, and exploits
cache locality.

The new form is highly expressive and greatly enhances the utility
of the many python functions and methods that accept iterable arguments.

I like it.

The final example in point 2 is incorrect. It currently says:

g = (x**2 for i in range(10))

x and i should be changed to match.

John Roth







John Roth
 
R

rzed

Raymond Hettinger said:
Peter Norvig's creative thinking triggered renewed interest in PEP 289.
That led to a number of contributors helping to re-work the pep details
into a form that has been well received on the python-dev list:

http://www.python.org/peps/pep-0289.html

In brief, the PEP proposes a list comprehension style syntax for
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)
min(d.temperature()*9/5 + 32 for d in days)
Set(word.lower() for word in text.split() if len(word) < 5)
dict( (k, somefunc(k)) for k in keylist )
dotproduct = sum(x*y for x,y in itertools.izip(xvec, yvec))
bestplayer, bestscore = max( (p.score, p.name) for p in players )

Each of the above runs without creating a full list in memory,
which saves allocation time, conserves resources, and exploits
cache locality.

The new form is highly expressive and greatly enhances the utility
of the many python functions and methods that accept iterable arguments.

I like it! How about in conjunction with the print statement?
print "%s\n" % (s for s in strlist)
??
 
C

Christoph Becker-Freyseng

Raymond said:
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)


+3.1415

Especially I like the boost of use of many python-standard-functions --
like min, max, ... :)
Furthermore the syntax is quite clear and readable.
However I'm a bit worried if some could get into trouble because of the
difference between using Generator Expressions in function-call with one
argument and a call with more than one arguments.


Christoph Becker-Freyseng


BTW: Python is getting more and more stuff. So I fear that some code
might get harder to read for newbies (not knowing all the new stuff).

Would it be possible to make an expression-/statement-parser that can
analyse a code snippet (one line of python-code maybe more) and tell the
user what syntax-constructs were used.

This might be a big help when reading code and if you don't know some
special construct.

e.g.:
newstuff.py:
...
sum(x*x for x in roots)
...
??? Mmmm What's that
pyscryer "sum(x*x for x in roots)"
<Function-Call>(<Generator Expression>)

Then you can search the Internet to close your knowledge gap (you can't
search for "sum(x*x for x in roots)", but "python Generator Expression"
might be succesful)
 
M

Mike Rovner

rzed said:
I like it! How about in conjunction with the print statement?
print "%s\n" % (s for s in strlist)

It's not better than

for s in strlist:
print s

or I didn't understand your intent.

Mike
 
P

Paul Rubin

In brief, the PEP proposes a list comprehension style syntax for
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)

This is a little confusing. Consider:

1) [x*x for x in roots]
2) (x*x for x in roots)

#2 looks like it should be a "sequence comprehension" and not a
generator expression. I'd go back to your earlier proposal of
using a yield keyword if you want a generator expression:

sum(yield x for x in roots)

Note there's no builtin function "sum". It's always annoyed me that
operator.add always takes exactly two arguments, so you have to use
reduce. I'd like to be able to say operator.add(2,3,4), or in the
generator expression case, operator.add(yield x*x for x in roots).
 
E

Erik Max Francis

rzed said:
I like it! How about in conjunction with the print statement?
print "%s\n" % (s for s in strlist)
??

Presuming you want this to result in a series of print statements, this
sounds like a very bad idea. All the PEP proposed is a new way of
creating an anonymous generator that uses a syntax similar to that of
list comprehensions; it doesn't actually change the way generators
themselves are already handled by the language. Consider:
.... i, j = 0, 1
.... yield i
.... yield j
.... while True:
.... i, j = j, i + j
.... yield j
.... <generator object at 0x402e3bcc>

What you're suggesting would grievously sacrifice consistency at the
expense of a tiny little bit of convenience. It isn't worth it.
 
B

Bjorn Pettersen

[...]
BTW: Python is getting more and more stuff. So I fear that some code
might get harder to read for newbies (not knowing all the new stuff).

Would it be possible to make an expression-/statement-parser that can
analyse a code snippet (one line of python-code maybe more) and tell the
user what syntax-constructs were used.

This might be a big help when reading code and if you don't know some
special construct.

e.g.:
newstuff.py:
...
sum(x*x for x in roots)
...
??? Mmmm What's that
pyscryer "sum(x*x for x in roots)"
<Function-Call>(<Generator Expression>)

Then you can search the Internet to close your knowledge gap (you can't
search for "sum(x*x for x in roots)", but "python Generator Expression"
might be succesful)

Something like:

compiler.parse("[x*x for x in roots]")

(not completely what you want, but close :)

-- bjorn
 
B

Bjorn Pettersen

In brief, the PEP proposes a list comprehension style syntax for
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)
[..]

Note there's no builtin function "sum". It's always annoyed me that
operator.add always takes exactly two arguments, so you have to use
reduce.
[..]

time to upgrade?

f:\>python
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.45

-- bjorn
 
D

Donald 'Paddy' McCarthy

Paul said:
In brief, the PEP proposes a list comprehension style syntax for
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)


This is a little confusing. Consider:

1) [x*x for x in roots]
2) (x*x for x in roots)

#2 looks like it should be a "sequence comprehension" and not a
generator expression. I'd go back to your earlier proposal of
using a yield keyword if you want a generator expression:

sum(yield x for x in roots)
I too prefer the above use of yield because then you can always
associate generators with the yield keyword.

Pad.
 
R

Rainer Deyke

Donald said:
I too prefer the above use of yield because then you can always
associate generators with the yield keyword.

I don't, because using 'yield' here would lead to greater confusion when
using generator expressions in generator functions.


yield [x for x in y] # Yields a list
yield x for x in y # Creates and discards a generator
yield yield x for x in y # Yields an iterator
 
H

Holger Krekel

Raymond said:
Peter Norvig's creative thinking triggered renewed interest in PEP 289.
That led to a number of contributors helping to re-work the pep details
into a form that has been well received on the python-dev list:

http://www.python.org/peps/pep-0289.html

In brief, the PEP proposes a list comprehension style syntax for
creating fast, memory efficient generator expressions on the fly:

sum(x*x for x in roots)
min(d.temperature()*9/5 + 32 for d in days)
Set(word.lower() for word in text.split() if len(word) < 5)
dict( (k, somefunc(k)) for k in keylist )
dotproduct = sum(x*y for x,y in itertools.izip(xvec, yvec))
bestplayer, bestscore = max( (p.score, p.name) for p in players )

Although most people on python-dev and here seems to like this PEP I
think it generally decreases readability. The above constructs seem
heavy and difficult to parse and thus i am afraid that they interfere
with the perceived simplicity of Python's syntax.

Sometimes it seems that introducing new syntax happens much easier
than improving or adding stdlib-modules.
Each of the above runs without creating a full list in memory,
which saves allocation time, conserves resources, and exploits
cache locality.

I do like these properties, though :)

At least I hope that generator expressions will only be introduced
via a __future__ statement in Python 2.4 much like it happened with
'yield' and generators. Maybe the PEP should have an "implementation
plan" chapter mentioning such details?

cheers,

holger
 
V

Ville Vainio

Holger Krekel said:
At least I hope that generator expressions will only be introduced
via a __future__ statement in Python 2.4 much like it happened with

Why? Does it break backwards compatibility somehow?

As far as readability goes, genexps seem to be pretty much as
readable/unreadable as list comprehensions. To me, BTW, the idea looks
great. However, I'm not completely sure about allowing them to be
creatad w/o parens in function calls, seems a bit too magical for my
taste.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top