removing list comprehensions in Python 3.0

[Steven Bethard]

>
> Sure, but the same holds for list comprehensions: list(i*i for i in
> xrange(10)). The difference is historic I guess; list comprehensions
> preceded generator expressions and so they cannot be removed, at least
> not before 3.0. I wonder if they will/should be in the language when
> the constraint of backwards compatibility is lifted. IMO they should
> not (TIOOWTDI, uniformity among builtin data structures, not
> overwhelmingly more useful than set or dict comprehensions), but
> there's a long way till that day.

I think the jury's still out on this one:

* Alex Martelli expects list comprehensions to be removed. [1]
* Robert Kern wants list comprehensions removed. [2]
* Raymond Hettinger encourages continued use of list comprehensions [3]
* Jeremy Bowers thinks list comprehensions should stay. [4]

I only searched a few relatively recent threads in c.l.py, so there are
probably more, but it looks to me like the final decision will have to
be made by a pronouncement from Guido.

[1]http://groups-beta.google.com/group/comp.lang.python/msg/f5613c00cb8c9539
[2]http://groups-beta.google.com/group/comp.lang.python/msg/b2cf0cd72d53fbe5
[3]http://groups-beta.google.com/group/comp.lang.python/msg/781dfab03701dd18
[4]http://groups-beta.google.com/group/comp.lang.python/msg/771a47d9eb24c863

 

[Kay Schluehr]

I think the jury's still out on this one:

* Alex Martelli expects list comprehensions to be removed. [1]
* Robert Kern wants list comprehensions removed. [2]
* Raymond Hettinger encourages continued use of list comprehensions [3]
* Jeremy Bowers thinks list comprehensions should stay. [4]

I only searched a few relatively recent threads in c.l.py, so there are
probably more, but it looks to me like the final decision will have to
be made by a pronouncement from Guido.

Well, I want to offer a more radical proposal: why not free squared
braces from the burden of representing lists at all? It should be
sufficient to write
list()

After being free one can use them for other purposes e.g. replacing the
ugly @ decorator character by the lovely [ .. ] notation or other
important features no one never trusted to implement waiting for the
right syntax sugar. More than this round braces together with lists can
be considered as a concession to the LISP programmer who was repelled
from Python by the decision to eliminate functional programming
features.

Kay

 

[Leif K-Brooks]

Well, I want to offer a more radical proposal: why not free squared
braces from the burden of representing lists at all? It should be
sufficient to write


list()

So then what would the expression list('foo') mean? Would it be
equivalent to ['foo'] (if so, how would you convert a string or other
iterable to a list under Py3k?), or would it be equivalent to ['f', 'o',
'o'] as it is in now (and is so, what gives?)?

 

[Devan L]

List comprehensions are faster than generator comprehensions for
iterating over smaller sequences.

 

[Steven Bethard]

Well, I want to offer a more radical proposal: why not free squared
braces from the burden of representing lists at all? It should be
sufficient to write
list()

After being free one can use them for other purposes e.g. replacing the
ugly @ decorator character by the lovely [ .. ] notation or other
important features no one never trusted to implement waiting for the
right syntax sugar. More than this round braces together with lists can
be considered as a concession to the LISP programmer who was repelled
from Python by the decision to eliminate functional programming
features.

Heh heh.

So how do I write a list literal then? Or do we only get tuple literals
in Python 3.0.

STeVe

 

[Steven Bethard]

List comprehensions are faster than generator comprehensions for
iterating over smaller sequences.

Could you give me an example? For the simple one below, the generator
expression was faster:

$ python -m timeit "for x in (i for i in xrange(10)): y = x"
100000 loops, best of 3: 4.75 usec per loop

$ python -m timeit "for x in [i for i in xrange(10)]: y = x"
100000 loops, best of 3: 5.33 usec per loop


And for another one, the results are basically indistinguishable:

$ python -m timeit "for x in (i for i in 'abcdefg'): y = x"
100000 loops, best of 3: 3.82 usec per loop

$ python -m timeit "for x in [i for i in 'abcdefg']: y = x"
100000 loops, best of 3: 3.87 usec per loop


I vaguely remember that in Python 2.4 conversion to tuples can take
longer because of the tuple extension code:

$ python -m timeit "tuple(i for i in xrange(10000))"
100 loops, best of 3: 2.24 msec per loop

$ python -m timeit "tuple([i for i in xrange(10000)])"
100 loops, best of 3: 1.85 msec per loop

But IIRC, this is supposed to be fixed in Python 2.5.


I found that for some longer sequences, generator expressions were
notably faster:

$ python -m timeit "set(i for i in xrange(10000))"
100 loops, best of 3: 3.77 msec per loop

$ python -m timeit "set([i for i in xrange(10000)])"
100 loops, best of 3: 4.54 msec per loop


Timings that validate your statement would be appreciated.

STeVe

 

[Dennis Lee Bieber]

I only searched a few relatively recent threads in c.l.py, so there are
probably more, but it looks to me like the final decision will have to
be made by a pronouncement from Guido.

Great... It takes me two releases of Python to get comfortable
with them, and then they are threatened to be removed again...

Might as well submit the language to ISO for standardization --
then I wouldn't be following an erratic target <G>

--

 

[Kay Schluehr]

Well, I want to offer a more radical proposal: why not free squared
braces from the burden of representing lists at all? It should be
sufficient to write


list()

So then what would the expression list('foo') mean? Would it be
equivalent to ['foo'] (if so, how would you convert a string or other
iterable to a list under Py3k?), or would it be equivalent to ['f', 'o',
'o'] as it is in now (and is so, what gives?)?

Spiltting a string and putting the characters into a list could be done
in method application style:
list('a','b','c')

Or equivalent from lists point of view:
list("a", "b", "c" )

Kay

 

[Steven Bethard]

Great... It takes me two releases of Python to get comfortable
with them, and then they are threatened to be removed again...

Might as well submit the language to ISO for standardization --
then I wouldn't be following an erratic target <G>

Two points:

(1) There's no reason to get uncomfortable even if they're removed.
You'd just replace [] with list().

(2) *IMPORTANT* If this happens *at all*, it won't happen until Python
3.0, which is probably at least 5 years away. And the Python 2.X branch
will still be available then, so if you don't like Python 3.0, you don't
have to use it.

STeVe

 

[Ron Adam]

Well, I want to offer a more radical proposal: why not free squared
braces from the burden of representing lists at all? It should be
sufficient to write


list()

list()

So then what would the expression list('foo') mean? Would it be
equivalent to ['foo'] (if so, how would you convert a string or other
iterable to a list under Py3k?), or would it be equivalent to ['f', 'o',
'o'] as it is in now (and is so, what gives?)?

Spiltting a string and putting the characters into a list could be done
in method application style:


list('a','b','c')

"abc".splitchrs()

There's already a str.split() to create a list of words,
and a str.splitline() to get a list of lines, so it would group related
methods together.

I don't thin adding sting methods to lists is a good idea.

Cheers,
Ron

 

[Leif K-Brooks]

list("a", "b", "c" )

I assume we'll also have list.from_list, list.from_tuple,
list.from_genexp, list.from_xrange, etc.?

 

[Devan L]

import timeit

t1 = timeit.Timer('list(i for i in xrange(10))')
t1.timeit() 27.267753024476576
t2 = timeit.Timer('[i for i in xrange(10)]')
t2.timeit() 15.050426800054197
t3 = timeit.Timer('list(i for i in xrange(100))')
t3.timeit() 117.61078097914682
t4 = timeit.Timer('[i for i in xrange(100)]')
t4.timeit()

83.502424470149151

Hrm, okay, so generators are generally faster for iteration, but not
for making lists(for small sequences), so list comprehensions stay.

 

[Bengt Richter]

Great... It takes me two releases of Python to get comfortable
with them, and then they are threatened to be removed again...

Might as well submit the language to ISO for standardization --
then I wouldn't be following an erratic target <G>

Two points:

(1) There's no reason to get uncomfortable even if they're removed.
You'd just replace [] with list().

So list(1, 2, 3) will be the same as [1, 2, 3] ??

Right now, Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: list() takes at most 1 argument (3 given)

have fun ;-)

(2) *IMPORTANT* If this happens *at all*, it won't happen until Python
3.0, which is probably at least 5 years away. And the Python 2.X branch
will still be available then, so if you don't like Python 3.0, you don't
have to use it.

STeVe

Regards,
Bengt Richter

 

[Ron Adam]

I assume we'll also have list.from_list, list.from_tuple,
list.from_genexp, list.from_xrange, etc.?

List from list isn't needed, nor list from tuple. That's what the * is
for. And for that matter neither is the str.splitchar() either.


class mylist(list):
def __init__(self,*args):
self[:]=args[:]


mylist(*[1,2,3]) -> [1,2,3]

mylist(*(1,2,3)) -> [1,2,3]

mylist(*"abc") -> ['a','b','c']

mylist(1,2,3) -> [1,2,3]

mylist([1],[2]) -> [[1],[2]]

mylist('hello','world') -> ['hello','world']



Works for me. ;-)

I always thought list([1,2,3]) -> [1,2,3] was kind of silly.


Cheers,
Ron

 

[Kay Schluehr]

I assume we'll also have list.from_list, list.from_tuple,
list.from_genexp, list.from_xrange, etc.?

One might unify all those factory functions into a single
list.from_iter that dispatches to the right constructor that still
lives under the hood. More conceptually: there is some abstract iter
base class providing a from_iter method which may be overwritten in
concrete subclasses like tuple, str or list.

I would further suggest a lazy iterator used to evaluate objects when
they get accessed the first time:

l = lazy( math.exp(100) , 27 )

l[1]

27

The first element won't ever be evaluated if it does not get accessed
explicitely. This is some very special kind of partial
evaluation/specialization.

Kay

 

[Peter Hansen]

(1) There's no reason to get uncomfortable even if they're removed.
You'd just replace [] with list().

So list(1, 2, 3) will be the same as [1, 2, 3] ??

No, the discussion is about list comprehensions. [1,2,3] is not a list
comprehension, as you know.

-Peter

 

[Raymond Hettinger]

In all probability, both list comprehensions and generator expressions
will be around in perpetuity. List comps have been a very successful
language feature.

The root of this discussion has been the observation that a list
comprehension can be expressed in terms of list() and a generator
expression. However, the former is faster when you actually want a
list result and many people (including Guido) like the square brackets.

After the advent of generators, it seemed for a while that all
functions and methods that returned lists would eventually return
iterators instead. What we are learning is that there is a place for
both. It is darned inconvenient to get an iterator when you really
need a list, when you want to slice the result, when you want to see a
few elements through repr(), and when you need to loop over the
contents more than once.


Raymond Hettinger

 

[Bengt Richter]

(1) There's no reason to get uncomfortable even if they're removed.
You'd just replace [] with list().

So list(1, 2, 3) will be the same as [1, 2, 3] ??

No, the discussion is about list comprehensions. [1,2,3] is not a list
comprehension, as you know.

D'oh. Sorry to have come in from contextual outer space ;-/

Regards,
Bengt Richter

 

[John Roth]

In all probability, both list comprehensions and generator expressions
will be around in perpetuity. List comps have been a very successful
language feature.

The root of this discussion has been the observation that a list
comprehension can be expressed in terms of list() and a generator
expression. However, the former is faster when you actually want a
list result and many people (including Guido) like the square brackets.

After the advent of generators, it seemed for a while that all
functions and methods that returned lists would eventually return
iterators instead. What we are learning is that there is a place for
both. It is darned inconvenient to get an iterator when you really
need a list, when you want to slice the result, when you want to see a
few elements through repr(), and when you need to loop over the
contents more than once.

I was wondering about what seemed like an ill-concieved rush to
make everything an iterator. Iterators are, of course, useful but there
are times when you really did want a list.

John Roth

 

[George Sakkis]

In all probability, both list comprehensions and generator expressions
will be around in perpetuity. List comps have been a very successful
language feature.

The root of this discussion has been the observation that a list
comprehension can be expressed in terms of list() and a generator
expression.

No, the root of the discussion, in this thread at least, was the answer
to "why not dict comprehensions ?", which was along the lines of "well,
you can do it in one line by dict(gen_expression)".

However, the former is faster when you actually want a
list result and many people (including Guido) like the square brackets.

Also dict comprehensions are faster if you actually want a dict result,
set comprehensions for set result, and so on and so forth.

After the advent of generators, it seemed for a while that all
functions and methods that returned lists would eventually return
iterators instead. What we are learning is that there is a place for
both.

Altering the result type of existing functions and methods is not
exactly the same with the discussion on the future of list
comprehensions; the latter affects only whether listcomps are special
enough to be granted special syntax support, when there is an
equivalent way to express the same thing. It's funny how one of the
arguments for removing lambda -- you can do the same by defining a
named function -- does not apply for list comprehensions.

It is darned inconvenient to get an iterator when you really
need a list, when you want to slice the result, when you want to see a
few elements through repr(), and when you need to loop over the
contents more than once.

Raymond Hettinger

Similar arguments can be given for dict comprehensions as well.

George