List comprehensions performance

[Neuruss]

I have a doubt regarding list comprehensions:
According to Mark Lutz in his book Learning Pyhon:

"...there is currently a substantial performance advantage to the
extra complexity in this case: based on tests run under Python 2.2,
map calls are roughly twice as fast as equivalent for loops, and list
comprehensions are usually very slightly faster than map. This speed
difference owes to the fact that map and list comprehensions run at C
language speed inside the interpreter, rather than stepping through
Python for loop code within the PVM."

but I also read in Python Performance Tips by Skip Montanaro that
Lists comprehensions are not generally faster than the for loop
version.

What I'd like to know is if using list comprehensions would give me a
performance advantage over traditional for loops or not.
I'm getting fond of list comprehensions, but I wonder if it's wise to
abuse of them...

 

[Skip Montanaro]

Neuruss> What I'd like to know is if using list comprehensions would
Neuruss> give me a performance advantage over traditional for loops or
Neuruss> not.

You can always run tests to see. <wink> There is some data dependency so it
makes sense to test map, listcomps and for loops using data that's typical
of the application you intend to use them in.

Skip

 

[Raymond Hettinger]

[Neuruss] What I'd like to know is if using list comprehensions would give me a

performance advantage over traditional for loops or not.

For Py2.4, list comprehensions are much faster than equivalent for-loops.

I'm getting fond of list comprehensions, but I wonder if it's wise to
abuse of them...

Use whatever is clearest.
Don't abuse anything.



Raymond Hettinger

 

[Alex Martelli]

What I'd like to know is if using list comprehensions would give me a
performance advantage over traditional for loops or not.

Probably, a little, depending on the exact Python release you're using,
and on exactly what you're doing -- as others have suggested, timeit.py
can help. Don't expect _dramatic_ differences.

I'm getting fond of list comprehensions, but I wonder if it's wise to
abuse of them...

No, by definition of "abuse". Use list comprehensions to make lists,
not instead of perfectly normal loops. Consider for example:

kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' 'for x in
xrange(999): f()'
1000 loops, best of 3: 1.25e+03 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' 'for x in
xrange(999): f()'
1000 loops, best of 3: 1.29e+03 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' '[f() for x in
xrange(999)]'
1000 loops, best of 3: 1.45e+03 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' '[f() for x in
xrange(999)]'
1000 loops, best of 3: 1.44e+03 usec per loop

So, in this case, abusing list comprehensions slows you down by over
10%.

More generally, _who cares_ about this kind of speedups?! Premature
optimization is the root of all evil in programming. Make your programs
clear, simple, readable -- that's ALWAYS important! So is using general
algorithms and data structures with good O() characteristics if the
input size is liable to grow a lot. But squeezing out 10% or 20% here
or there is going to matter in a TINY minority of cases. Don't distort
your coding for such purposes!


Alex

 

[Alex Martelli]

[Neuruss] What I'd like to know is if using list comprehensions would give
me a > performance advantage over traditional for loops or not.

For Py2.4, list comprehensions are much faster than equivalent for-loops.

....but if the for loop is NOT equivalent (it doesn't accumulate results
into a resulting list), it's still faster. As I posted:

kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' 'for x in
xrange(999): f()'
1000 loops, best of 3: 1.29e+03 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' '[f() for x in
xrange(999)]'
1000 loops, best of 3: 1.45e+03 usec per loop

the LC is paying for the building of a list of 999 references to None,
which the for loop easily avoids, so the for loop is much faster here.

Of course, it WAS way more pronounced in 2.3:

kallisti:~/cb alex$ python2.3 timeit.py -s'def f()ass' 'for x in
xrange(999): f()'
1000 loops, best of 3: 1.76e+03 usec per loop
kallisti:~/cb alex$ python2.3 timeit.py -s'def f()ass' '[f() for x in
xrange(999)]'
100 loops, best of 3: 2.43e+03 usec per loop

Use whatever is clearest.
Don't abuse anything.

Absolutely. And: use 2.4 -- note that the SLOWER solution in 2.4 still
gains a good 20% over the FASTER one in 2.3... anybody who cares about
performance should be giving 2.4 an intense workout already, rarin' for
the day in which it will be officially released so it can be sensibly
used in customer-delivered software... (and, for those not in the know,
I should point out that Raymond was the one most responsible for the
overall impressive speed-up 2.2->2.3->2.4...!!!-).


Alex

 

[Remco Boerma]

Yup, use the standard timeit module for testing.
Made for these purposes. .
http://epydoc.sourceforge.net/stdlib/public/timeit-module.html

Cheers!
Remco

 

[Neuruss]

Thank you guys!
I will investigate the timeit module as suggested for these kind of tests...

 

[Bengt Richter]

[Neuruss] What I'd like to know is if using list comprehensions would give
me a > performance advantage over traditional for loops or not.

For Py2.4, list comprehensions are much faster than equivalent for-loops.

...but if the for loop is NOT equivalent (it doesn't accumulate results
into a resulting list), it's still faster. As I posted:

kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' 'for x in
xrange(999): f()'
1000 loops, best of 3: 1.29e+03 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' '[f() for x in
xrange(999)]'
1000 loops, best of 3: 1.45e+03 usec per loop

the LC is paying for the building of a list of 999 references to None,
which the for loop easily avoids, so the for loop is much faster here.

What if you abuse the LC so it makes an empty list? E.g.,
[None for x in xrange(999) if f() and False]

Not that I'm trying to promote LC abuse ;-)

Regards,
Bengt Richter

 

[Alex Martelli]

...but if the for loop is NOT equivalent (it doesn't accumulate results
into a resulting list), it's still faster. As I posted:

kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' 'for x in
xrange(999): f()'
1000 loops, best of 3: 1.29e+03 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' '[f() for x in
xrange(999)]'
1000 loops, best of 3: 1.45e+03 usec per loop

the LC is paying for the building of a list of 999 references to None,
which the for loop easily avoids, so the for loop is much faster here.

What if you abuse the LC so it makes an empty list? E.g.,
[None for x in xrange(999) if f() and False]

Now tell me, is timing so HARD...?! This obfuscation times (on 2.4 and
the same old iBook as above) to 1.38e+03 usec per loop, still slower
than the plain old loop -- the if/and rigmarole costs...


Alex

 

[Bengt Richter]

...but if the for loop is NOT equivalent (it doesn't accumulate results
into a resulting list), it's still faster. As I posted:

kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' 'for x in
xrange(999): f()'
1000 loops, best of 3: 1.29e+03 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'def f()ass' '[f() for x in
xrange(999)]'
1000 loops, best of 3: 1.45e+03 usec per loop

the LC is paying for the building of a list of 999 references to None,
which the for loop easily avoids, so the for loop is much faster here.

What if you abuse the LC so it makes an empty list? E.g.,
[None for x in xrange(999) if f() and False]

Now tell me, is timing so HARD...?! This obfuscation times (on 2.4 and
the same old iBook as above) to 1.38e+03 usec per loop, still slower
than the plain old loop -- the if/and rigmarole costs...

Sure, I was just wondering what would happen if you traded that for list growth overhead.
Anyway, thanks. It _is_ hard to compare with someone else's previous timing numbers ;-)
Maybe timeit.py should have an option for bogomips normalization or returning
time*bogomips. My system needs an upgrade ;-)

Regards,
Bengt Richter