Performance (pystone) of python 2.4 lower then python 2.3 ???

[Lucas Hofman]

Hi,

Just installed Python 2.4 on a machine (RH8.0 Linux) that also has python 2.3
and python 2.2 installed. The latter came with the linux distribution, the other
are compiled from source tarballs.

Comparing them gives the following unexpected result:

[lucas@oslwb03 test]$ /usr/bin/python pystone.py
Pystone(1.1) time for 50000 passes = 1.86
This machine benchmarks at 26881.7 pystones/second
[lucas@oslwb03 test]$ /usr/local/bin/python2.3 pystone.py
Pystone(1.1) time for 50000 passes = 1.22
This machine benchmarks at 40983.6 pystones/second

This is ok, a 52% speed increase, but:

lucas@oslwb03 test]$ /usr/local/bin/python2.4 pystone.py
Pystone(1.1) time for 50000 passes = 1.31
This machine benchmarks at 38167.9 pystones/second

A 7% speed DECREASE??? According to the documentation it should be a 5% increase?

The machine is a 3.0 GHz Xeon box.

Both python 2.3 and 2.4 where configure without any options.

Anyone who understands what is going on?

Regards, Lucas

 

[Peter Hansen]

lucas@oslwb03 test]$ /usr/local/bin/python2.4 pystone.py
Pystone(1.1) time for 50000 passes = 1.31
This machine benchmarks at 38167.9 pystones/second

A 7% speed DECREASE??? According to the documentation it should be a 5% increase?

The machine is a 3.0 GHz Xeon box.

Both python 2.3 and 2.4 where configure without any options.

For comparison, I do get a decent speedup. Machine is an
AMD Athlon XP 2500+ (1.82GHz) running Win XP Pro SP2.

Python 2.3.4: 36393 pystones.
Python 2.4: 39400 pystones.

....about an 8% speedup.

 

[Istvan Albert]

Anyone who understands what is going on?

It is difficult to measure a speedup that might be
well within your measurement error.

Run the same pystone benchmark repeatedly and
see what variation you get.

Istvan.

 

[Nick Craig-Wood]

For comparison, I do get a decent speedup. Machine is an
AMD Athlon XP 2500+ (1.82GHz) running Win XP Pro SP2.

Python 2.3.4: 36393 pystones.
Python 2.4: 39400 pystones.

...about an 8% speedup.

On my 2.6 GHz P4 running debian testing I got the following results :-

$ for p in 2.1 2.2 2.3 2.4; do echo $p; python$p pystone.py 1000000 ; done

2.1
Pystone(1.1) time for 1000000 passes = 40.67
This machine benchmarks at 24588.1 pystones/second
2.2
Pystone(1.1) time for 1000000 passes = 39.64
This machine benchmarks at 25227 pystones/second
2.3
Pystone(1.1) time for 1000000 passes = 32.49
This machine benchmarks at 30778.7 pystones/second
2.4
Pystone(1.1) time for 1000000 passes = 29.88
This machine benchmarks at 33467.2 pystones/second

Showing that 2.4 is the fastest so far! (And is also a good advert
for AMD ;-)

 

[Lucas Hofman]

It is difficult to measure a speedup that might be
well within your measurement error.

Run the same pystone benchmark repeatedly and
see what variation you get.

Istvan.

Very little variation actually. The system measured is only lightly loaded (and
it is a 2 processor box). I ran the benchmark > 4 times and got 3 results that
are within 1% of each other.

Lucas

 

[Lucas Hofman]

On my 2.6 GHz P4 running debian testing I got the following results :-

$ for p in 2.1 2.2 2.3 2.4; do echo $p; python$p pystone.py 1000000 ; done

2.1
Pystone(1.1) time for 1000000 passes = 40.67
This machine benchmarks at 24588.1 pystones/second
2.2
Pystone(1.1) time for 1000000 passes = 39.64
This machine benchmarks at 25227 pystones/second
2.3
Pystone(1.1) time for 1000000 passes = 32.49
This machine benchmarks at 30778.7 pystones/second
2.4
Pystone(1.1) time for 1000000 passes = 29.88
This machine benchmarks at 33467.2 pystones/second

Showing that 2.4 is the fastest so far! (And is also a good advert
for AMD

I got this list on a single processor P4 1.6 Ghz:
2.1
Pystone(1.1) time for 100000 passes = 6.74
This machine benchmarks at 14836.8 pystones/second
2.2
Pystone(1.1) time for 100000 passes = 6.36
This machine benchmarks at 15723.3 pystones/second
2.3
Pystone(1.1) time for 100000 passes = 4.92
This machine benchmarks at 20325.2 pystones/second
2.4
Pystone(1.1) time for 100000 passes = 4.51
This machine benchmarks at 22172.9 pystones/second

Which shows the expected speedup.

On a dual Xeon 3.0 Ghz:
2.2
Pystone(1.1) time for 1000000 passes = 37.45
This machine benchmarks at 26702.3 pystones/second
2.3
Pystone(1.1) time for 1000000 passes = 25.28
This machine benchmarks at 39557 pystones/second
2.4
Pystone(1.1) time for 1000000 passes = 25.94
This machine benchmarks at 38550.5 pystones/second

Which shows a decrease in performance. Could this have anything to do with the
fact that is is a dual processor box?

Lucas

 

[Simon Wittber]

Which shows a decrease in performance. Could this have anything to do with the

fact that is is a dual processor box?

Doubtful. I'm running dual Pentium 4 2.8 Ghz (Win XP) and get the
following results:

C:\>python23\python Python23\lib\test\pystone.py
Pystone(1.1) time for 50000 passes = 1.43068
This machine benchmarks at 34948.3 pystones/second

C:\>python24\python Python24\lib\test\pystone.py
Pystone(1.1) time for 50000 passes = 1.28359
This machine benchmarks at 38953.2 pystones/second

Sw.

 

[Mark Asbach]

Hi Lucas,

On a dual Xeon 3.0 Ghz:
[...]

Which shows a decrease in performance. Could this have anything to do with the
fact that is is a dual processor box?

Maybe. But my 3Gh P4/HT is also detected as a dual processor machine
(Kernel 2.6), so it might be a general problem with the Xeon?

BTW: The discussion got me interested so I compiled Python 2.4 myself
with different compiler switches and different compilers, just to find
that generating machine dependant code doesn't speed up the pystone
benchmark in any way (with gcc 3.3.4) and using intel's icc V8 only
results in a small speedup (from approx. 40000 with gcc to approx. 41500
with icc).

So python performance doesn't seem to be that dependant on machine code
representation. Maybe you're running into a different problem like cache
size.

In addition: running python2.3 with psyco 1.2 proxying all 8 Procs of
pystone resulted in approx 155000 pystones, while python2.4 with psyco
1.3 resulted in approx 200000 pystones ...

Mark

 

[Raymond Hettinger]

"Lucas Hofman

This machine benchmarks at 38167.9 pystones/second

Pystone is an abyssmally bad benchmark for comparing the relative speeds of
different versions of python (it nets out all eval loop improvements and it
exercises only a microscopic portion of the language).

I would be interested in seeing other people's comparitive results for pybench,
test_decimal, and parrotbench.

To run test_decimal.py, you first have to copy Lib/test/decimal.py into Py2.3's
lib directory. On an old PentiumIII running WinMe, I get 48.940 sec in Py2.3
and 44.820 sec in Py2.4, a 8.4% improvement.

For pybench, I get 7418.90ms in Py2.3 and 6320.07 ms in Py2.4, a 14.8%
improvement.

For parrotbench, I get 54.517 seconds in Py2.3 and 45.009 seconds in Py2.4, a
17.4% improvement.

It is also interesting to time specific features not covered by the above
benchmarks. For example, list comprehensions got a nice 60% boost on my
machine:

C:\py24\Lib>\python23\python timeit.py -r9 "[i for i in xrange(1000)]"
100 loops, best of 9: 1.11 msec per loop

C:\py24\Lib>\python24\python timeit.py -r9 "[i for i in xrange(1000)]"
1000 loops, best of 9: 417 usec per loop


Raymond Hettinger

 

[Lucas Hofman]

Hi Lucas,

On a dual Xeon 3.0 Ghz:

[...]


Which shows a decrease in performance. Could this have anything to do with the
fact that is is a dual processor box?

Maybe. But my 3Gh P4/HT is also detected as a dual processor machine
(Kernel 2.6), so it might be a general problem with the Xeon?

Hi Mark,

No,the reason that you see 2 times as many processors as really are
installed is the hyperthreading feature of the Xeon (see
http://www.infoworld.com/infoworld/article/02/02/25/020225plxeon_1.html)

I turned it off (in the BIOS). The machine I tested on has 2 (pysical)
processors installed. Turning on or off does not influence the pystone
number significantly..

Regards, Lucas

 

[Dan]

A 7% speed DECREASE??? According to the documentation it should be a 5% increase?

I also see an 8-10% speed decrease in 2.4 (I built) from 2.3.3 (shipped
w/Fedora2) in the program I'm writing (best of 3 trials each). Memory
use seems to be about the same.

2.4: real 2m44.131s
2.3.3: real 2m29.865s

/Dan

 

[Skip Montanaro]

Dan> I also see an 8-10% speed decrease in 2.4 (I built) from 2.3.3
Dan> (shipped w/Fedora2) in the program I'm writing (best of 3 trials
Dan> each). Memory use seems to be about the same.

How do you how the compiler flags were the same if you didn't compile both
versions yourself?

Skip

 

[Mark Asbach]

Hi Lucas,

No,the reason that you see 2 times as many processors as really are
installed is the hyperthreading feature of the Xeon (see
http://www.infoworld.com/infoworld/article/02/02/25/020225plxeon_1.html)

I turned it off (in the BIOS). The machine I tested on has 2 (pysical)
processors installed. Turning on or off does not influence the pystone
number significantly..

Sorry for the confusion - that was what I wanted to say: from my
perspective it is unlikely that the performance decrease seen on your
dual processor CPU is related to having two processors. Because python
thinks it has two processors on my machine, too (because of
hyperthreading) and should exhibit the same decrease then (IF it would
be coupled to two processors).

Yours,

Mark

 

[Dan]

Dan> I also see an 8-10% speed decrease in 2.4 (I built) from 2.3.3
Dan> (shipped w/Fedora2) in the program I'm writing (best of 3 trials
Dan> each). Memory use seems to be about the same.

How do you how the compiler flags were the same if you didn't compile both
versions yourself?

I don't, and I also don't know if the Redhat folks modified the sources
in some way. That's why I mentioned the difference in builds. But I
suppose that makes this as good as no data.

/Dan