python performance on Solaris

[inaf]

I have been following this group for quite some time and I figured
(after searching enough on google --and on this group-- and not
finding anything useful) I could pose this question here. Can anyone
shed some light on python's performance on Solaris? My code seem to
return lookups from a in memory data structure I build combining bunch
of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
on a Solaris zone. Is there anything that needs to be done
specifically when installing/compiling python to improve performance
or is it a known thing that python does not perform that well on
solaris?

Thanks..

 

[Antoine Pitrou]

My code seem to
return lookups from a in memory data structure I build combining bunch
of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
on a Solaris zone.

Well, if your workload is CPU-bound, the issue here is not really Solaris vs.
Linux but rather CPU power. You should try to run a generic (non-Python) CPU
benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only
Python shows such a performance difference, on the other hand, perhaps you can
give us more precisions on those systems.

Regards

Antoine.


(*) for example one of the C programs on http://shootout.alioth.debian.org/u64/c.php

 

[inaf]

inaf <cem.ezberci <at> gmail.com> writes:




Well, if your workload is CPU-bound, the issue here is not really Solaris vs.
Linux but rather CPU power. You should try to run a generic (non-Python) CPU
benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only
Python shows such a performance difference, on the other hand, perhaps you can
give us more precisions on those systems.

Regards

Antoine.

(*) for example one of the C programs onhttp://shootout.alioth.debian.org/u64/c.php

Good point. I failed to compare the CPU power on these machines.. 32
bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
guess that explains Thank you for the tip..

 

[Antoine Pitrou]

Good point. I failed to compare the CPU power on these machines.. 32
bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
guess that explains Thank you for the tip..

You have to compare not only CPU frequencies but the CPU models.
Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
"UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
difference in single-threaded performance compared to a modern Intel or AMD CPU
is totally expected.

Regards

Antoine.

 

[Jorgen Grahn]

You have to compare not only CPU frequencies but the CPU models.

Yes, at least that. Megahertz figures have been useless for decades,
except in advertising.

Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
"UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
difference in single-threaded performance compared to a modern Intel
or AMD CPU
is totally expected.

(Had to Google it. A "Solaris Zone" is apparently some kind of
virtualization thing, with low CPU overhead.)

s/multi-threading/multi-programming/ I suppose. I certainly hope you
can still get performance while running many separate true processes in
parallel.

/Jorgen

 

[inaf]

inaf <cem.ezberci <at> gmail.com> writes:




You have to compare not only CPU frequencies but the CPU models.
Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
"UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
difference in single-threaded performance compared to a modern Intel or AMD CPU
is totally expected.

Regards

Antoine.

Antonie -- yes, you are right. Even the architecture of the two types
make a difference. I was under the impression that RISC based CPUs did
not need to have a very high clock speed and that they can perform
similarly compared to an x86 processor with higher clock speed. That
is why I was a bit surprised. I guess there could be other factors at
play. That's why I was asking if there are specific things to be done
while compiling Python on Solaris. I found some tips online which led
me to compile it with a different threading lib resulting in slightly
better performance after my original post.

In terms of the processors I have, please see below for details:

Status of virtual processor 40 as of: 10/14/2009 17:13:51
on-line since 07/23/2009 18:48:21.
The sparcv9 processor operates at 1415 MHz,
and has a sparcv9 floating point processor.

So I guess this is not one of those you are talking about..

Thanks..

 

[John Nagle]

I have been following this group for quite some time and I figured
(after searching enough on google --and on this group-- and not
finding anything useful) I could pose this question here. Can anyone
shed some light on python's performance on Solaris?

Note that multithreaded compute-bound Python programs really suck
on multiprocessors. Adding a second CPU makes the program go slower,
due to a lame mechanism for resolving conflicts over the global interpreter
lock.

John Nagle

 

[Antoine Pitrou]

Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écrit :

Note that multithreaded compute-bound Python programs really suck
on multiprocessors. Adding a second CPU makes the program go slower,
due to a lame mechanism for resolving conflicts over the global
interpreter lock.

I'm not sure what you're talking about. Python has no "mechanism for
resolving conflicts over the global interpreter lock" (let alone a lame
one ), it just trusts the OS to schedule a thread only when it is not
waiting on an unavailable resource (a lock). The GIL is just an OS-level
synchronization primitive and its behaviour (fairness, performance) will
depend on the behaviour of the underlying OS.

If this belief is derived from Dave Beazley's slides (*) about Python's
multi-threading issues, I'd say some of his observations are rather
questionable. First because the measurements were done on OS X, which has
its own serious concurrency problems (**). Second because one of the
benchmarks is so synthetic that it doesn't reflect real-world use of
Python at all.

This is not to say there aren't any issues, of course.


(*) http://www.dabeaz.com/python/GIL.pdf

(**) http://www.nabble.com/Mutex-performance-td24892454.html

 

[Aahz]

I have been following this group for quite some time and I figured
(after searching enough on google --and on this group-- and not finding
anything useful) I could pose this question here. Can anyone shed
some light on python's performance on Solaris? My code seem to return
lookups from a in memory data structure I build combining bunch of
dictionaries and lists 6-8 times faster on a 32 bit Linux box than on a
Solaris zone.

Also, assuming that a "zone" is some kind of virtual machine, it will
definitely impose its own overhead.
--
Aahz ([email protected]) <*> http://www.pythoncraft.com/

"To me vi is Zen. To use vi is to practice zen. Every command is a
koan. Profound to the user, unintelligible to the uninitiated. You
discover truth everytime you use it." (e-mail address removed)

 

[Dieter Maurer]

Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écrit :

I'm not sure what you're talking about. Python has no "mechanism for
resolving conflicts over the global interpreter lock" (let alone a lame
one ), it just trusts the OS to schedule a thread only when it is not
waiting on an unavailable resource (a lock). The GIL is just an OS-level
synchronization primitive and its behaviour (fairness, performance) will
depend on the behaviour of the underlying OS.

But, independent from the OS and the fairness/performance of the GIL
management itself: the GIL is there to prevent concurrent execution
of Python code. Thus, at any time, at most one thread (in a process)
is executing Python code -- other threads may run as well, as long
as they are inside non Python code but cannot be executing Python bytecode,
independent of available CPU resources. This implies that Python cannot
fully exploit the power of multiprocessors.

It is also true that adding CPUs may in fact reduce performance for
compute bound multithreaded Python programs. While the additional
computational resources cannot be use by Python, the additional overhead
(switching between CPUs) may reduce overall performance.
I agree with you that it is difficult to understand when this overhead
were really significant.

Dieter

 

[Antoine Pitrou]

Le Sat, 17 Oct 2009 06:54:29 +0200, Dieter Maurer a écrit :

It is also true that adding CPUs may in fact reduce performance for
compute bound multithreaded Python programs. While the additional
computational resources cannot be use by Python, the additional overhead
(switching between CPUs) may reduce overall performance. I agree with
you that it is difficult to understand when this overhead were really
significant.

For what it's worth, I just wrote a little benchmark script to measure
this kind of things:
http://svn.python.org/view/sandbox/trunk/ccbench/

Regards

Antoine.