python performance on Solaris

Discussion in 'Python' started by inaf, Oct 10, 2009.

  1. inaf

    inaf Guest

    I have been following this group for quite some time and I figured
    (after searching enough on google --and on this group-- and not
    finding anything useful) I could pose this question here. Can anyone
    shed some light on python's performance on Solaris? My code seem to
    return lookups from a in memory data structure I build combining bunch
    of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
    on a Solaris zone. Is there anything that needs to be done
    specifically when installing/compiling python to improve performance
    or is it a known thing that python does not perform that well on
    solaris?

    Thanks..
     
    inaf, Oct 10, 2009
    #1
    1. Advertising

  2. inaf <cem.ezberci <at> gmail.com> writes:
    >
    > My code seem to
    > return lookups from a in memory data structure I build combining bunch
    > of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
    > on a Solaris zone.


    Well, if your workload is CPU-bound, the issue here is not really Solaris vs.
    Linux but rather CPU power. You should try to run a generic (non-Python) CPU
    benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only
    Python shows such a performance difference, on the other hand, perhaps you can
    give us more precisions on those systems.

    Regards

    Antoine.


    (*) for example one of the C programs on http://shootout.alioth.debian.org/u64/c.php
     
    Antoine Pitrou, Oct 11, 2009
    #2
    1. Advertising

  3. inaf

    inaf Guest

    On Oct 11, 6:59 am, Antoine Pitrou <> wrote:
    > inaf <cem.ezberci <at> gmail.com> writes:
    >
    >
    >
    > > My code seem to
    > > return lookups from a in memory data structure I build combining bunch
    > > of dictionaries and lists 6-8 times faster on a 32 bit Linux box than
    > > on a Solaris zone.

    >
    > Well, if your workload is CPU-bound, the issue here is not really Solaris vs.
    > Linux but rather CPU power. You should try to run a generic (non-Python) CPU
    > benchmark (*) on both systems, perhaps this 6-8 factor is expected. If only
    > Python shows such a performance difference, on the other hand, perhaps you can
    > give us more precisions on those systems.
    >
    > Regards
    >
    > Antoine.
    >
    > (*) for example one of the C programs onhttp://shootout.alioth.debian.org/u64/c.php


    Good point. I failed to compare the CPU power on these machines.. 32
    bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
    guess that explains :) Thank you for the tip..
     
    inaf, Oct 12, 2009
    #3
  4. inaf <cem.ezberci <at> gmail.com> writes:
    >
    > Good point. I failed to compare the CPU power on these machines.. 32
    > bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
    > guess that explains :) Thank you for the tip..


    You have to compare not only CPU frequencies but the CPU models.
    Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
    "UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
    single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
    difference in single-threaded performance compared to a modern Intel or AMD CPU
    is totally expected.

    Regards

    Antoine.
     
    Antoine Pitrou, Oct 14, 2009
    #4
  5. inaf

    Jorgen Grahn Guest

    On Wed, 2009-10-14, Antoine Pitrou wrote:
    > inaf <cem.ezberci <at> gmail.com> writes:
    >>
    >> Good point. I failed to compare the CPU power on these machines.. 32
    >> bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
    >> guess that explains :) Thank you for the tip..

    >
    > You have to compare not only CPU frequencies but the CPU models.


    Yes, at least that. Megahertz figures have been useless for decades,
    except in advertising.

    > Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
    > "UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
    > single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
    > difference in single-threaded performance compared to a modern Intel
    > or AMD CPU
    > is totally expected.


    (Had to Google it. A "Solaris Zone" is apparently some kind of
    virtualization thing, with low CPU overhead.)

    s/multi-threading/multi-programming/ I suppose. I certainly hope you
    can still get performance while running many separate true processes in
    parallel.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
     
    Jorgen Grahn, Oct 14, 2009
    #5
  6. inaf

    inaf Guest

    On Oct 14, 7:15 am, Antoine Pitrou <> wrote:
    > inaf <cem.ezberci <at> gmail.com> writes:
    >
    >
    >
    > > Good point. I failed to compare the CPU power on these machines.. 32
    > > bit linux box I have is 2666 Mhz vs the Solaris zone is 1415 Mhz.. I
    > > guess that explains :) Thank you for the tip..

    >
    > You have to compare not only CPU frequencies but the CPU models.
    > Recently Sun has been selling CPUs optimized for multi-threading (e.g. the
    > "UltraSPARC T2" or Niagara CPUs) which have, by design, very poor
    > single-threaded performance. If your Solaris zone uses such a CPU then a 6-8x
    > difference in single-threaded performance compared to a modern Intel or AMD CPU
    > is totally expected.
    >
    > Regards
    >
    > Antoine.


    Antonie -- yes, you are right. Even the architecture of the two types
    make a difference. I was under the impression that RISC based CPUs did
    not need to have a very high clock speed and that they can perform
    similarly compared to an x86 processor with higher clock speed. That
    is why I was a bit surprised. I guess there could be other factors at
    play. That's why I was asking if there are specific things to be done
    while compiling Python on Solaris. I found some tips online which led
    me to compile it with a different threading lib resulting in slightly
    better performance after my original post.

    In terms of the processors I have, please see below for details:

    Status of virtual processor 40 as of: 10/14/2009 17:13:51
    on-line since 07/23/2009 18:48:21.
    The sparcv9 processor operates at 1415 MHz,
    and has a sparcv9 floating point processor.

    So I guess this is not one of those you are talking about..

    Thanks..
     
    inaf, Oct 14, 2009
    #6
  7. inaf

    John Nagle Guest

    inaf wrote:
    > I have been following this group for quite some time and I figured
    > (after searching enough on google --and on this group-- and not
    > finding anything useful) I could pose this question here. Can anyone
    > shed some light on python's performance on Solaris?


    Note that multithreaded compute-bound Python programs really suck
    on multiprocessors. Adding a second CPU makes the program go slower,
    due to a lame mechanism for resolving conflicts over the global interpreter
    lock.

    John Nagle
     
    John Nagle, Oct 15, 2009
    #7
  8. Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écrit :
    >
    > Note that multithreaded compute-bound Python programs really suck
    > on multiprocessors. Adding a second CPU makes the program go slower,
    > due to a lame mechanism for resolving conflicts over the global
    > interpreter lock.


    I'm not sure what you're talking about. Python has no "mechanism for
    resolving conflicts over the global interpreter lock" (let alone a lame
    one :)), it just trusts the OS to schedule a thread only when it is not
    waiting on an unavailable resource (a lock). The GIL is just an OS-level
    synchronization primitive and its behaviour (fairness, performance) will
    depend on the behaviour of the underlying OS.

    If this belief is derived from Dave Beazley's slides (*) about Python's
    multi-threading issues, I'd say some of his observations are rather
    questionable. First because the measurements were done on OS X, which has
    its own serious concurrency problems (**). Second because one of the
    benchmarks is so synthetic that it doesn't reflect real-world use of
    Python at all.

    This is not to say there aren't any issues, of course.


    (*) http://www.dabeaz.com/python/GIL.pdf

    (**) http://www.nabble.com/Mutex-performance-td24892454.html
     
    Antoine Pitrou, Oct 15, 2009
    #8
  9. inaf

    Aahz Guest

    In article <>,
    inaf <> wrote:
    >
    >I have been following this group for quite some time and I figured
    >(after searching enough on google --and on this group-- and not finding
    >anything useful) I could pose this question here. Can anyone shed
    >some light on python's performance on Solaris? My code seem to return
    >lookups from a in memory data structure I build combining bunch of
    >dictionaries and lists 6-8 times faster on a 32 bit Linux box than on a
    >Solaris zone.


    Also, assuming that a "zone" is some kind of virtual machine, it will
    definitely impose its own overhead.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "To me vi is Zen. To use vi is to practice zen. Every command is a
    koan. Profound to the user, unintelligible to the uninitiated. You
    discover truth everytime you use it."
     
    Aahz, Oct 15, 2009
    #9
  10. Antoine Pitrou <> writes on Thu, 15 Oct 2009 16:25:43 +0000 (UTC):
    > Le Wed, 14 Oct 2009 22:39:14 -0700, John Nagle a écrit :
    > >
    > > Note that multithreaded compute-bound Python programs really suck
    > > on multiprocessors. Adding a second CPU makes the program go slower,
    > > due to a lame mechanism for resolving conflicts over the global
    > > interpreter lock.

    >
    > I'm not sure what you're talking about. Python has no "mechanism for
    > resolving conflicts over the global interpreter lock" (let alone a lame
    > one :)), it just trusts the OS to schedule a thread only when it is not
    > waiting on an unavailable resource (a lock). The GIL is just an OS-level
    > synchronization primitive and its behaviour (fairness, performance) will
    > depend on the behaviour of the underlying OS.


    But, independent from the OS and the fairness/performance of the GIL
    management itself: the GIL is there to prevent concurrent execution
    of Python code. Thus, at any time, at most one thread (in a process)
    is executing Python code -- other threads may run as well, as long
    as they are inside non Python code but cannot be executing Python bytecode,
    independent of available CPU resources. This implies that Python cannot
    fully exploit the power of multiprocessors.

    It is also true that adding CPUs may in fact reduce performance for
    compute bound multithreaded Python programs. While the additional
    computational resources cannot be use by Python, the additional overhead
    (switching between CPUs) may reduce overall performance.
    I agree with you that it is difficult to understand when this overhead
    were really significant.

    Dieter
     
    Dieter Maurer, Oct 17, 2009
    #10
  11. Re: multi-threaded performance

    Le Sat, 17 Oct 2009 06:54:29 +0200, Dieter Maurer a écrit :
    >
    > It is also true that adding CPUs may in fact reduce performance for
    > compute bound multithreaded Python programs. While the additional
    > computational resources cannot be use by Python, the additional overhead
    > (switching between CPUs) may reduce overall performance. I agree with
    > you that it is difficult to understand when this overhead were really
    > significant.


    For what it's worth, I just wrote a little benchmark script to measure
    this kind of things:
    http://svn.python.org/view/sandbox/trunk/ccbench/

    Regards

    Antoine.
     
    Antoine Pitrou, Oct 17, 2009
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. jm
    Replies:
    1
    Views:
    540
    alien2_51
    Dec 12, 2003
  2. thomhashi
    Replies:
    2
    Views:
    841
    Nils O. =?iso-8859-1?Q?Sel=E5sdal?=
    Oct 31, 2003
  3. bugbear
    Replies:
    4
    Views:
    2,930
    Arne Vajhøj
    Mar 28, 2008
  4. Colin Mackenzie

    Ruby on Solaris 10 performance problems

    Colin Mackenzie, Jan 28, 2009, in forum: Ruby
    Replies:
    15
    Views:
    281
    Charles Oliver Nutter
    Jan 30, 2009
  5. Software Engineer
    Replies:
    0
    Views:
    368
    Software Engineer
    Jun 10, 2011
Loading...

Share This Page