Embedding Python, threading and scalability

Discussion in 'Python' started by Wenning Qiu, Jul 8, 2003.

  1. Wenning Qiu

    Wenning Qiu Guest

    I am researching issues related to emdedding Python in C++ for a
    project.

    My project will be running on an SMP box and requires scalability.
    However, my test shows that Python threading has very poor performance
    in terms of scaling. In fact it doesn't scale at all.

    I wrote a simple test program to complete given number of iterations
    of a simple loop. The total number of iterations can be divided evenly
    among a number of threads. My test shows that as the number of threads
    grows, the CPU usage grows and the response time gets longer. For
    example, to complete the same amount of work, one thread takes 10
    seconds, 2 threads take 20 seconds and 3 threads take 30 seconds.

    The fundamental reason for lacking scalability is that Python uses a
    global interpreter lock for thread safety. That global lock must be
    held by a thread before it can safely access Python objects.

    I thought I might be able to make embedded Python scalable by
    embedding multiple interpreters and have them run independently in
    different threads. However "Python/C API Reference Manual" chapter 8
    says that "The global interpreter lock is also shared by all threads,
    regardless of to which interpreter they belong". Therefore with
    current implementation, even multiple interpreters do not provide
    scalability.

    Has anyone on this list run into the same problem that I have, or does
    anyone know of any plan of totally insulating multiple embedded Python
    interpreters?

    Thanks,
    Wenning Qiu
    Wenning Qiu, Jul 8, 2003
    #1
    1. Advertising

  2. Wenning Qiu

    Andrew Dalke Guest

    Wenning Qiu:
    > I am researching issues related to emdedding Python in C++ for a
    > project.


    > Has anyone on this list run into the same problem that I have, or does
    > anyone know of any plan of totally insulating multiple embedded Python
    > interpreters?


    Ahh, the Global Interpreter Lock (GIL).

    Years ago, Greg Stein had a version of Python 1.4 running with no GIL.

    http://www.python.org/ftp/python/contrib-09-Dec-1999/System/threading.README
    Search for "free threading" to get more hits on this topic.

    As I recalled, it slowed down the performance on
    single-processor/single-threaded
    machines, so the general take was to keep the GIL. In addition, see
    http://groups.google.com/groups?selm=mailman.1008992607.2279.python-list@p
    ython.org&oe=UTF-8&output=gplain
    Tim Peters:
    ] The prospects for another version of that grow dimmer. Everyone (incl.
    ] Greg) has noticed that CPython internals, over time, increase their
    reliance
    ] on the thread-safety guarantees of the global interpreter lock.

    The only solutions I know of are explicit multi-process solutions:
    - a generic system, like XML-RPC/SOAP/PVM/MPI/CORBA, on
    which you build your own messaging system
    - use systems like Pyro or Twisted, which understand Python objects
    and implement 'transparent' proxying via network communications
    - use POSH, which does the proxying through shared memory (but
    this uses Intel-specific assembly)

    Andrew
    Andrew Dalke, Jul 8, 2003
    #2
    1. Advertising

  3. Wenning Qiu

    Afanasiy Guest

    On 8 Jul 2003 14:54:22 -0700, (Wenning Qiu)
    wrote:

    >I am researching issues related to emdedding Python in C++ for a
    >project.
    >
    >My project will be running on an SMP box and requires scalability.
    >However, my test shows that Python threading has very poor performance
    >in terms of scaling. In fact it doesn't scale at all.
    >
    >I wrote a simple test program to complete given number of iterations
    >of a simple loop. The total number of iterations can be divided evenly
    >among a number of threads. My test shows that as the number of threads
    >grows, the CPU usage grows and the response time gets longer. For
    >example, to complete the same amount of work, one thread takes 10
    >seconds, 2 threads take 20 seconds and 3 threads take 30 seconds.
    >
    >The fundamental reason for lacking scalability is that Python uses a
    >global interpreter lock for thread safety. That global lock must be
    >held by a thread before it can safely access Python objects.


    I asked once and was told it was best fixed by removing the documentation
    which mentioned it. Others also stated it was unlikely to be fixed.

    http://groups.google.com/groups?hl=...=53u1evk5jmdcgma5e8eupbe3tn45js302i%404ax.com

    However, I believe Lua since 4-work4, just before Lua 5, solved this.
    Unfortunately Lua is not Python.

    Another thing to consider if you care about SMP, is your C/C++ memory
    management, assuming you aren't using something custom already, maybe a
    shared heap. I have worked wonders with libhoard (and SmartHeap,
    commercially). Some applications will run slower on SMP than if you
    removed one of the processors.

    www.hoard.org
    www.microquill.com

    mmm, graphs

    -AB
    Afanasiy, Jul 10, 2003
    #3
  4. Wenning Qiu

    Aahz Guest

    In article <>,
    Wenning Qiu <> wrote:
    >
    >My project will be running on an SMP box and requires scalability.
    >However, my test shows that Python threading has very poor performance
    >in terms of scaling. In fact it doesn't scale at all.


    That's true for pure Python code.

    >The fundamental reason for lacking scalability is that Python uses a
    >global interpreter lock for thread safety. That global lock must be
    >held by a thread before it can safely access Python objects.


    Correct. The problem is that the GIL makes Python more efficient in
    many ways, because there's no need for fine-grained locking. You're
    using Python inside-out for this purpose -- the way to scale Python in a
    threaded environment is to call out to a C extension that releases the
    GIL.

    >Has anyone on this list run into the same problem that I have, or does
    >anyone know of any plan of totally insulating multiple embedded Python
    >interpreters?


    Sure! Use multiple processes.

    Other people have mentioned Perl and Tcl in this thread. I wonder how
    they deal with the problem of loading DLLs with static data.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Not everything in life has a clue in front of it...." --JMS
    Aahz, Jul 10, 2003
    #4
  5. Wenning Qiu

    Jeff Epler Guest

    On Thu, Jul 10, 2003 at 03:54:14PM -0400, Aahz wrote:
    > Other people have mentioned Perl and Tcl in this thread. I wonder how
    > they deal with the problem of loading DLLs with static data.


    As far as I know, tcl enforces a one interpreter to one thread requirement.
    An extension should have only thread-local data, using a Tcl-supplied API.

    Jeff
    Jeff Epler, Jul 10, 2003
    #5
  6. Wenning Qiu

    Aahz Guest

    In article <>,
    Jeff Epler <> wrote:
    >On Thu, Jul 10, 2003 at 03:54:14PM -0400, Aahz wrote:
    >>
    >> Other people have mentioned Perl and Tcl in this thread. I wonder how
    >> they deal with the problem of loading DLLs with static data.

    >
    >As far as I know, tcl enforces a one interpreter to one thread
    >requirement. An extension should have only thread-local data, using a
    >Tcl-supplied API.


    What happens when Tcl wants to interact with some 3rd-party DLL that is
    *not* thread-safe?
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Not everything in life has a clue in front of it...." --JMS
    Aahz, Jul 11, 2003
    #6
  7. Wenning Qiu

    Jeff Epler Guest

    On Thu, Jul 10, 2003 at 07:48:57PM -0400, Aahz wrote:
    > In article <>,
    > Jeff Epler <> wrote:
    > >On Thu, Jul 10, 2003 at 03:54:14PM -0400, Aahz wrote:
    > >>
    > >> Other people have mentioned Perl and Tcl in this thread. I wonder how
    > >> they deal with the problem of loading DLLs with static data.

    > >
    > >As far as I know, tcl enforces a one interpreter to one thread
    > >requirement. An extension should have only thread-local data, using a
    > >Tcl-supplied API.

    >
    > What happens when Tcl wants to interact with some 3rd-party DLL that is
    > *not* thread-safe?


    I guess you'd have to do your own locking. Tcl has standard C APIs for
    Conditions, Mutexes, and thread-specific data, see the Thread(3) manpage.
    You'd have to surround all non-reentrant calls with Tcl_MutexLock(m)
    .... Tcl_MutexUnlock(m). If two extensions wanted to use the same
    non-thread-safe library, they'd have to cooperate in some way to use
    the same 'm' to Tcl_Mutex*(). I don't know if there's a standard way to
    do this, but I think that having the mutex defined in a shared lib they
    both link might work.

    Jeff
    Jeff Epler, Jul 11, 2003
    #7
  8. Wenning Qiu

    Aahz Guest

    In article <>,
    Jeff Epler <> wrote:
    >On Thu, Jul 10, 2003 at 07:48:57PM -0400, Aahz wrote:
    >> In article <>,
    >> Jeff Epler <> wrote:
    >>>On Thu, Jul 10, 2003 at 03:54:14PM -0400, Aahz wrote:
    >>>>
    >>>> Other people have mentioned Perl and Tcl in this thread. I wonder how
    >>>> they deal with the problem of loading DLLs with static data.
    >>>
    >>>As far as I know, tcl enforces a one interpreter to one thread
    >>>requirement. An extension should have only thread-local data, using a
    >>>Tcl-supplied API.

    >>
    >> What happens when Tcl wants to interact with some 3rd-party DLL that is
    >> *not* thread-safe?

    >
    >I guess you'd have to do your own locking. Tcl has standard C APIs for
    >Conditions, Mutexes, and thread-specific data, see the Thread(3) manpage.
    >You'd have to surround all non-reentrant calls with Tcl_MutexLock(m)
    >... Tcl_MutexUnlock(m). If two extensions wanted to use the same
    >non-thread-safe library, they'd have to cooperate in some way to use
    >the same 'm' to Tcl_Mutex*(). I don't know if there's a standard way to
    >do this, but I think that having the mutex defined in a shared lib they
    >both link might work.


    Yup. And that's exactly why there has been little movement to remove
    the GIL from Python. One of Python's core strengths is the ease with
    which random DLLs can be used from Python.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Not everything in life has a clue in front of it...." --JMS
    Aahz, Jul 11, 2003
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. tharma
    Replies:
    4
    Views:
    482
    Jason Kester
    Sep 13, 2005
  2. Jeff Epler
    Replies:
    3
    Views:
    328
    Afanasiy
    Jul 10, 2003
  3. Simon Wittber (Maptek)

    RE: Embedding Python, threading and scalability

    Simon Wittber (Maptek), Jul 11, 2003, in forum: Python
    Replies:
    7
    Views:
    352
  4. David Harrison
    Replies:
    3
    Views:
    309
    David Harrison
    May 10, 2005
  5. Replies:
    0
    Views:
    319
Loading...

Share This Page