Multiple independent Python interpreters in a C/C++ program?

Discussion in 'Python' started by skip@pobox.com, Apr 11, 2008.

  1. Guest

    This question was posed to me today. Given a C/C++ program we can clearly
    embed a Python interpreter in it. Is it possible to fire up multiple
    interpreters in multiple threads? For example:

    C++ main
    thread 1
    Py_Initialize()
    thread 2
    Py_Initialize()

    Do I wind up with two completely independent interpreters, one per thread?
    I'm thinking this doesn't work (there are bits which aren't thread-safe and
    are only protected by the GIL), but wanted to double-check to be sure.

    Thanks,

    Skip
     
    , Apr 11, 2008
    #1
    1. Advertising

  2. schrieb:
    > This question was posed to me today. Given a C/C++ program we can clearly
    > embed a Python interpreter in it. Is it possible to fire up multiple
    > interpreters in multiple threads? For example:
    >
    > C++ main
    > thread 1
    > Py_Initialize()
    > thread 2
    > Py_Initialize()
    >
    > Do I wind up with two completely independent interpreters, one per thread?
    > I'm thinking this doesn't work (there are bits which aren't thread-safe and
    > are only protected by the GIL), but wanted to double-check to be sure.


    AFAIK there was a thread a few month ago that stated that this is
    actually possible - but mostly 3rd-party-c-extension aren't capable of
    supporting that. Martin von Loewis had a word in that, maybe googling
    with that in mind reveals the discussion.

    And of course its a *bad* idea to pass objects between threads...

    Diez
     
    Diez B. Roggisch, Apr 11, 2008
    #2
    1. Advertising

  3. Matimus Guest

    On Apr 11, 9:24 am, wrote:
    > This question was posed to me today. Given a C/C++ program we can clearly
    > embed a Python interpreter in it. Is it possible to fire up multiple
    > interpreters in multiple threads? For example:
    >
    > C++ main
    > thread 1
    > Py_Initialize()
    > thread 2
    > Py_Initialize()
    >
    > Do I wind up with two completely independent interpreters, one per thread?
    > I'm thinking this doesn't work (there are bits which aren't thread-safe and
    > are only protected by the GIL), but wanted to double-check to be sure.
    >
    > Thanks,
    >
    > Skip


    AFAIK it is only `sort of' possible. But not like that. See the API
    documentation on Py_NewInterpreter http://www.python.org/doc/api/initialization.html#l2h-825

    As you will see from the documentation it creates an interpreter that
    isn't 100% separate. It may work for your application though.

    Also, I don't think using the new interpreter is as simple as just
    instantiating the new interpreter in a separate thread. But there is
    much relevant information on the page that I linked to.

    Matt
     
    Matimus, Apr 11, 2008
    #3
  4. Diez B. Roggisch schrieb:
    > AFAIK there was a thread a few month ago that stated that this is
    > actually possible - but mostly 3rd-party-c-extension aren't capable of
    > supporting that. Martin von Loewis had a word in that, maybe googling
    > with that in mind reveals the discussion.
    >
    > And of course its a *bad* idea to pass objects between threads...


    http://www.python.org/dev/peps/pep-3121/

    Christian
     
    Christian Heimes, Apr 12, 2008
    #4
  5. On Apr 11, 10:24 am, wrote:
    > This question was posed to me today. Given a C/C++ program we can clearly
    > embed a Python interpreter in it. Is it possible to fire up multiple
    > interpreters in multiple threads? For example:
    >
    > C++ main
    > thread 1
    > Py_Initialize()
    > thread 2
    > Py_Initialize()
    >
    > Do I wind up with two completely independent interpreters, one per thread?
    > I'm thinking this doesn't work (there are bits which aren't thread-safe and
    > are only protected by the GIL), but wanted to double-check to be sure.


    Since they interpreters would never truly be separate, I think it's
    best to bite the bullet and use multiple threads within one
    interpreter. The only really difference is pure python modules aren't
    duplicated, but why would you need that? Surely not for any kind of
    security.
     
    Rhamphoryncus, Apr 12, 2008
    #5
  6. sturlamolden Guest

    On Apr 11, 6:24 pm, wrote:

    > Do I wind up with two completely independent interpreters, one per thread?
    > I'm thinking this doesn't work (there are bits which aren't thread-safe and
    > are only protected by the GIL), but wanted to double-check to be sure.


    You can create a new interpreter with a call to Py_NewInterpreter.

    However, the GIL is a global object for the process. If you have more
    than two interpreters in the process, they share the same GIL.

    In tcl, each thread has its own interpreter instance and no GIL is
    shared. This circumvents most of the problems with a global GIL.

    In theory, a GIL private to each interpreter would make Python more
    scalable. The current GIL behaves like the BKL in earlier Linux
    kernels. However, some third-party software, notably Apache's
    mod_python, is claimed to depend on this behaviour.
     
    sturlamolden, Apr 12, 2008
    #6
  7. sturlamolden Guest

    On Apr 11, 6:24 pm, wrote:

    > Do I wind up with two completely independent interpreters, one per thread?
    > I'm thinking this doesn't work (there are bits which aren't thread-safe and
    > are only protected by the GIL), but wanted to double-check to be sure.


    You can create a new subinterpreter with a call to Py_NewInterpreter.
    You get a nwe interpreter, but not an independent one. The GIL is a
    global object for the process. If you have more than one interpreter
    in the process, they share the same GIL.

    In tcl, each thread has its own interpreter instance and no GIL is
    shared. This circumvents most of the problems with a global GIL.

    In theory, a GIL private to each (sub)interpreter would make Python
    more scalable. The current GIL behaves like the BKL in earlier Linux
    kernels. However, some third-party software, notably Apache's
    mod_python, is claimed to depend on this behaviour.
     
    sturlamolden, Apr 12, 2008
    #7
  8. sturlamolden Guest

    C API design flaw (was: Re: Multiple independent Pythoninterpreters in a C/C++ program?)

    On Apr 12, 7:05 pm, sturlamolden <> wrote:

    > In theory, a GIL private to each (sub)interpreter would make Python
    > more scalable. The current GIL behaves like the BKL in earlier Linux
    > kernels. However, some third-party software, notably Apache's
    > mod_python, is claimed to depend on this behaviour.


    I just looked into the reason why ctypes, mod_python, etc. depend on a
    shared GIL. Well ... PyGILState_Ensure() does not take an argument, so
    it does not know which interpreter's GIL to acquire. Duh!

    The sad fact is, the functions in Python's C API does not take the
    interpreter as an argument, so they default to the one that is
    currently active (i.e. the one that PyThreadState_Get()->interp points
    to). This is unlike Java's JNI, in which all functions take the
    "environment" (a Java VM instance) as the first argument.

    IMHO, I consider this a major design flaw in Python's C API. In a more
    well thought API, PyGILState_Ensure would take the interpreter
    returned by Py_NewInterpreter as an argument, and thus know the
    interpreter with which to synchronize.

    I complained about this before, but the answer I got was that the
    simplified GIL API would not work if interpreters has a separate GIL.
    Obviously it would not work as long as the PyGILState_Ensure does not
    take any arguments. The lack of a leading VM argument in
    PyGILState_Ensure, and almost every function in Python's C API, is the
    heart of the problem.
     
    sturlamolden, Apr 12, 2008
    #8
  9. On Apr 13, 3:05 am, sturlamolden <> wrote:
    > On Apr 11, 6:24 pm, wrote:
    >
    > > Do I wind up with two completely independent interpreters, one per thread?
    > > I'm thinking this doesn't work (there are bits which aren't thread-safe and
    > > are only protected by the GIL), but wanted to double-check to be sure.

    >
    > You can create a new subinterpreter with a call to Py_NewInterpreter.
    > You get a nwe interpreter, but not an independent one. The GIL is a
    > global object for the process. If you have more than one interpreter
    > in the process, they share the same GIL.
    >
    > In tcl, each thread has its own interpreter instance and no GIL is
    > shared. This circumvents most of the problems with a global GIL.
    >
    > In theory, a GIL private to each (sub)interpreter would make Python
    > more scalable. The current GIL behaves like the BKL in earlier Linux
    > kernels. However, some third-party software, notably Apache'smod_python, is claimed to depend on this behaviour.


    I wouldn't use mod_python as a good guide on how to do this as it
    doesn't properly use PyGILState_Ensure() for main interpreter like it
    should. If you want an example of how to do this, have a look at code
    for mod_wsgi instead. If you want it to work for Python 3.0 as well as
    Python 2.X, make sure you look at mod_wsgi source code from subversion
    repository trunk as tweak had to be made to source to support Python
    3.0. This is because in Python 3.0 it is no longer sufficient to hold
    only the GIL when using string/unicode functions, you also need a
    proper thread state to be active now.

    Do note that although multiple sub interpreters can be made to work,
    destroying sub interpreters within the context of a running process,
    ie., before the process ends, can be a cause for various problems with
    third party C extension modules and thus would advise that once a sub
    interpreter has been created, you keep it and use it for the life of
    the process.

    Graham
     
    Graham Dumpleton, Apr 12, 2008
    #9
  10. Re: C API design flaw (was: Re: Multiple independent Pythoninterpreters in a C/C++ program?)

    On Apr 12, 2:02 pm, sturlamolden <> wrote:
    > On Apr 12, 7:05 pm, sturlamolden <> wrote:
    >
    > > In theory, a GIL private to each (sub)interpreter would make Python
    > > more scalable. The current GIL behaves like the BKL in earlier Linux
    > > kernels. However, some third-party software, notably Apache's
    > > mod_python, is claimed to depend on this behaviour.

    >
    > I just looked into the reason why ctypes, mod_python, etc. depend on a
    > shared GIL. Well ... PyGILState_Ensure() does not take an argument, so
    > it does not know which interpreter's GIL to acquire. Duh!
    >
    > The sad fact is, the functions in Python's C API does not take the
    > interpreter as an argument, so they default to the one that is
    > currently active (i.e. the one that PyThreadState_Get()->interp points
    > to). This is unlike Java's JNI, in which all functions take the
    > "environment" (a Java VM instance) as the first argument.
    >
    > IMHO, I consider this a major design flaw in Python's C API. In a more
    > well thought API, PyGILState_Ensure would take the interpreter
    > returned by Py_NewInterpreter as an argument, and thus know the
    > interpreter with which to synchronize.
    >
    > I complained about this before, but the answer I got was that the
    > simplified GIL API would not work if interpreters has a separate GIL.
    > Obviously it would not work as long as the PyGILState_Ensure does not
    > take any arguments. The lack of a leading VM argument in
    > PyGILState_Ensure, and almost every function in Python's C API, is the
    > heart of the problem.


    Alternatively, the multiple interpreter API could be ripped out, and
    you could just use separate threads. Do you have a use case that
    requires it?
     
    Rhamphoryncus, Apr 13, 2008
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Torsten Mohr

    allocate TWO interpreters in a C program?

    Torsten Mohr, Apr 4, 2004, in forum: Python
    Replies:
    2
    Views:
    277
    David E. Konerding DSD staff
    Apr 5, 2004
  2. Craig Ringer
    Replies:
    1
    Views:
    400
    Mustafa Demirhan
    Nov 18, 2004
  3. vishnu
    Replies:
    0
    Views:
    377
    vishnu
    Mar 31, 2007
  4. Marcin Kalicinski
    Replies:
    3
    Views:
    605
    Graham Dumpleton
    Jun 10, 2007
  5. grbgooglefan
    Replies:
    24
    Views:
    1,797
    Graham Dumpleton
    Sep 8, 2009
Loading...

Share This Page