using python interpreters per thread in C++ program

Discussion in 'Python' started by grbgooglefan, Sep 7, 2009.

  1. grbgooglefan

    grbgooglefan Guest

    Hi

    I've a multi-threaded C++ program, in which I want to use embedded
    python interpreter for each thread. I am using Python 2.6.2 on Linux
    for this.

    When I tried to embed python interpreter per thread, I got crash when
    the threads were calling Python's C APIs.

    Can we not use python interpreters even private to each multiple
    thread?

    What is best way to embed python in multi-threaded C++ application?

    Please guide.
     
    grbgooglefan, Sep 7, 2009
    #1
    1. Advertising

  2. grbgooglefan

    sturlamolden Guest

    On 7 Sep, 07:17, grbgooglefan <> wrote:


    > What is best way to embed python in multi-threaded C++ application?


    Did you remeber to acquire the GIL? The GIL is global to the process
    (hence the name).

    void foobar(void)
    {
    PyGILState_STATE state = PyGILState_Ensure();

    /* Safe to use Python C API here */

    PyGILState_Release(state);
    }


    S.M.











    > Please guide.
     
    sturlamolden, Sep 7, 2009
    #2
    1. Advertising

  3. grbgooglefan

    sturlamolden Guest

    On 7 Sep, 07:17, grbgooglefan <> wrote:

    > Can we not use python interpreters even private to each multiple
    > thread?


    You can use multiple interpreters, but they share GIL. For example,
    Python extension modules are DLLs and will be loaded only once for
    each process - the OS makes sure of that. Since they are objects too,
    there can only be one GIL per process.

    The same goes for files: If an interpreter calls close on a file
    handle, it is closed for the whole process, not just locally to the
    interpreter. Global synchronization is therefore needed. (Maybe not
    for your threaded app, but for any conceivable use of Python.)

    If you need two isolated interpreters, you need to run them in
    different processes.
     
    sturlamolden, Sep 7, 2009
    #3
  4. grbgooglefan

    ganesh Guest

    > Did you remeber to acquire the GIL? The GIL is global to the process

    No, I did not use GIL.

    -- Why do we need to use GIL even though python is private to each
    thread?
    -- For using GIL, do we need to initialize GIL at startup and destroy/
    finalize it at end?
    -- With GIL, we are not achieiving concurrent operations on python
    interpreters across threads. When it comes to calling Py-C APIs,
    program is working like in a single threaded mode. Is there any way,
    we can avoid the GIL locking, etc?

    Please guide. Thanks.
     
    ganesh, Sep 7, 2009
    #4
  5. grbgooglefan

    ganesh Guest

    > Did you remeber to acquire the GIL? The GIL is global to the process
    > (hence the name).


    No, I did not use GIL.
    -- For using GIL, do we need to initialize GIL at startup and destroy/
    finalize it at end?
    -- Are there any configuration & build related flags that I need to
    use to make this work?

    Please guide. Thanks.
     
    ganesh, Sep 7, 2009
    #5
  6. grbgooglefan

    sturlamolden Guest

    On 7 Sep, 07:59, ganesh <> wrote:

    > No, I did not use GIL.
    > -- For using GIL, do we need to initialize GIL at startup and destroy/
    > finalize it at end?
    > -- Are there any configuration & build related flags that I need to
    > use to make this work?
    >
    > Please guide. Thanks.


    I just showed you how...
     
    sturlamolden, Sep 7, 2009
    #6
  7. grbgooglefan

    ganesh Guest

    On Sep 7, 2:04 pm, sturlamolden <> wrote:
    > I just showed you how...


    Modified the thread function to use these APIs, but the call to
    PyGILState_Ensure() is not returning at all.

    void *callPyFunction(void * arg)
    {
    // Method two to get function eval
    long thridx=(long)arg;
    printf("\n---->my num=%d, calling showstr pyfunction\n",thridx);
    char callpyf[] = "showstr(100)\n";
    PyGILState_STATE state = PyGILState_Ensure();
    printf("after PyGILState_Ensure\n");
    PyObject* pycall = PyRun_String(callpyf,Py_file_input,glb, loc);
    if(pycall == NULL || PyErr_Occurred()){
    printf("PyRun_String failed\n");
    PyErr_Print();
    } else
    printf("%d thread called showstr pyfunction ok\n",thridx);
    PyGILState_Release(state);
    pthread_exit(NULL);
    }
     
    ganesh, Sep 7, 2009
    #7
  8. On Sep 7, 3:42 pm, sturlamolden <> wrote:
    > On 7 Sep, 07:17, grbgooglefan <> wrote:
    >
    > > What is best way to embed python in multi-threaded C++ application?

    >
    > Did you remeber to acquire the GIL? The GIL is global to the process
    > (hence the name).
    >
    > void foobar(void)
    > {
    >     PyGILState_STATE state = PyGILState_Ensure();
    >
    >     /* Safe to use Python C API here */
    >
    >     PyGILState_Release(state);


    You can't use that in this case as they specifically are talking about
    a interpreter per thread, which implies creating additional sub
    interpreters. The simplified GIL state API you mentioned only works
    for threads operating in the main (first) interpreter created within
    the process.

    The OP can do what they want, but they need to user lower level
    routines for creating their own thread state objects and acquiring the
    GIL against them.

    Graham
     
    Graham Dumpleton, Sep 7, 2009
    #8
  9. grbgooglefan

    ganesh Guest

    On Sep 7, 3:41 pm, Graham Dumpleton <>
    wrote:
    > On Sep 7, 3:42 pm, sturlamolden <> wrote:
    > interpreters. The simplified GIL state API you mentioned only works
    > for threads operating in the main (first) interpreter created within
    > the process.


    I modified my program to have Py_Initialize and compilation of one
    Python function done in main() thread. Then I am calling only that
    function in callPyFunction() thread. But, this thread does not come
    out of PyGILState_Ensure() function.

    > The OP can do what they want, but they need to user lower level
    > routines for creating their own thread state objects and acquiring the
    > GIL against them.
    >
    > Graham


    What are the "lower level routines" for creating own thread state
    objects & acquiring GILs.
    Also, where can I find more information about those routines?

    Please guide. Thanks.
     
    ganesh, Sep 7, 2009
    #9
  10. On Sep 7, 6:47 pm, ganesh <> wrote:
    > On Sep 7, 3:41 pm, Graham Dumpleton <>
    > wrote:
    >
    > > On Sep 7, 3:42 pm, sturlamolden <> wrote:
    > > interpreters. The simplified GIL state API you mentioned only works
    > > for threads operating in the main (first) interpreter created within
    > > the process.

    >
    > I modified my program to have Py_Initialize and compilation of one
    > Python function done in main() thread. Then I am calling only that
    > function in callPyFunction() thread. But, this thread does not come
    > out of PyGILState_Ensure() function.
    >
    > > The OP can do what they want, but they need to user lower level
    > > routines for creating their own thread state objects and acquiring the
    > > GIL against them.

    >
    > > Graham

    >
    > What are the "lower level routines" for creating own thread state
    > objects & acquiring GILs.
    > Also, where can I find more information about those routines?


    Documentation is at:

    http://docs.python.org/c-api/init.html

    Are you really using sub interpreters though? There is no evidence of
    that in the code you posted earlier.

    Sure you just don't understand enough about it and so are using the
    wrong terminology and all you really want to do is run externally
    created threads through the one interpreter.

    Using sub interpreters is not for the feint of heart and not something
    you want to do unless you want to understand Python C API internals
    very well.

    Graham
     
    Graham Dumpleton, Sep 7, 2009
    #10
  11. ganesh wrote:
    >> Did you remeber to acquire the GIL? The GIL is global to the process

    >
    > No, I did not use GIL.
    >
    > -- Why do we need to use GIL even though python is private to each
    > thread?


    Quoting from above: "The GIL is global to the process". So no, it is NOT
    private to each thread which means "python" isn't either.

    At least that is my understanding of the issue.

    Uli

    --
    Sator Laser GmbH
    Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
     
    Ulrich Eckhardt, Sep 7, 2009
    #11
  12. grbgooglefan

    ganesh Guest

    Actually, I modified my program to have a single shared Py-interpreter
    across all threads to test the usage of GIL. So, I did Py_Initialize
    in main() function and only called that python function in different
    threads.

    But this is not the way I want to use interpreters in my code.

    I am looking for using sub-interpreters, only that I did not know this
    this term, till you mentioned it here.
    So bvefore this, I was calling py_Initialize in each of the C++ level
    threads, which was wrong.

    I should be using Py_NewInterpreter() in my threads and Py_Initialize
    () in main() thread. Please correct me if I am wrong.

    >>>>not something you want to do unless you want to understand Python C API internals very well.

    Are these APIs really tricky to use with C++ embedding? Can't they be
    implemented thru C APIs?

    I need to use these to get the proper concurrency in my multi-threaded
    application without any synchronization mechanisms.
     
    ganesh, Sep 7, 2009
    #12
  13. grbgooglefan

    sturlamolden Guest

    On 7 Sep, 13:53, ganesh <> wrote:

    > I need to use these to get the proper concurrency in my multi-threaded
    > application without any synchronization mechanisms.


    Why will multiple interpreters give you better concurrency? You can
    have more than one thread in the same interpreter.

    Here is the API explained:

    http://docs.python.org/c-api/init.html
    http://www.linuxjournal.com/article/3641
     
    sturlamolden, Sep 7, 2009
    #13
  14. grbgooglefan

    sturlamolden Guest

    On 7 Sep, 13:17, Ulrich Eckhardt <> wrote:

    > Quoting from above: "The GIL is global to the process". So no, it is NOT
    > private to each thread which means "python" isn't either.
    >
    > At least that is my understanding of the issue.


    Strictly speaking, the GIL is global to the Python DLL, not the
    process. You can load it multiple times from different paths, and if
    you do GIL with not be shared.

    If you make several clean installs of Python on Windows (say c:
    \Python26-0 ... c:\Python26-3), you can embed multiple interpreters in
    a process, and they will not share GIL or anything else. But be
    careful: all *.pyd files must reside in these directories, or they
    will be loaded once and refcounts screw up. The Python DLL must also
    be in these directories, not in c:\windows\system32. It is in fact
    possible to make Python utilize dual- and quad-core processors on
    Windows this way. You can even use ctypes for embedding Python into
    Python, so no C is required. See:

    http://groups.google.no/group/comp....read/thread/2d537ad8df9dab67/812013f9ef3a766d

    To make this hack really work one need multiple complete installs, not
    just copies of one DLL as I assumed in the thread. But the general
    method is the same.

    You can see this as a form of primitive object orientation :p


    Sturla Molden
     
    sturlamolden, Sep 7, 2009
    #14
  15. grbgooglefan

    MRAB Guest

    sturlamolden wrote:
    > On 7 Sep, 13:53, ganesh <> wrote:
    >
    >> I need to use these to get the proper concurrency in my multi-threaded
    >> application without any synchronization mechanisms.

    >
    > Why will multiple interpreters give you better concurrency? You can
    > have more than one thread in the same interpreter.
    >
    > Here is the API explained:
    >
    > http://docs.python.org/c-api/init.html
    > http://www.linuxjournal.com/article/3641


    CPython's GIL means that multithreading on multiple processors/cores has
    limitations. Each interpreter has its own GIL, so processor-intensive
    applications work better using the multiprocessing module than with the
    threading module.
     
    MRAB, Sep 7, 2009
    #15
  16. grbgooglefan

    sturlamolden Guest

    On 7 Sep, 14:50, MRAB <> wrote:

    > CPython's GIL means that multithreading on multiple processors/cores has
    > limitations. Each interpreter has its own GIL, so processor-intensive
    > applications work better using the multiprocessing module than with the
    > threading module.


    We incur a 200x speed-penalty from Python if the code is CPU-bound.
    How much do you gain from one extra processor? Start by adding in some
    C, and you will gain much more performance, even without parallel
    processing.

    Processor-intensive code should therefore use extension libraries that
    release the GIL. Then you have option of getting parallel concurrency
    by Python threads or OpenMP in C/Fortran. Most processor-intensive
    code depend on numerical libraries written in C or Fortran anyway
    (NumPy, ATLAS, LAPACK, FFTW, GSL, MKL, etc.) When you do this, the GIL
    does not get in your way. The GIL is infact an advantage if one
    library happen not to be thread safe. Many old Fortran 77 libraries
    have functions with re-entrancy issues, due to SAVE attribute on
    variables. Thus, the GIL is often an advantage for processor-intensive
    code.

    multiprocessing is fragile and unreliable, btw.
     
    sturlamolden, Sep 7, 2009
    #16
  17. grbgooglefan

    Mark Hammond Guest

    On 7/09/2009 10:50 PM, MRAB wrote:
    > sturlamolden wrote:
    >> On 7 Sep, 13:53, ganesh <> wrote:
    >>
    >>> I need to use these to get the proper concurrency in my multi-threaded
    >>> application without any synchronization mechanisms.

    >>
    >> Why will multiple interpreters give you better concurrency? You can
    >> have more than one thread in the same interpreter.
    >>
    >> Here is the API explained:
    >>
    >> http://docs.python.org/c-api/init.html
    >> http://www.linuxjournal.com/article/3641

    >
    > CPython's GIL means that multithreading on multiple processors/cores has
    > limitations. Each interpreter has its own GIL, so processor-intensive
    > applications work better using the multiprocessing module than with the
    > threading module.


    I believe you will find the above is incorrect - even with multiple
    interpreter states you still have a single GIL.

    Mark
     
    Mark Hammond, Sep 7, 2009
    #17
  18. On Mon, Sep 7, 2009 at 6:31 PM, Mark Hammond<> wrote:
    > On 7/09/2009 10:50 PM, MRAB wrote:
    >>
    >> sturlamolden wrote:
    >>>
    >>> On 7 Sep, 13:53, ganesh <> wrote:
    >>>
    >>>> I need to use these to get the proper concurrency in my multi-threaded
    >>>> application without any synchronization mechanisms.
    >>>
    >>> Why will multiple interpreters give you better concurrency? You can
    >>> have more than one thread in the same interpreter.
    >>>
    >>> Here is the API explained:
    >>>
    >>> http://docs.python.org/c-api/init.html
    >>> http://www.linuxjournal.com/article/3641

    >>
    >> CPython's GIL means that multithreading on multiple processors/cores has
    >> limitations. Each interpreter has its own GIL, so processor-intensive
    >> applications work better using the multiprocessing module than with the
    >> threading module.

    >
    > I believe you will find the above is incorrect - even with multiple
    > interpreter states you still have a single GIL.
    >

    not according to the docs.

    http://docs.python.org/library/multiprocessing.html :

    multiprocessing is a package that supports spawning processes using an
    API similar to the threading module. The multiprocessing package
    offers both local and remote concurrency, effectively side-stepping
    the Global Interpreter Lock by using subprocesses instead of threads.
    Due to this, the multiprocessing module allows the programmer to fully
    leverage multiple processors on a given machine. It runs on both Unix
    and Windows.
    > Mark
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    Benjamin Kaplan, Sep 7, 2009
    #18
  19. grbgooglefan

    Mark Hammond Guest

    On 8/09/2009 9:16 AM, Grant Edwards wrote:
    > On 2009-09-07, Mark Hammond<> wrote:
    >
    >>> CPython's GIL means that multithreading on multiple
    >>> processors/cores has limitations. Each interpreter has its own
    >>> GIL, so processor-intensive applications work better using the
    >>> multiprocessing module than with the threading module.

    >>
    >> I believe you will find the above is incorrect - even with
    >> multiple interpreter states you still have a single GIL.

    >
    > Please explain how multiple processes, each with a separate
    > Python interpreter, share a single GIL.



    Sorry, my mistake, I misread the original - using multiple Python
    processes does indeed have a GIL per process. I was referring to the
    'multiple interpreters in one process' feature of Python which is
    largely deprecated, but if used, all 'interpreters' share the same GIL.

    To clarify: in a single process there will only ever be one GIL, but in
    multiple processes there most certainly will be multiple GILs.

    Apologies for the confusion...

    Cheers,

    Mark
     
    Mark Hammond, Sep 8, 2009
    #19
  20. grbgooglefan

    ganesh Guest

    My application is a TCP server having multiple client connectons. C++
    PTHREADS are for each connected socket and the message received on the
    socket is evaluated by python functions.
    If I use only one process level python interpreter, then every thread
    has to lock the GIL & so blocking the other threads from executing the
    python code even if it is not the same python function that locking
    thread is calling.

    -- That's why I tried using python interpreters per thread. But that
    also required GIL locking & so cannot be used.

    -- I cannot use python threads inside the Pyton intrepreter because
    then I will have to have some mechanism for communiaction between C++
    Pthreads with these python threads.

    I think there is no way that we can achieve this because of the GIL
    being a process level state. Least I can do is have one python
    interpreter initialized in main thread and lock the GIL in every
    thread for python calls.
     
    ganesh, Sep 8, 2009
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Torsten Mohr

    allocate TWO interpreters in a C program?

    Torsten Mohr, Apr 4, 2004, in forum: Python
    Replies:
    2
    Views:
    274
    David E. Konerding DSD staff
    Apr 5, 2004
  2. bmatt
    Replies:
    2
    Views:
    310
    Frithiof Andreas Jensen
    Sep 29, 2004
  3. Replies:
    3
    Views:
    319
  4. Replies:
    9
    Views:
    790
    Rhamphoryncus
    Apr 13, 2008
  5. liu yang
    Replies:
    4
    Views:
    2,008
    Antoninus Twink
    Jul 28, 2008
Loading...

Share This Page