Right way to initialize python embedded in a multi-threaded application

Discussion in 'Python' started by shooshx, May 17, 2012.

  1. shooshx

    shooshx Guest

    I'm embedding python in a multi-threaded C application.
    I've taken care to wrap every call to the Python C API with

    gstate = PyGILState_Ensure();
    // call python code
    PyGILState_Release(gstate);

    But I'm stumped with what to do in the initialization.
    Right after the call to Py_IsInitialized() I've added a call:

    PyEval_InitThreads();

    The docs say that this function leaves the GIL locked when it returns.
    I do some more initializations like importing modules and then I call

    PyEval_ReleaseLock();

    This seems to cause a problem since not long after a call to
    PyGILState_Release(gstate) that's made in a different thread crashes.
    with

    "Fatal Python error: This thread state must be current when releasing"

    If I don't do the call to PyEval_ReleaseLock() in the main thread
    right after initialization, the GIL seems to be released
    after the first PyGILState_Ensure() - PyGILState_Release() pair.

    So what am I doing wrong here?
    What is the correct way of initializing a multi-threaded application?
     
    shooshx, May 17, 2012
    #1
    1. Advertising

  2. shooshx

    Guest

    Hi,

    Maybe you already fixed the issue, but for the record, I've got the same
    problem and finally it turned out that I was calling PyEval_InitThreads twice
    and also after fixing that, I also had to move the call to PyEval_ReleaseLock(); at the end of the entire initialization (not just after PyEval_initThreads).

    The key thing there is to follow:

    <<at initialization thread>>
    Py_Initialize ();
    PyEval_InitThreads();
    /* now call here to initialize all python code by loading external files
    or internal module loading (i.e. Py_InitModule3) */
    /* ..and now, once no more Python C/API call is required, release
    the GIL so other threads can come into play */
    PyEval_ReleaseLock ();

    <<and now, from other threads, use>>
    /* wait til gil acquired */
    state = PyGILState_Ensure();
    /* your code */
    /* release GIL */
    PyGILState_Release (state);

    Hope it helps, Cheers!


    > I'm embedding python in a multi-threaded C application.
    >
    > I've taken care to wrap every call to the Python C API with
    >
    >
    >
    > gstate = PyGILState_Ensure();
    >
    > // call python code
    >
    > PyGILState_Release(gstate);
    >
    >
    >
    > But I'm stumped with what to do in the initialization.
    >
    > Right after the call to Py_IsInitialized() I've added a call:
    >
    >
    >
    > PyEval_InitThreads();
    >
    >
    >
    > The docs say that this function leaves the GIL locked when it returns.
    >
    > I do some more initializations like importing modules and then I call
    >
    >
    >
    > PyEval_ReleaseLock();
    >
    >
    >
    > This seems to cause a problem since not long after a call to
    >
    > PyGILState_Release(gstate) that's made in a different thread crashes.
    >
    > with
    >
    >
    >
    > "Fatal Python error: This thread state must be current when releasing"
    >
    >
    >
    > If I don't do the call to PyEval_ReleaseLock() in the main thread
    >
    > right after initialization, the GIL seems to be released
    >
    > after the first PyGILState_Ensure() - PyGILState_Release() pair.
    >
    >
    >
    > So what am I doing wrong here?
    >
    > What is the correct way of initializing a multi-threaded application?
     
    , May 11, 2013
    #2
    1. Advertising

  3. shooshx

    Guest

    Just clarify there's no problem about calling twice to PyEval_InitThreads ()
    as indicated by Python's doc.

    > Hi,
    >
    >
    >
    > Maybe you already fixed the issue, but for the record, I've got the same
    >
    > problem and finally it turned out that I was calling PyEval_InitThreads twice
    >
    > and also after fixing that, I also had to move the call to PyEval_ReleaseLock(); at the end of the entire initialization (not just after PyEval_initThreads).
    >
    >
    >
    > The key thing there is to follow:
    >
    >
    >
    > <<at initialization thread>>
    >
    > Py_Initialize ();
    >
    > PyEval_InitThreads();
    >
    > /* now call here to initialize all python code by loading external files
    >
    > or internal module loading (i.e. Py_InitModule3) */
    >
    > /* ..and now, once no more Python C/API call is required, release
    >
    > the GIL so other threads can come into play */
    >
    > PyEval_ReleaseLock ();
    >
    >
    >
    > <<and now, from other threads, use>>
    >
    > /* wait til gil acquired */
    >
    > state = PyGILState_Ensure();
    >
    > /* your code */
    >
    > /* release GIL */
    >
    > PyGILState_Release (state);
    >
    >
    >
    > Hope it helps, Cheers!
    >
    >
    >
    >
    >
    > > I'm embedding python in a multi-threaded C application.

    >
    > >

    >
    > > I've taken care to wrap every call to the Python C API with

    >
    > >

    >
    > >

    >
    > >

    >
    > > gstate = PyGILState_Ensure();

    >
    > >

    >
    > > // call python code

    >
    > >

    >
    > > PyGILState_Release(gstate);

    >
    > >

    >
    > >

    >
    > >

    >
    > > But I'm stumped with what to do in the initialization.

    >
    > >

    >
    > > Right after the call to Py_IsInitialized() I've added a call:

    >
    > >

    >
    > >

    >
    > >

    >
    > > PyEval_InitThreads();

    >
    > >

    >
    > >

    >
    > >

    >
    > > The docs say that this function leaves the GIL locked when it returns.

    >
    > >

    >
    > > I do some more initializations like importing modules and then I call

    >
    > >

    >
    > >

    >
    > >

    >
    > > PyEval_ReleaseLock();

    >
    > >

    >
    > >

    >
    > >

    >
    > > This seems to cause a problem since not long after a call to

    >
    > >

    >
    > > PyGILState_Release(gstate) that's made in a different thread crashes.

    >
    > >

    >
    > > with

    >
    > >

    >
    > >

    >
    > >

    >
    > > "Fatal Python error: This thread state must be current when releasing"

    >
    > >

    >
    > >

    >
    > >

    >
    > > If I don't do the call to PyEval_ReleaseLock() in the main thread

    >
    > >

    >
    > > right after initialization, the GIL seems to be released

    >
    > >

    >
    > > after the first PyGILState_Ensure() - PyGILState_Release() pair.

    >
    > >

    >
    > >

    >
    > >

    >
    > > So what am I doing wrong here?

    >
    > >

    >
    > > What is the correct way of initializing a multi-threaded application?
     
    , May 11, 2013
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mustafa Demirhan
    Replies:
    0
    Views:
    305
    Mustafa Demirhan
    Nov 9, 2004
  2. Mustafa Demirhan
    Replies:
    0
    Views:
    282
    Mustafa Demirhan
    Nov 9, 2004
  3. Mustafa Demirhan
    Replies:
    0
    Views:
    609
    Mustafa Demirhan
    Nov 9, 2004
  4. ian douglas
    Replies:
    2
    Views:
    1,008
    Randy Howard
    Jul 30, 2004
  5. Yue Fei
    Replies:
    0
    Views:
    321
    Yue Fei
    Dec 20, 2007
Loading...

Share This Page