Multithreaded C API Python questions

Discussion in 'Python' started by Svein Seldal, Nov 9, 2006.

  1. Svein Seldal

    Svein Seldal Guest

    Hi

    A couple of mulithreaded C API python questions:

    I) The PyGILState_Ensure() simply ensures python api call ability, it
    doesnt actually lock the GIL, right?

    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    II) Am I required to lock the GIL prior to running any python functions
    (in a threaded app)?

    PyThreadState *pts = PyGILState_GetThisThreadState();
    PyEval_AcquireThread(pts);
    PyObject_CallObject(...);

    III) Shouldn't the GIL be released after 100 bytecodes or so to enable
    other waiting threads to get the GIL?

    I'm unable to get access to python as long as another python call is
    executing. The PyEval_AcquireThread() call blocks until the first call
    returns. I was hoping that the python system itself would release the
    GIL after some execution, but it itsnt.

    I am dependent upon the ability to have to threads executing in python
    land at the same time. How can this be done?


    Regards,
    Svein Seldal
     
    Svein Seldal, Nov 9, 2006
    #1
    1. Advertising

  2. Svein Seldal

    robert Guest

    Svein Seldal wrote:
    > Hi
    >
    > A couple of mulithreaded C API python questions:
    >
    > I) The PyGILState_Ensure() simply ensures python api call ability, it
    > doesnt actually lock the GIL, right?
    >
    > PyGILState_STATE gstate;
    > gstate = PyGILState_Ensure();
    >
    > II) Am I required to lock the GIL prior to running any python functions
    > (in a threaded app)?
    >
    > PyThreadState *pts = PyGILState_GetThisThreadState();
    > PyEval_AcquireThread(pts);
    > PyObject_CallObject(...);
    >
    > III) Shouldn't the GIL be released after 100 bytecodes or so to enable
    > other waiting threads to get the GIL?
    >
    > I'm unable to get access to python as long as another python call is
    > executing. The PyEval_AcquireThread() call blocks until the first call
    > returns. I was hoping that the python system itself would release the
    > GIL after some execution, but it itsnt.
    >
    > I am dependent upon the ability to have to threads executing in python
    > land at the same time. How can this be done?
    >


    PyGILState_Ensure/Release guarantees to establish the GIL - even if it was already held (useful if you deal with complex call ordering/dependencies)


    PyObject_CallObject(...) calls Python code. Thus the interpreter will switch and do as usuall during that.


    In your/C/system... code you are responsible to release the GIL or not to enable other Python threads (and prevent from deadlocks)
    Usually you'd do this
    * if you do time consuming C/system stuff
    * or if the code can possibly renter Python through the system (e.g. when you call a Windows function which itself can create Windows messages to be routed back into Python message handlers)


    -robert
     
    robert, Nov 9, 2006
    #2
    1. Advertising

  3. Svein Seldal

    Svein Seldal Guest

    robert wrote:
    > PyGILState_Ensure/Release guarantees to establish the GIL - even if it
    > was already held (useful if you deal with complex call
    > ordering/dependencies)


    I understand this to mean that I dont need to explicitly lock the GIL
    (with PyEval_AcquireLock() or PyEval_AcquireThread()).

    Well I cant figure out the PyGILState_Ensure() because if I only use
    this function to establish the GIL, when calling python, python will
    shortly after crash with "Fatal Python error: ceval: tstate mix-up".
    This happens consistently when the main app and the extra thread has
    called python and both are busy executing python code.

    > PyObject_CallObject(...) calls Python code. Thus the interpreter
    > will switch and do as usuall during that.


    BTW What do you mean by switch?

    > In your/C/system... code you are responsible to release the GIL or not
    > to enable other Python threads (and prevent from deadlocks) Usually
    > you'd do this * if you do time consuming C/system stuff
    > * or if the code can possibly renter Python through the system (e.g.
    > when you call a Windows function which itself can create Windows
    > messages to be routed back into Python message handlers)


    The main problem is that not done this way, it's the other way around.
    My main C app will call a python function which will be a lengthy time
    consuming process.

    The main C app will call python and it will never return from
    PyObject_CallObject(...), while the extra thread should call a py
    function to deliver occational messages into C.


    Svein
     
    Svein Seldal, Nov 9, 2006
    #3
  4. Svein Seldal wrote:
    > I'm unable to get access to python as long as another python call is
    > executing. The PyEval_AcquireThread() call blocks until the first call
    > returns. I was hoping that the python system itself would release the
    > GIL after some execution, but it itsnt.
    >
    > I am dependent upon the ability to have to threads executing in python
    > land at the same time. How can this be done?


    Are you creating your threads through python or through C code? If you
    are only creating it through C code, then make sure you initialize the
    GIL with the following call at the beginning of your application:

    PyEval_InitThreads()

    From my experience, calling PyGILState_Ensure() was enough. I didn't
    need to call any extra threading functions. But make sure that every
    call to PyGILState_Ensure() is matched with a call to PyGILState_Release()

    -Farshid
     
    Farshid Lashkari, Nov 9, 2006
    #4
  5. Svein Seldal

    robert Guest

    Svein Seldal wrote:
    > robert wrote:
    >> PyGILState_Ensure/Release guarantees to establish the GIL - even if it
    >> was already held (useful if you deal with complex call
    >> ordering/dependencies)

    >
    > I understand this to mean that I dont need to explicitly lock the GIL
    > (with PyEval_AcquireLock() or PyEval_AcquireThread()).
    >
    > Well I cant figure out the PyGILState_Ensure() because if I only use
    > this function to establish the GIL, when calling python, python will
    > shortly after crash with "Fatal Python error: ceval: tstate mix-up".
    > This happens consistently when the main app and the extra thread has
    > called python and both are busy executing python code.


    Do did't tell enough to see what you want to do on the big picture.

    usually you create a thread through Python.
    If you boot Python at all from C you have to do
    PyEval_InitThreads
    or maybe more simple use the high level layer PyRun_...

    If you boot a new Python thread manually or a separated interpreter (with separated module state) from outside
    the you have to also do:

    (PyInterpreterState* PyInterpreterState_New() )
    PyThreadState* PyThreadState_New( PyInterpreterState *interp)
    void PyEval_AcquireThread( PyThreadState *tstate)


    If you are already in a Python thread but don't for some reason not know the current thread state:
    PyThreadState* PyThreadState_Get( )
    but usually you release with PyThreadState* PyEval_SaveThread( ) / Py_BEGIN_ALLOW_THREADS



    >> PyObject_CallObject(...) calls Python code. Thus the interpreter
    >> will switch and do as usuall during that.

    >
    > BTW What do you mean by switch?


    the automatic scheduling every sys.getcheckinterval() etc.. you mentioned
    You'd only have to take care for that, if your C-code does things (e.g. calling python basic funcs) for a long time and doesn't release the lock with Py_BEGIN_ALLOW_THREADS ...

    > The main problem is that not done this way, it's the other way around.
    > My main C app will call a python function which will be a lengthy time
    > consuming process.
    >
    > The main C app will call python and it will never return from
    > PyObject_CallObject(...), while the extra thread should call a py
    > function to deliver occational messages into C.


    Maybe simply boot Python in the main thread (Py_Initialize( )) and run off Python possibly as simple as PyRun_String
    and let Python do thread.start_new(...)
    You'd not have to worry much about states.

    ( and later you'd probably even more simple do "python myapp.py" and expose your C app as extension module :) )


    -robert
     
    robert, Nov 9, 2006
    #5
  6. "Svein Seldal" <"svein at seldal dot com">wrote:
    8<---------------------------------------------------
    > I am dependent upon the ability to have to threads executing in python
    > land at the same time. How can this be done?


    call time.sleep(0.001) in each, as well as the main thread, to politely give the
    rest a chance....

    Fiddle with the sleep times in your setup to get the best performance - and
    don't be misled into thinking that faster is better - its not always true -
    there comes a point where the overhead of swapping a thread consumes the total
    resource of the process, and no work gets done.

    - HTH

    - Hendrik
     
    Hendrik van Rooyen, Nov 10, 2006
    #6
  7. Svein Seldal

    Svein Seldal Guest

    Hi!

    I think I've found the bug, but I need to confirm this behavior.

    My findings is that if you use PyEval_InitThreads(), it is crucial to
    release the GIL with PyEval_ReleaseThread() afterwards.

    The API docs states that you can release the GIL with
    PyEval_ReleaseLock() or PyEval_ReleaseThread() AFAICS.
    http://docs.python.org/api/threads.html under docs of void
    PyEval_InitThreads().

    However, if I do use PyEval_ReleaseLock() it will crash shortly after
    with "Fatal Python error: ceval: tstate mix-up" in a multithreaded C
    environment.

    If I use PyEval_ReleaseThread() to release the GIL, my app seems stable.
    I release the GIL like this:

    PyThreadState *pts = PyGILState_GetThisThreadState();
    PyEval_ReleaseThread(pts);

    Now, is this a feature or expected behavior in python (i.e. unclear API
    documentation), or is this a bug in python itself?


    Regards,
    Svein


    PS:

    For reference I did something like this in pseudo-code:

    Py_Initialize();
    PyEval_InitThreads();

    other_py_inits(); // Load py modules, etc.

    PyEval_ReleaseLock(); // <-- MAKES THE APP UNSTABLE

    create_c_thread();

    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    call_py_function_main(); // Py main() wont return

    PyGILState_Release(gstate);


    And the "main" of the C thread function looks like this:
    while(1)
    {
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    call_py_function_send();

    PyGILState_Release(gstate);
    }
     
    Svein Seldal, Nov 14, 2006
    #7
  8. Svein Seldal

    robert Guest

    Svein Seldal wrote:

    You seem to use the functions in very rude manner - wanting to force the GIL around at lowest level.
    Better forget the word GIL and think about acquiring and releasing Threads.
    For each thread wanting to execute Python stuff there has to be a thread state (ts). And then you have to enter/leave for executing Python by using the thread state.


    PyEval_InitThreads enters Python (incl. the GIL) first time.
    To let Python free in main thread do
    ts=PyEval_SaveThread() // or more easy: Py_BEGIN_ALLOW_THREADS

    to reenter Python do simply:

    PyEval_RestoreThread(ts) // or: Py_END_ALLOW_THREADS

    Forget all the low level PyGIL... functions.

    when you really want another c thread to enter Python first time with no thread state existing through Python itsel (better make the thread in Python with thread.start_new ?), then you have to do once

    ts_cthread = PyThreadState_New(interp)

    then enter Python in this thread:

    PyEval_AcquireThread( ts_cthread )

    to leave again:

    PyEval_ReleaseThread( ts_cthread )


    If you just loop the call_py_function_send() mainly in this thread you probably better create the thread at all in Python and make the loop there. You probably stick too much to C-level at any price :) Probably you just do a PyRun_xxxx in main thread and then everything else in Python, and expose C-parts for the thread-loop to Python as function (in other_py_inits) - where in the c-function you probably have the usual Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS bracket during time consuming/system/reenter-endangered message stuff.


    Robert





    > I think I've found the bug, but I need to confirm this behavior.
    >
    > My findings is that if you use PyEval_InitThreads(), it is crucial to
    > release the GIL with PyEval_ReleaseThread() afterwards.
    >
    > The API docs states that you can release the GIL with
    > PyEval_ReleaseLock() or PyEval_ReleaseThread() AFAICS.
    > http://docs.python.org/api/threads.html under docs of void
    > PyEval_InitThreads().
    >
    > However, if I do use PyEval_ReleaseLock() it will crash shortly after
    > with "Fatal Python error: ceval: tstate mix-up" in a multithreaded C
    > environment.
    >
    > If I use PyEval_ReleaseThread() to release the GIL, my app seems stable.
    > I release the GIL like this:
    >
    > PyThreadState *pts = PyGILState_GetThisThreadState();
    > PyEval_ReleaseThread(pts);
    >
    > Now, is this a feature or expected behavior in python (i.e. unclear API
    > documentation), or is this a bug in python itself?
    >
    >
    > Regards,
    > Svein
    >
    >
    > PS:
    >
    > For reference I did something like this in pseudo-code:
    >
    > Py_Initialize();
    > PyEval_InitThreads();
    >
    > other_py_inits(); // Load py modules, etc.
    >
    > PyEval_ReleaseLock(); // <-- MAKES THE APP UNSTABLE
    >
    > create_c_thread();
    >
    > PyGILState_STATE gstate;
    > gstate = PyGILState_Ensure();
    >
    > call_py_function_main(); // Py main() wont return
    >
    > PyGILState_Release(gstate);
    >
    >
    > And the "main" of the C thread function looks like this:
    > while(1)
    > {
    > PyGILState_STATE gstate;
    > gstate = PyGILState_Ensure();
    >
    > call_py_function_send();
    >
    > PyGILState_Release(gstate);
    > }
     
    robert, Nov 16, 2006
    #8
  9. Svein Seldal

    Svein Seldal Guest

    robert wrote:

    > Forget all the low level PyGIL... functions.'


    Quite the contrary, I would say!

    From the Py C API doc, I interpret the PyGIL... functions to be higher
    leveled than eg. PyEval_SaveThread(). I've checked into the source of
    python to find out what's really going on, and I've learnt a couple of
    interesting facts. The PyGIL... are helper functions build over the
    PyEval... functions, not the other way around.

    A real beauty about the PyGILState_Ensure() is the fact that if it's
    called from a thread that is (yet) unknown to python, it will actually
    create the thread state object for you (with PyThreadState_New) and
    acquire the lock. Hence, I dont have to worry about the process of
    swapping thread states with PyEval_xxxThread() functions. I ONLY need to
    use PyGILState_Ensure() prior to any py-ops.

    From my previous post, I asked about the difference between
    PyEval_ReleaseLock() and PyEval_ReleaseThread() and I've found one major
    difference. Both release the GIL as the docs states, but the
    PyEval_ReleaseThread() saves the current thread state as well. When I
    used PyEval_ReleaseLock() this didnt happen, preventing proper saving of
    the current thread, causing py crash when control were handed back to
    this thread.

    By using:

    PyEval_InitThreads();
    py_ops();
    PyThreadState *pts = PyGILState_GetThisThreadState();
    PyEval_ReleaseThread(pts);

    And for each py-op later on (from arbitrary thread):

    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();
    py_ops();
    PyGILState_Release(gstate);

    Then you're home free. Everything related to threading is handled by
    Py/API itself.

    > Probably you just do a PyRun_xxxx in main thread and then everything
    > else in Python, and expose C-parts for the thread-loop to Python as
    > function (in other_py_inits) - where in the c-function you probably have
    > the usual Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS bracket during
    > time consuming/system/reenter-endangered message stuff.


    The app i'm building is a plugin to a server system, and I cant control
    nor remove any threads that the server uses. I am given one main thread
    to run python forever, and messages that are to be delivered into python
    are called from another.

    I could do it like you propose: The data coming from my server will
    arrive into a c-function called by a server thread. The c-part of the
    thread-loop would then be run as another thread (started from python).
    Its fully feasible, yet the challenge is to make proper data sync-ing
    between these two threads.

    Well, I think I've achieved what I wanted. Python seems apparently
    stable, and reading from the python sources, I cant see any immediate
    reasons why it shouldn't.

    Thanks for letting me closer to a working solution!


    Regads,
    Svein
     
    Svein Seldal, Nov 16, 2006
    #9
  10. Svein Seldal

    robert Guest

    Svein Seldal wrote:

    > A real beauty about the PyGILState_Ensure() is the fact that if it's
    > called from a thread that is (yet) unknown to python, it will actually
    > create the thread state object for you (with PyThreadState_New) and
    > acquire the lock. Hence, I dont have to worry about the process of
    > swapping thread states with PyEval_xxxThread() functions. I ONLY need to
    > use PyGILState_Ensure() prior to any py-ops.


    Interesting - the PyGILState_Ensure was new to py2.3. I didn't notice so far, that it will even auto-create a fresh tread state (from default InterpreterState only). From the docs its not that clear.

    The confusion in usage is that one has to do

    Py_Initialize()
    PyEval_InitThreads()
    ts=PyEval_SaveThread()

    in order to start clean for further pure "meta-high-level" PyGILState_Ensure/Release.

    They obviously meant PyGILState_Ensure to make live easier, but the discussion shows, it adds additional worries probably forcing each user into investigating the Python C-code :)

    There should probably a extra init function for this meta-high-level usage like ..

    PyGILState_PyInitThreads() // which would not pre-acquire the lock! - as anyone expects to enter then with PyGILState_Ensure

    .. and these 3 Functions grouped together in the docs for an easy-to-use meta-high-level thread state API.

    The current thread state API doc, as you read it from top to bottom now, is in fact totally confusing for anyone who didn't develop Python himself :)


    Robert
     
    robert, Nov 17, 2006
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Fernando Rodriguez
    Replies:
    6
    Views:
    430
    Nick Vargish
    Nov 20, 2003
  2. Praveen, Tayal (IE10)
    Replies:
    0
    Views:
    398
    Praveen, Tayal (IE10)
    Mar 17, 2005
  3. Leonard J. Reder

    Multithreaded Python FSM (Harel State Machines)

    Leonard J. Reder, Jun 9, 2005, in forum: Python
    Replies:
    3
    Views:
    896
    Leonard J. Reder
    Jun 18, 2005
  4. Sidd
    Replies:
    7
    Views:
    1,335
  5. Replies:
    2
    Views:
    332
Loading...

Share This Page