problem to write a THREAD enabled python extension

Discussion in 'Python' started by Andreas Otto, Aug 24, 2009.

  1. Andreas Otto

    Andreas Otto Guest

    Hi,

    the following scenario:

    1. using GIL
    2. a pthread is created in a library and have to be "announced" to python
    3. the thread is created in a "PyObject_CallObject" function call from
    extension c code using a other extension c-code function called
    from python code

    python CRASH with invalid thread-state object
    http://www.linuxjournal.com/article/3641
    has information that GIL is NOT able to keep thread state information
    for cross thread operation.

    workaround ->
    .......
    PyGILState_STATE gilState = PyGILState_Ensure(); // PyEval_RestoreThread
    // Call Python/C API...
    PyGILState_Release(gilState); // PyEval_SaveThread
    .......
    // Create new thread...
    .......
    PyGILState_STATE gilState = PyGILState_Ensure(); // PyEval_RestoreThread
    // Call Python/C API...
    PyGILState_Release(gilState); // PyEval_SaveThread
    .......

    the problem is that PyGILState_Release have to be done in the call of
    "PyObject_CallObject" and not in the extension c-code

    -> for now i call it impossible to provide thread support if thread is
    created in extension code

    -> more info by valgrind
    ==29105==
    ==29105== Invalid read of size 4
    ==29105== at 0x4F099AB: PyEval_EvalFrameEx (ceval.c:3728)
    ==29105== by 0x4F0B476: PyEval_EvalCodeEx (ceval.c:3180)
    ==29105== by 0x4E9DE50: function_call (funcobject.c:630)
    ==29105== by 0x4E7610C: PyObject_Call (abstract.c:2160)
    ==29105== by 0x4E8BEFB: method_call (classobject.c:323)
    ==29105== by 0x4E7610C: PyObject_Call (abstract.c:2160)
    ==29105== by 0x4F04975: PyEval_CallObjectWithKeywords (ceval.c:3624)
    ==29105== by 0x72132DF: pymsgque_ProcCall (misc_python.c:48)
    ==29105== by 0x742A23E: pTokenInvoke (token.c:327)
    ==29105== by 0x742E7E2: sMqEventStart (msgque.c:908)
    ==29105== by 0x7425365: pEventStart (event.c:360)
    ==29105== by 0x742ECBF: MqProcessEvent (msgque.c:1081)
    ==29105== by 0x7215F24: pymsgque_ProcessEvent (context_python.c:176)
    ==29105== by 0x4F09BFA: PyEval_EvalFrameEx (ceval.c:3961)
    ==29105== by 0x4F0B476: PyEval_EvalCodeEx (ceval.c:3180)
    ==29105== by 0x4F0B58A: PyEval_EvalCode (ceval.c:650)
    ==29105== by 0x4F2EE29: PyRun_FileExFlags (pythonrun.c:1697)
    ==29105== by 0x4F2F134: PyRun_SimpleFileExFlags (pythonrun.c:1182)
    ==29105== by 0x4F43CFB: Py_Main (main.c:625)
    ==29105== by 0x400D11: main (python.c:152)
    ==29105== Address 0x24 is not stack'd, malloc'd or (recently) free'd


    -> extension code ... (including some debugging output)

    M0
    PyGILState_STATE gstate = PyGILState_Ensure();
    PyObject *result;
    PyObject * const self = (PyObject*) CONTEXT->self;
    PyObject * const callable = (PyObject*) dataP;
    enum MqErrorE ret = MQ_OK;

    // clean Python and libmsgque error
    PyErr_Clear();
    MqErrorReset(msgque->error);

    // call the function
    M1
    printP(PyThreadState_Get())
    if (PyMethod_Check(callable) && PyMethod_Self(callable) == self) {
    //PyGILState_Release(gstate);
    result = PyObject_CallObject(callable, NULL);
    //gstate = PyGILState_Ensure();
    } else {
    //PyGILState_Release(gstate);
    result = PyObject_CallFunctionObjArgs(callable, self, NULL);
    //gstate = PyGILState_Ensure();
    }
    M2
    printP(PyThreadState_Get())
    Py_XDECREF(result);

    // no error return OK
    if (PyErr_Occurred() != NULL) {
    NS(ErrorSet) (self);
    ret = MqErrorGetCode(msgque->error);
    }
    PyGILState_Release(gstate);
    return ret;
    }

    with GIL wrapper 'PyGILState_Release' and 'PyGILState_Ensure' enabled for
    'PyObject_CallObject' the following SEG happen

    if (PyMethod_Check(callable) && PyMethod_Self(callable) == self) {
    PyGILState_Release(gstate);
    result = PyObject_CallObject(callable, NULL);
    gstate = PyGILState_Ensure();
    } else {
    PyGILState_Release(gstate);
    result = PyObject_CallFunctionObjArgs(callable, self, NULL);
    gstate = PyGILState_Ensure();
    }
    pymsgque_ProcCall(misc_python.c:32) -> 00000000000000000
    pymsgque_ProcCall(misc_python.c:44) -> 11111111111111111
    pymsgque_ProcCall(misc_python.c:45) -> PyThreadState_Get()<0x68c2ad8>
    ==29223==
    ==29223== Thread 2:
    ==29223== Invalid read of size 4
    ==29223== at 0x4E760EA: PyObject_Call (abstract.c:2158)
    ==29223== by 0x4F04975: PyEval_CallObjectWithKeywords (ceval.c:3624)
    ==29223== by 0x72132F7: pymsgque_ProcCall (misc_python.c:48)
    ==29223== by 0x7430092: MqLinkCreate (msgque.c:644)
    ==29223== by 0x7430AB0: MqDefaultLinkCreate (msgque.c:681)
    ==29223== by 0x742F73B: MqLinkCreate (msgque.c:346)
    ==29223== by 0x7432C3D: sSysServerThreadCreate (sys.c:502)
    ==29223== by 0x521606F: start_thread (in /lib64/libpthread-2.9.so)
    ==29223== by 0x5B5B10C: clone (in /lib64/libc-2.9.so)
    ==29223== Address 0x18 is not stack'd, malloc'd or (recently) free'd
    ==29223==
    ==29223== Process terminating with default action of signal 11 (SIGSEGV)


    mfg

    aotto1968
     
    Andreas Otto, Aug 24, 2009
    #1
    1. Advertising

  2. Andreas Otto wrote:
    > Hi,
    >
    > the following scenario:
    >
    > 1. using GIL
    > 2. a pthread is created in a library and have to be "announced" to python
    > 3. the thread is created in a "PyObject_CallObject" function call from
    > extension c code using a other extension c-code function called
    > from python code
    >
    > python CRASH with invalid thread-state object


    You forgot to create a thread state for the new thread. See the
    PyThreadState_New() function.

    Stefan
     
    Stefan Behnel, Aug 24, 2009
    #2
    1. Advertising

  3. Andreas Otto

    Andreas Otto Guest

    Stefan Behnel wrote:

    >
    > You forgot to create a thread state for the new thread. See the
    > PyThreadState_New() function.
    >


    this does not really solve the problem even if the original error does no
    happen anymore ...

    but a bucket of other error happen >>> sometimes <<< and this mean
    that time and (I call it so) internal state stuff has influence on the
    problem and create errors like:

    1. http://mail.python.org/pipermail/python-list/2004-January/242740.html
    2. various PyThreadState_Get -> PyThreadState_Get: no current thread
    3. sometimes ThreadState has NULL sometimes not and NULL create an
    CORE for "tstate" pointer access operations
    4. ... a lot of other problems too

    all problems together I come to the conclusion:

    1. I spend ~ 1week for a job done in C# or java in a day - and have no
    usable solution
    2. python need a couple of 100+ C API function for the same job
    java need 10+
    3. the design on the C-API is amateurish -> significant parts are
    implemented as GLOBAL variables
    4. the thread support is not more than a joke -> go to 3
    5. after pay a lot of effort to get something working the final solution
    is significant slower than C# or JAVA
    6. you can not truss any-thing in python because you never know
    if under some circumstance you get a CORE

    the resulting PYTHON code is usable but not significant more readable
    as JAVA, C* or even C++ code


    mfg

    aotto1968
     
    Andreas Otto, Aug 25, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gabor
    Replies:
    3
    Views:
    627
    Jacob Yang [MSFT]
    Aug 26, 2003
  2. Anand
    Replies:
    3
    Views:
    899
    Tim Daneliuk
    Nov 8, 2003
  3. Christian Seberino
    Replies:
    3
    Views:
    1,190
    Christian Seberino
    Feb 5, 2004
  4. Andreas Otto
    Replies:
    0
    Views:
    308
    Andreas Otto
    Oct 12, 2009
  5. Andreas Otto

    problem with perl + thread + extension

    Andreas Otto, Oct 12, 2009, in forum: Perl Misc
    Replies:
    6
    Views:
    136
    Andreas Otto
    Oct 13, 2009
Loading...

Share This Page