segfault when calling Python from C thread

[Travis Berg]

I'm running into a problem when trying to perform a callback to a Python
function from a C extension. Specifically, the callback is being made by
a pthread that seems to cause the problem. If I call the callback from
the parent process, it works fine. The PyObject is static, and holds the
same value in both Parent and thread, so I'm at a loss as to what would be
different in the pthread from the parent that would cause a segfault on
the callback. The machine specifics are an x86 intel processor with
RedHat linux.

Here is some clips of the C callback code showing how I'm storing the
callback, then what the actual callback function is like. Any ideas?
The function being called is a simply to display the string of text, and
execution never seems to reach back to the Python code at all.

Thanks,
Travis B.


/* callback function to the Python code */
static PyObject * my_callback = NULL;

/* setting callback function */
static PyObject * my_set_callback(PyObject *dummy, PyObject *args)
{
PyObject *result = NULL;
PyObject *temp;
PyObject *arglist;

if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
if (!PyCallable_Check(temp)) {
PyErr_SetString(PyExc_TypeError,
"parameter must be callable");
return NULL;
}
Py_XINCREF(temp); /* Add a reference to new callback */
Py_XDECREF(my_callback); /* Dispose of previous callback */
my_callback = temp; /* Remember new callback */
/* return "None" */
Py_INCREF(Py_None);
result = Py_None;
}
return result;
}

/* calling callback */
void callback(char * str) {
PyObject *arglist;
PyObject *result;
if(str == NULL)
return;

if(my_callback == NULL) {
printf("no callback function provided, returning...\n");
return;
}

/* Time to call the callback */
arglist = Py_BuildValue("(s)", str);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if(result == NULL)
return;
Py_DECREF(result);
}

 

[Greg Chapman]

I'm running into a problem when trying to perform a callback to a
Python function from a C extension. Specifically, the callback is
being made by a pthread that seems to cause the problem. If I call
the callback from the parent process, it works fine. The PyObject is
static, and holds the same value in both Parent and thread, so I'm at
a loss as to what would be different in the pthread from the parent
that would cause a segfault on the callback. The machine specifics
are an x86 intel processor with RedHat linux.


/* calling callback */
void callback(char * str) {
PyObject *arglist;
PyObject *result;
if(str == NULL)
return;

if(my_callback == NULL) {
printf("no callback function provided, returning...\n");
return;
}

/* Time to call the callback */
arglist = Py_BuildValue("(s)", str);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if(result == NULL)
return;
Py_DECREF(result);
}

Your callback function needs to hold the Python GIL (and have a vaild
threadstate) before it calls any Python C-API functions. Change the
last part of it to:

PyGILState_STATE state;

/* ... */

/* Time to call the callback */

state = PyGILState_Ensure();

arglist = Py_BuildValue("(s)", str);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if(result == NULL)
return;
Py_DECREF(result);

PyGILState_Release(state);
}

Also, somewhere in your main thread you should call PyEval_InitThreads
before any of the callback threads execute. (This call is made
automatically if you are creating new threads using Python's thread
module, but if the new threads are created by some C code, you need to
call it yourself.)

 

[Fredrik Lundh]

Your callback function needs to hold the Python GIL (and have a vaild
threadstate) before it calls any Python C-API functions. Change the
last part of it to:

PyGILState_STATE state;

/* ... */

/* Time to call the callback */

state = PyGILState_Ensure();

arglist = Py_BuildValue("(s)", str);
result = PyEval_CallObject(my_callback, arglist);
Py_DECREF(arglist);
if(result == NULL)
return;
Py_DECREF(result);

PyGILState_Release(state);
}

you might wish to make sure you release the GIL even if the callback
raises an exception...

</F>

 

[Greg Chapman]

you might wish to make sure you release the GIL even if the callback
raises an exception...

</F>

Argh, thanks for catching that. You probably put that too politely
though (in case anyone sees this who might think that is optional): one
should absolutely make sure all code paths which call PyGILState_Ensure
have a matching call to PyGILState_Release.