Endless GIL and thread confusion

[enska]

Can someone clarify the steps needed to make access to the interpreter
safe from multiple threads?

I've been reading the docs for days and I still find them very confusing
and misleading. For example does the PyGILState_Ensure() function lock the
GIL or just create the thread state? Is thread supposed to call it once on
creation and then the release function before exiting or is this the
preferred mechanism to handle the locking? Most of the confusion stems
from this segment in the documentation.

"...when threads are created from C, they don't have the global
interpreter lock, nor is there a thread state data structure for them.
Such threads must bootstrap themselves into existence, by first creating a
thread state data structure, then acquiring the lock, and finally storing
their thread state pointer, before they can start using the Python/C API.
When they are done, they should reset the thread state pointer, release
the lock, and finally free their thread state data structure.

Beginning with version 2.3, threads can now take advantage of the
PyGILState_*() functions to do all of the above automatically."


Anyway, currently my code looks like this:

void foobar(...)
{
PyGILState state = PyGILState_Ensure();

// make python calls

PyGILState_Release(state);
}

void blablah(...)
{
PyGILState state = PyGILState_Ensure();

// make python calls

PyGILState_Release(state);
}

My python Initialization code looks like this

void init_python(char* progname, char* bindir)
{
Py_SetProgramName(progname);
PyEval_InitThreads();
Py_InitializeEx(0);

char* argv[] = {progname, bindir};
PySys_SetArgv(2, argv);
}

calling foobar() or blablah() from the main thread works as expected, but
if the second thread calls them it locks up in the call to
PyGILState_Ensure().

I have tried adding a call to PyEval_ReleaseLock() in the init_python
function to make sure that the main thread is not holding the GIL, but
this causes the application to segfault when it is calling Py_Finalize()
so it is clear that this is not correct.

Please advice.

Thanks.

 

[kyosohma]

Can someone clarify the steps needed to make access to the interpreter
safe from multiple threads?

I've been reading the docs for days and I still find them very confusing
and misleading. For example does the PyGILState_Ensure() function lock the
GIL or just create the thread state? Is thread supposed to call it once on
creation and then the release function before exiting or is this the
preferred mechanism to handle the locking? Most of the confusion stems
from this segment in the documentation.

"...when threads are created from C, they don't have the global
interpreter lock, nor is there a thread state data structure for them.
Such threads must bootstrap themselves into existence, by first creating a
thread state data structure, then acquiring the lock, and finally storing
their thread state pointer, before they can start using the Python/C API.
When they are done, they should reset the thread state pointer, release
the lock, and finally free their thread state data structure.

Beginning with version 2.3, threads can now take advantage of the
PyGILState_*() functions to do all of the above automatically."

Anyway, currently my code looks like this:

void foobar(...)
{
PyGILState state = PyGILState_Ensure();

// make python calls

PyGILState_Release(state);

}

void blablah(...)
{
PyGILState state = PyGILState_Ensure();

// make python calls

PyGILState_Release(state);

}

My python Initialization code looks like this

void init_python(char* progname, char* bindir)
{
Py_SetProgramName(progname);
PyEval_InitThreads();
Py_InitializeEx(0);

char* argv[] = {progname, bindir};
PySys_SetArgv(2, argv);

}

calling foobar() or blablah() from the main thread works as expected, but
if the second thread calls them it locks up in the call to
PyGILState_Ensure().

I have tried adding a call to PyEval_ReleaseLock() in the init_python
function to make sure that the main thread is not holding the GIL, but
this causes the application to segfault when it is calling Py_Finalize()
so it is clear that this is not correct.

Please advice.

Thanks.

I use the Threading module whenever I need to use threads. It works
quite nicely and I have yet to have any problems with it, except for a
little goofiness with WMI that was explained to me long ago. You might
check that module out.

The wiki for wxPython actually has a good tutorial on threads:

http://wiki.wxpython.org/LongRunningTasks

And here's an article on usurping the GIL:

http://www.pyzine.com/Issue001/Section_Articles/article_ThreadingGlobalInterpreter.html

Hope that helps a little.

Mike

 

[Aahz]

Can someone clarify the steps needed to make access to the interpreter
safe from multiple threads?

I've been reading the docs for days and I still find them very confusing
and misleading. For example does the PyGILState_Ensure() function lock the
GIL or just create the thread state? Is thread supposed to call it once on
creation and then the release function before exiting or is this the
preferred mechanism to handle the locking? Most of the confusion stems
from this segment in the documentation.

If you haven't already, try (e-mail address removed).

 

[Aahz]

I use the Threading module whenever I need to use threads. It works
quite nicely and I have yet to have any problems with it, except for a
little goofiness with WMI that was explained to me long ago. You might
check that module out.

The wiki for wxPython actually has a good tutorial on threads:

http://wiki.wxpython.org/LongRunningTasks

And here's an article on usurping the GIL:

http://www.pyzine.com/Issue001/Section_Articles/article_ThreadingGlobalInterpreter.html

Unfortunately, that's all only relevant to someone writing Python code
(and I'd forgotten about that article!) -- the OP needs advice with C.