How to call python from a foreign language thread (C++)

[Victor Lin]

Hi,

I am developing a program that use DirectShow to grab audio data from
media files. DirectShow use thread to pass audio data to the callback
function in my program, and I let that callback function call another
function in Python.

I use Boost.Python to wrapper my library, the callback function :

class PythonCallback {
private:
object m_Function;
public:
PythonCallback(object obj)
: m_Function(obj)
{}

void operator() (double time, const AudioData &data) {
// Call the callback function in python
m_Function(time, data);
}
};

Here comes the problem, a thread of DirectShow calls my
PythonCallback, namely, call the function in Python. Once it calls, my
program just crash. I found this should be threading problem. Then I
found this document:

http://docs.python.org/c-api/init.html

It seems that my program can't call to Python's function from thread
directly, because there is Global Interpreter Lock. The python's GIL
is so complex, I have no idea how it works. I'm sorry, what I can do
is to ask. My question is. What should I do before and after I call a
Python function from threads?

It may looks like this.

void operator() (double time, const AudioData &data) {
// acquire lock
m_Function(time, data);
// release lock
}

Thanks.
Victor Lin.

 

[Philip Semanchuk]

It seems that my program can't call to Python's function from thread
directly, because there is Global Interpreter Lock. The python's GIL
is so complex, I have no idea how it works. I'm sorry, what I can do
is to ask. My question is. What should I do before and after I call a
Python function from threads?

Hi Victor,
I asked a similar question yesterday and have gotten no response yet
-- I hope you'll have better luck. I'm writing a C extension that
wants to implement a callback in a new thread. Here's what I think
I've learned from reading the threads API doc. Please note that this
is a classic case of the blind leading the blind! I'm sure some (most?
all?) of the ideas below are wrong, but I'm hoping that thinking
through some of this "out loud" will help both of us. Or maybe some
more knowledgeable person will take pity on/be appalled by my
ignorance and come to our rescue. =)

Python's infamous GIL doesn't exist when a program is single-threaded.
Before a new thread is created, the main thread must call
PyEval_InitThreads() to create the GIL. However, "It is not safe to
call this function when it is unknown which thread (if any) currently
has the global interpreter lock." Therefore my extension must do this:

i_have_the_gil = 0;
if (!PyEval_ThreadsInitialized()) {
PyEval_InitThreads();
/* "...when this function initializes the lock, it also acquires
it." */
i_have_the_gil = 1;
}

That ensures that the GIL is created.

My extension will be calling from a newly-created C thread. The
*Python* thread doesn't exist yet; so next I have to create it.
Therefore --

if (!i_have_the_gil)
PyEval_AcquireLock();

// Might not actually be the main thread but conceptually
// it is OK to assume so here.
main_thread = PyThreadState_Get();

callback_thread = PyThreadState_New(main_thread->interp);

PyThreadState_Swap(callback_thread);

gstate = PyGILState_Ensure();

call_callback_function();

// Now unwind the above

PyGILState_Release(gstate);

PyThreadState_Swap(main_thread);

PyThreadState_Clear(callback_thread);

PyThreadState_Delete(callback_thread);

PyEval_ReleaseLock();


I haven't yet tested this! But hopefully it is the right idea and just
needs a little fine tuning.

I found this discussion useful:
http://mail.python.org/pipermail/python-list/2006-November/413088.html

It includes the quote, "The current thread state API doc, as you read
it from top to bottom now, is in fact totally confusing for anyone who
didn't develop Python himself"

I like! =)

Cheers
Philip

 

[Victor Lin]

Hi Victor,
I asked a similar question yesterday and have gotten no response yet
-- I hope you'll have better luck. I'm writing a C extension that
wants to implement a callback in a new thread. Here's what I think
I've learned from reading the threads API doc. Please note that this
is a classic case of the blind leading the blind! I'm sure some (most?
all?) of the ideas below are wrong, but I'm hoping that thinking
through some of this "out loud" will help both of us. Or maybe some
more knowledgeable person will take pity on/be appalled by my
ignorance and come to our rescue. =)

Python's infamous GIL doesn't exist when a program is single-threaded.
Before a new thread is created, the main thread must call
PyEval_InitThreads() to create the GIL. However, "It is not safe to
call this function when it is unknown which thread (if any) currently
has the global interpreter lock." Therefore my extension must do this:

i_have_the_gil = 0;
if (!PyEval_ThreadsInitialized()) {
PyEval_InitThreads();
/* "...when this function initializes the lock, it also acquires
it." */
i_have_the_gil = 1;

}

That ensures that the GIL is created.

My extension will be calling from a newly-created C thread. The
*Python* thread doesn't exist yet; so next I have to create it.
Therefore --

if (!i_have_the_gil)
PyEval_AcquireLock();

// Might not actually be the main thread but conceptually
// it is OK to assume so here.
main_thread = PyThreadState_Get();

callback_thread = PyThreadState_New(main_thread->interp);

PyThreadState_Swap(callback_thread);

gstate = PyGILState_Ensure();

call_callback_function();

// Now unwind the above

PyGILState_Release(gstate);

PyThreadState_Swap(main_thread);

PyThreadState_Clear(callback_thread);

PyThreadState_Delete(callback_thread);

PyEval_ReleaseLock();

I haven't yet tested this! But hopefully it is the right idea and just
needs a little fine tuning.

I found this discussion useful:http://mail.python.org/pipermail/python-list/2006-November/413088.html

It includes the quote, "The current thread state API doc, as you read
it from top to bottom now, is in fact totally confusing for anyone who
didn't develop Python himself"

I like! =)

Cheers
Philip

Hi Philip,

It does not work. But however, thanks your help. I have tired so many
methods to do. But it
crash...crash..deadlock...deadlock..crash...crash... I have no any
tried success. I am going crazy. Could someone help me, thanks.

 

[Mark Hammond]

It may looks like this.

void operator() (double time, const AudioData&data) {
// acquire lock
m_Function(time, data);
// release lock
}

You want something like:

void operator() (double time, const AudioData&data) {
PyGILState_STATE old = PyGILState_Acquire();
m_Function(time, data);
PyGILState_Release(old);

Note that Phillip is also correct; you will need to initialize threading
if this is an app which embeds Python...

HTH,

Mark

 

[Mark Hammond]

PyGILState_STATE old = PyGILState_Acquire();

Make that PyGILState_Ensure();

Mark

 

[Philip Semanchuk]

It does not work. But however, thanks your help. I have tired so many
methods to do. But it
crash...crash..deadlock...deadlock..crash...crash... I have no any
tried success. I am going crazy. Could someone help me, thanks.

Hi Victor,
I have some code that works, although I'm not terribly confident of
it. The Python documentation in this area is as clear as mud. I'd be
interested to see if my code works for you as well.

This is part of my function that the Python code calls to set up the
callback:

if (!PyEval_ThreadsInitialized()) {
DPRINTF("calling PyEval_InitThreads()\n");
PyEval_InitThreads();
// PyEval_InitThreads() acquires the GIL on my behalf but
// I don't want it at the moment.
PyEval_ReleaseLock();
}

This sets up the GIL if necessary (i.e. if this is a single-threaded
program) and is a no-op otherwise (i.e. if the app has already created
a Python thread).

Then I have this function to perform the callback. It is invoked in a
new C thread. Comments are inline.

void process_notification(union sigval notification_data) {
/* Invoked by the system in a new thread as notification of a
message
arriving in the queue. */
PyObject *arglist;
PyObject *result;
PyGILState_STATE gstate;
PyThreadState *main_thread;
PyThreadState *callback_thread;
MessageQueue *self = notification_data.sival_ptr;

DPRINTF("C thread %ld invoked\n", pthread_self());

// PyGILState_Ensure() implicitly acquires the GIL so I don't need
// to call PyEval_AcquireLock().
DPRINTF("Calling PyGILState_Ensure()\n");
gstate = PyGILState_Ensure();

// Get the current thread state so that I have an interpreter to
// which to point.
DPRINTF("Calling PyThreadState_Get()\n");
main_thread = PyThreadState_Get();

// Create a new Python thread for the callback.
DPRINTF("Calling PyThreadState_New()\n");
callback_thread = PyThreadState_New(main_thread->interp);

// Make the callback thread current.
DPRINTF("Calling PyThreadState_Swap()\n");
PyThreadState_Swap(callback_thread);

// Perform the callback.
arglist = Py_BuildValue("(O)", self->notification_function_param);
result = PyEval_CallObject(self->notification_function, arglist);
Py_DECREF(arglist);

DPRINTF("Done calling\n");

// Clean up my internal pointers
Py_XDECREF(self->notification_function);
Py_XDECREF(self->notification_function_param);
self->notification_function = NULL;
self->notification_function_param = NULL;

// Now unwind the Python thread/GIL stuff above
DPRINTF("Calling PyThreadState_Swap()\n");
PyThreadState_Swap(main_thread);

DPRINTF("Calling PyThreadState_Clear()\n");
PyThreadState_Clear(callback_thread);

DPRINTF("Calling PyThreadState_Delete()\n");
PyThreadState_Delete(callback_thread);

// PyGILState_Ensure() acquires the lock, but does
PyGILState_Release()
// release it? The documentation doesn't say, but it seems like
it does.
DPRINTF("Calling PyGILState_Release()\n");
PyGILState_Release(gstate);

DPRINTF("exiting thread\n");
};



This code works (in my limited testing) regardless of whether or not
the Python code has created a thread. For the threaded test, I created
a background thread that prints "ding!" every second. That thread
continued to run even after my callback thread was invoked which I
assume means that I released the GIL properly.

As I mentioned before, this is part of my posix_ipc extension. This
code (assuming I feel confident enough to release it) will be in the
next version that should be out soon, so you will have a full working
example with which to experiment.

HTH,
Philip