Calling into Python from a C thread

[Philip Semanchuk]

Hi all,
I'm trying call Python from inside of a C thread that's running in a
Python extension I've written and I am not having much luck. My C
thread function consists of simply this, and I get a segmentation
fault from Python:

void start_routine(union sigval foo) {
PyGILState_STATE gstate;

gstate = PyGILState_Ensure();

// Calls to Python code will go here...

PyGILState_Release(gstate);
};

I added a printf() and can see that the fault happens *after* the call
to PyGILState_Release(). Apparently I'm monkeying up something
fundamental in Python, and this is really simple so I must be missing
something big.

The context is that I'm adding support for mq_notify() to my posix_ipc
extension. A process can register via mq_notify() to have a new thread
started when a message appears in a message queue. I've got this
working when my thread code is pure C, now I'm trying to add a call to
a user-specified Python function.

Thanks
Philip

 

[Aahz]

[posted & e-mailed]

I'm trying call Python from inside of a C thread that's running in a
Python extension I've written and I am not having much luck. My C
thread function consists of simply this, and I get a segmentation
fault from Python:

Because it's been a week without response, I suggest you try the capi
mailing list.

 

[Philip Semanchuk]

[posted & e-mailed]

I'm trying call Python from inside of a C thread that's running in a
Python extension I've written and I am not having much luck. My C
thread function consists of simply this, and I get a segmentation
fault from Python:

Because it's been a week without response, I suggest you try the capi
mailing list.

I didn't know there *was* such a thing. Thanks for the tip! For those
who might be interested, the list is here:
http://mail.python.org/mailman/listinfo/capi-sig


FYI, I got my code working and it is in the latest release of posix_ipc:
http://semanchuk.com/philip/posix_ipc/

The function MessageQueue_request_notification() does some necessary
setup and the function process_notification() does the rest of the work.


Cheers
Philip

 

[Philip Semanchuk]

Let me guess. You either forgot to start Python's threading system or
you didn't register you thread with Python. You need to deal with
PyThread_init_thread(), PyEval_InitThreads(), PyThreadState_New(),
PyThreadState_Clear() and PyThreadState_DeleteCurrent(). The module
Modules/threadmodule.c contains some examples.

Yes, that's accurate except for the word "forgot". To forget something
one must first know it. =) I found the threading API documentation
difficult to follow, but I suppose that what I'm doing is a little
unusual so if it is not well-documented territory, that's what I get
for wandering off the map.

 

[Philip Semanchuk]

I see. Apparently the threading and embedding docs don't cover your
case. The docs at
http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock
and http://docs.python.org/extending/embedding.html should explain how
and when to initialize Python threads. Please report a documentation
bug at
http://bugs.python.org/.

I read that several times and eventually got my code working. I
wouldn't say that the documentation didn't cover my case (implementing
a callback). I'd say that it doesn't cover *any* cases in particular.
It's more like a list of tools for manipulating the GIL and thread
state and an explanation of the need for them rather than a tutorial
on how to use them. The latter is what I needed, but as I said, what
I'm doing is atypical and I think it is reasonable to expect thin
documentation on atypical situations.

PS: I know a great way to learn more about Python's internal threading
CAPI: fix your code, test it and attach a doc patch to your bug
report.
*hint*

I will file a bug on a least one specific point which is that
PyGILState_Ensure() acquires the GIL and PyGILState_Release() appears
to release it; the latter point is undocumented.

Cheers
Philip