Need some help with Python/C api and threading

[Steve Menard]

Here is my problem.

I have this library thats hosts another language within python, and
allows that language to call back INTO python.

All is good as long as the other languages calls back on the same
thread. If the callback arrives on a different thread, all hell break
loose and the program dies horribly.

looking at the C api documentation, I came upon the following block of
code :

PyThreadState *tstate;
PyObject *result;

/* interp is your reference to an interpreter object. */
tstate = PyThreadState_New(interp);
PyEval_AcquireThread(tstate);

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result */

/* Release the thread. No Python API allowed beyond this point. */
PyEval_ReleaseThread(tstate);

/* You can either delete the thread state, or save it
until you need it the next time. */
PyThreadState_Delete(tstate);


Which would seem to be what I need. However, I have no idea how to get
at that interp pointer. I tried the following :

PyInterpreterState* interp = PyInterpreterState_New();
PyThreadState *tstate = PyThreadState_New(interp);
PyEval_AcquireThread(tstate);

but then it crashes on the second line ...

Anybody ever done this? As a side note, the hosted language can start an
arbitrary number of threads ...

Steve

 

[Les Smithson]

Steve> Here is my problem. I have this library thats hosts
Steve> another language within python, and allows that language to
Steve> call back INTO python.

Steve> All is good as long as the other languages calls back on
Steve> the same thread. If the callback arrives on a different
Steve> thread, all hell break loose and the program dies horribly.

Steve> looking at the C api documentation, I came upon the
Steve> following block of code :

Steve> PyThreadState *tstate; PyObject *result;

Steve> /* interp is your reference to an interpreter
Steve> object. */ tstate = PyThreadState_New(interp);
Steve> PyEval_AcquireThread(tstate);

Steve> /* Perform Python actions here. */ result =
Steve> CallSomeFunction(); /* evaluate result */

Steve> /* Release the thread. No Python API allowed beyond
Steve> this point. */ PyEval_ReleaseThread(tstate);

Steve> /* You can either delete the thread state, or save it
Steve> until you need it the next time. */
Steve> PyThreadState_Delete(tstate);


Steve> Which would seem to be what I need. However, I have no idea
Steve> how to get at that interp pointer. I tried the following :

Steve> PyInterpreterState* interp =
Steve> PyInterpreterState_New(); PyThreadState *tstate =
Steve> PyThreadState_New(interp); PyEval_AcquireThread(tstate);

Steve> but then it crashes on the second line ...

Steve> Anybody ever done this? As a side note, the hosted language
Steve> can start an arbitrary number of threads ...

Steve> Steve

I haven't done this for a while and I'm a little hazy on it, so this
may be incorrect:

I used 'PyThreadState *ts = Py_NewInterpreter();' to set a new
sub-interpreter state if called in a new thread.

If the embedded script calls back into the extension, it restores that
thread state and acquires the GIL before making any other Py* calls by
calling 'PyEval_RestoreThread(ts);'. Before returning, it calls
'PyEval_SaveThread()'.

 

[Steve Menard]

I haven't done this for a while and I'm a little hazy on it, so this
may be incorrect:

I used 'PyThreadState *ts = Py_NewInterpreter();' to set a new
sub-interpreter state if called in a new thread.

If the embedded script calls back into the extension, it restores that
thread state and acquires the GIL before making any other Py* calls by
calling 'PyEval_RestoreThread(ts);'. Before returning, it calls
'PyEval_SaveThread()'.

Thanks, however I dont think thid will work. The doc for
Py_NewInterpreter says that it created a "an (almost) totally separate
environment for the execution of Python code. In particular, the new
interpreter has separate, independent versions of all imported modules".
This is not good for me, as the callbacks must come in the "main"
interpreter context.

Is there a tutorial somewhere? Or a particularly well written extension
module whose source code I could take a look at?

Let me summarize my situation :

I am writing a python extension, not embedding python. As such, I have
no control over the interpreter, or the threads.

The library I am embedding is not of my own writing. It can create any
number of threads. It can make callbacks into the Python interpreter on
any such thread.

A given thread can original either in python or the library, but control
can go back and forth : A python method can call a library method, which
in turn calls back into python, which calls a linrary method, etc ...
This is a potential problem, because trying to grab in GIL twice from
the same thread will cause a deadlock.


So far, here is what I am doing (without success).

1) In the init_XXX method, I call PyEval_InitThreads().

2) Every time I pass control to the library, I wrap the call into a
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS pair. Note that I adde this
recently, and get an error the second time Py_BEGIN_ALLOW_THREADS is
called, with the following error "Fatal Python error: PyEval_SaveThread:
NULL tstate"

3) Finally, whenever I receive a callback from the library, I added
these lines to the start and end of the method :

PyInterpreterState* interp = PyInterpreterState_New();
PyThreadState *tstate = PyThreadState_New(interp);
PyEval_AcquireThread(tstate);

and

PyEval_ReleaseThread(tstate);
PyThreadState_Delete(tstate);
PyInterpreterState_Delete(interp);


Thats about it. I am sure someone, somewhere has done what I need


Thanks for any help you can provide,

Steve

 

[Simon Dahlbacka]

Thanks, however I dont think thid will work. The doc for
Py_NewInterpreter says that it created a "an (almost) totally separate
environment for the execution of Python code. In particular, the new
interpreter has separate, independent versions of all imported modules".
This is not good for me, as the callbacks must come in the "main"
interpreter context.

Is there a tutorial somewhere? Or a particularly well written extension
module whose source code I could take a look at?

Let me summarize my situation :

I am writing a python extension, not embedding python. As such, I have
no control over the interpreter, or the threads.

The library I am embedding is not of my own writing. It can create any
number of threads. It can make callbacks into the Python interpreter on
any such thread.

A given thread can original either in python or the library, but control
can go back and forth : A python method can call a library method, which
in turn calls back into python, which calls a linrary method, etc ...
This is a potential problem, because trying to grab in GIL twice from
the same thread will cause a deadlock.


So far, here is what I am doing (without success).

1) In the init_XXX method, I call PyEval_InitThreads().

2) Every time I pass control to the library, I wrap the call into a
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS pair. Note that I adde this
recently, and get an error the second time Py_BEGIN_ALLOW_THREADS is
called, with the following error "Fatal Python error: PyEval_SaveThread:
NULL tstate"

3) Finally, whenever I receive a callback from the library, I added
these lines to the start and end of the method :

PyInterpreterState* interp = PyInterpreterState_New();
PyThreadState *tstate = PyThreadState_New(interp);
PyEval_AcquireThread(tstate);

and

PyEval_ReleaseThread(tstate);
PyThreadState_Delete(tstate);
PyInterpreterState_Delete(interp);


Thats about it. I am sure someone, somewhere has done what I need


Thanks for any help you can provide,

would this:
http://www.python.org/peps/pep-0311.html

...be of any help?

/Simon

 

[Steve Menard]

would this:
http://www.python.org/peps/pep-0311.html

..be of any help?

/Simon

YES!!!! thats exactly it!!! thanks!

Though I gotta ask, in which version of Python was this introduced?
It'll have a big influence over the compativility list.


Steve

 

[Simon Dahlbacka]

YES!!!! thats exactly it!!! thanks!

Though I gotta ask, in which version of Python was this introduced?
It'll have a big influence over the compativility list.


Steve

...looking through the cvs logs it seems to me it should be available
from 2.3beta1

(http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Include/pystate.h)

 

[Ronald Oussoren]

If you're using Python 2.3 I'd use the API described in PEP 311:

http://www.python.org/peps/pep-0311.html


Ronald

 

[Dave Cole]

Thanks, however I dont think thid will work. The doc for
Py_NewInterpreter says that it created a "an (almost) totally separate
environment for the execution of Python code. In particular, the new
interpreter has separate, independent versions of all imported modules".
This is not good for me, as the callbacks must come in the "main"
interpreter context.

Is there a tutorial somewhere? Or a particularly well written extension
module whose source code I could take a look at?

Let me summarize my situation :

I am writing a python extension, not embedding python. As such, I have
no control over the interpreter, or the threads.

The library I am embedding is not of my own writing. It can create any
number of threads. It can make callbacks into the Python interpreter on
any such thread.

A given thread can original either in python or the library, but control
can go back and forth : A python method can call a library method, which
in turn calls back into python, which calls a linrary method, etc ...
This is a potential problem, because trying to grab in GIL twice from
the same thread will cause a deadlock.


So far, here is what I am doing (without success).

1) In the init_XXX method, I call PyEval_InitThreads().

2) Every time I pass control to the library, I wrap the call into a
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS pair. Note that I adde this
recently, and get an error the second time Py_BEGIN_ALLOW_THREADS is
called, with the following error "Fatal Python error: PyEval_SaveThread:
NULL tstate"

3) Finally, whenever I receive a callback from the library, I added
these lines to the start and end of the method :

PyInterpreterState* interp = PyInterpreterState_New();
PyThreadState *tstate = PyThreadState_New(interp);
PyEval_AcquireThread(tstate);

and

PyEval_ReleaseThread(tstate);
PyThreadState_Delete(tstate);
PyInterpreterState_Delete(interp);


Thats about it. I am sure someone, somewhere has done what I need

In the code for the Sybase extension module I ended up with some code to
manage the GIL and a lock per database context and database connection.

http://www.object-craft.com.au/projects/sybase/download.html

The code is a little complicated in that you can disable all locking
support at compile time if it is not important to you.

- Dave