per interpreter storage for C extensions

[Robin Becker]

As part of some django usage I need to get some ReportLab C extensions into a
state where they can be safely used with mod_python.

Unfortunately we have C code that seems incompatible with mod_python and I'm
looking for ways to improve the situation.

Basically the main things that are wrong are all privately cached copies of
python variables. This sample code


static PyObject *_pdfmetrics_fonts = NULL;
static PyObject *_pdfmetrics_ffar = NULL;

...........
if(!_pdfmetrics_fonts){
res = PyImport_ImportModule("reportlab.pdfbase.pdfmetrics");
if(!res) ERROR_EXIT();
_o1 = _GetAttrString(res,"_fonts");
if(!_o1) ERROR_EXIT();
_o2 = _GetAttrString(res,"findFontAndRegister");
if(!_o2) ERROR_EXIT();
_pdfmetrics_fonts = _o1;
_pdfmetrics_ffar = _o2;
Py_DECREF(res); _o1 = _o2 = res = NULL;
}
if((res = PyObject_GetItem(_pdfmetrics_fonts,fontName))) return res;
............

illustrates the general problem. If called first by interpreter A I'll get the
values from interpreter A. When later called by interpreter B I'll be using the
wrong values and _pdfmetrics_ffar can alter the wrong interpreter's data.


The functions themselves are fairly small, but may be called many times so
they're worth speeding up.

Is there a simple/cheap way for C code to cache these sorts of module level
globals on a per interpreter basis? Is there even a way to tell which
interpreter I'm being called in?

Anything too costly will probably bust any speedup. Luckily I don't think we
have too many of these cached variables so it's possible we may be able to get
away with just dropping the caching for some and eliminating others.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Is there a simple/cheap way for C code to cache these sorts of module
level globals on a per interpreter basis? Is there even a way to tell
which interpreter I'm being called in?

There is no cheap way to add to the interpreter state. As Chris Mellon
explains, you can use PyThreadState_Get()->interp to find the
interpreter state. I recommend to do the caching only in the
single-interpreter case (i.e. for the first interpreter).

Notice that comparing interpreter states by identity is somewhat
dangerous: the interpreter may have been deleted with you still
holding a pointer to it, and a new interpreter may get allocated
at the same address, making you believe it is the same interpreter.

Regards,
Martin

 

[Robin Becker]

There is no cheap way to add to the interpreter state. As Chris Mellon
explains, you can use PyThreadState_Get()->interp to find the
interpreter state. I recommend to do the caching only in the
single-interpreter case (i.e. for the first interpreter).

Notice that comparing interpreter states by identity is somewhat
dangerous: the interpreter may have been deleted with you still
holding a pointer to it, and a new interpreter may get allocated
at the same address, making you believe it is the same interpreter.

.....
I can eliminate most of the problems by not keeping private pointers to
the cache variables.

What is worrying is that in the extension init we're creating an
exception and version string etc and holding a pointer to them in C; is
it safe to use the same exception in different interpeters? Currently
the exception is added to the module at init time, but my reading of the
documentation implies that all interpreters will see that exception as
it is copied from the first.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

What is worrying is that in the extension init we're creating an
exception and version string etc and holding a pointer to them in C; is
it safe to use the same exception in different interpeters?

It is safe as long as the base exception classes are also shared across
interpreters, otherwise, the hierarchical exception matching will break.
The builtin exceptions are all "global" (i.e. shared across
interpreters) (I believe).

In any case, it looks like that the "multiple interpreters" feature of
Python is just broken.

Regards,
Martin