per interpreter storage for C extensions

Discussion in 'Python' started by Robin Becker, Dec 28, 2006.

  1. Robin Becker

    Robin Becker Guest

    As part of some django usage I need to get some ReportLab C extensions into a
    state where they can be safely used with mod_python.

    Unfortunately we have C code that seems incompatible with mod_python and I'm
    looking for ways to improve the situation.

    Basically the main things that are wrong are all privately cached copies of
    python variables. This sample code


    static PyObject *_pdfmetrics_fonts = NULL;
    static PyObject *_pdfmetrics_ffar = NULL;

    ...........
    if(!_pdfmetrics_fonts){
    res = PyImport_ImportModule("reportlab.pdfbase.pdfmetrics");
    if(!res) ERROR_EXIT();
    _o1 = _GetAttrString(res,"_fonts");
    if(!_o1) ERROR_EXIT();
    _o2 = _GetAttrString(res,"findFontAndRegister");
    if(!_o2) ERROR_EXIT();
    _pdfmetrics_fonts = _o1;
    _pdfmetrics_ffar = _o2;
    Py_DECREF(res); _o1 = _o2 = res = NULL;
    }
    if((res = PyObject_GetItem(_pdfmetrics_fonts,fontName))) return res;
    ............

    illustrates the general problem. If called first by interpreter A I'll get the
    values from interpreter A. When later called by interpreter B I'll be using the
    wrong values and _pdfmetrics_ffar can alter the wrong interpreter's data.


    The functions themselves are fairly small, but may be called many times so
    they're worth speeding up.

    Is there a simple/cheap way for C code to cache these sorts of module level
    globals on a per interpreter basis? Is there even a way to tell which
    interpreter I'm being called in?

    Anything too costly will probably bust any speedup. Luckily I don't think we
    have too many of these cached variables so it's possible we may be able to get
    away with just dropping the caching for some and eliminating others.
    --
    Robin Becker
     
    Robin Becker, Dec 28, 2006
    #1
    1. Advertising

  2. Robin Becker schrieb:
    > Is there a simple/cheap way for C code to cache these sorts of module
    > level globals on a per interpreter basis? Is there even a way to tell
    > which interpreter I'm being called in?


    There is no cheap way to add to the interpreter state. As Chris Mellon
    explains, you can use PyThreadState_Get()->interp to find the
    interpreter state. I recommend to do the caching only in the
    single-interpreter case (i.e. for the first interpreter).

    Notice that comparing interpreter states by identity is somewhat
    dangerous: the interpreter may have been deleted with you still
    holding a pointer to it, and a new interpreter may get allocated
    at the same address, making you believe it is the same interpreter.

    Regards,
    Martin
     
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Dec 30, 2006
    #2
    1. Advertising

  3. Robin Becker

    Robin Becker Guest

    Martin v. Löwis wrote:
    > Robin Becker schrieb:
    >> Is there a simple/cheap way for C code to cache these sorts of module
    >> level globals on a per interpreter basis? Is there even a way to tell
    >> which interpreter I'm being called in?

    >
    > There is no cheap way to add to the interpreter state. As Chris Mellon
    > explains, you can use PyThreadState_Get()->interp to find the
    > interpreter state. I recommend to do the caching only in the
    > single-interpreter case (i.e. for the first interpreter).
    >
    > Notice that comparing interpreter states by identity is somewhat
    > dangerous: the interpreter may have been deleted with you still
    > holding a pointer to it, and a new interpreter may get allocated
    > at the same address, making you believe it is the same interpreter.

    .....
    I can eliminate most of the problems by not keeping private pointers to
    the cache variables.

    What is worrying is that in the extension init we're creating an
    exception and version string etc and holding a pointer to them in C; is
    it safe to use the same exception in different interpeters? Currently
    the exception is added to the module at init time, but my reading of the
    documentation implies that all interpreters will see that exception as
    it is copied from the first.
    --
    Robin Becker
     
    Robin Becker, Dec 30, 2006
    #3
  4. Robin Becker schrieb:
    > What is worrying is that in the extension init we're creating an
    > exception and version string etc and holding a pointer to them in C; is
    > it safe to use the same exception in different interpeters?


    It is safe as long as the base exception classes are also shared across
    interpreters, otherwise, the hierarchical exception matching will break.
    The builtin exceptions are all "global" (i.e. shared across
    interpreters) (I believe).

    In any case, it looks like that the "multiple interpreters" feature of
    Python is just broken.

    Regards,
    Martin
     
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Dec 30, 2006
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?S01aX3N0YXRl?=

    Quick one - Is SESSION per browser instance or per IP Address?

    =?Utf-8?B?S01aX3N0YXRl?=, Apr 4, 2006, in forum: ASP .Net
    Replies:
    7
    Views:
    5,890
    gerry
    Apr 10, 2006
  2. Razvan
    Replies:
    1
    Views:
    416
    tony vee
    Sep 10, 2004
  3. Replies:
    5
    Views:
    2,553
  4. Replies:
    0
    Views:
    350
  5. sarathy
    Replies:
    2
    Views:
    666
    sarathy
    Jul 17, 2006
Loading...

Share This Page