(Win32 API) callback to Python, threading hiccups

Discussion in 'Python' started by Francois De Serres, Jul 5, 2005.

  1. Hiho,

    could somebody please enlighten me about the mechanics of C callbacks to
    Python? My domain is more specifically callbacks from the win32 API, but
    I'm not sure that's where the problem lies. Here's a description...

    I want a callback-based MIDI input/processing, so PortMidi was not an
    alternative. I have written a C extension module that links to the mmsys
    MIDI API. I separated the win32-dependant code from the Python extension
    code, so a) the module interface is system-neutral, b) the
    implementation can be tested (re-used) outside of Python. So, that code
    is tested OK, but it might be useful to sketch it's design:
    - as you may know, the MIDI input on win32 is already managed thru a
    callback mechanism; the driver calls back your program with data buffers
    - yet, I don't call directly into Python from the callback, since some
    restrictions apply on what system calls you can make from there, and I
    don't know what Python's interpreter / scripts might call.
    - so, on callback, I create a new thread, after checking that the
    previous one has returned already (WaitOnSingleObject(mythread)) so we
    only have one thread involved.
    - this is this thread that calls the user callback, yet this callback
    isn't yet a Python callable, we're still in native C code.
    - on the side of the extension module now, I merely copied the callback
    example from the Python/C API doc, and added GIL management around the call:

    static PyObject * my_callback = NULL; //this gets fixed by a
    setCallback() func
    static void external_callback(const MidiData * const data) {
    if (my_callback && (my_callback != Py_None)) {
    if (! data) {
    PyErr_SetString(PyExc_IndexError, getLastErrorMessage());
    } else {
    PyObject * arglist = NULL;
    PyObject * result = NULL;
    arglist = Py_BuildValue("(i,i,s#)", data->deviceIndex,
    data->timestamp, data->buffer, data->size);// 0, 0, "test", 4);//

    PyGILState_STATE gil = PyGILState_Ensure();
    result = PyEval_CallObject(my_callback, arglist);
    PyGILState_Release(gil);

    Py_DECREF(arglist);
    Py_DECREF(result);
    }
    }
    }

    - this one above is what is actually passed as callback to the 'native'
    C part. So, here's what (I presume) happens:
    1. the driver calls into my C code
    2. my C code spawns a thread that calls into the extension
    3. the extension calls into Python, after acquiring the GIL

    Now, here's the hiccup:
    inside a Python script, anytime a Python object is accessed by both the
    (Python) callback and the main program, I get a GPF :/

    You bet I tried to use locks, Queues and whatnot, but nothing will do,
    it seems I get protection faults on accessing... thread exclusion objects.

    Yet, there is a way where it works seamlessly: it's when the __main__
    actually spawns a threading.Thread to access the shared object. Hence I
    am lost.

    This works:

    def input_callback(msg):
    midiio.send(msg) ##MIDI thru, a call into my extension that wraps
    the implementation call with BEGIN_ALLOW_THREADS and END_...
    __main__:
    raw_input()

    This results in GPF as soon as the callback is fired:

    tape = Queue.Queue()
    def input_callback(msg):
    tape.put(msg)
    __main__:
    while True:
    print tape.get()

    This works like a charm:

    tape = Queue.Queue()
    def input_callback(msg):
    tape.put(msg)
    def job():
    while True:
    print tape.get()
    __main__:
    t = threading.Thread(target = job)
    t.start()
    raw_input()

    Note that I also get a GPF in the later case if I stop the program with
    KeyInterrupt, but I guess it's something I'll look into afterwards...

    While I can think of hiding away the added threading within a wrapper
    module, I won't sleep well untill I figure out what's really happening.
    Could someone save my precious and already sparse sleeping time, by
    pointing me to what I'm missing here?

    WinXP SP1
    Python 2.4.1
    MinGW GCC 3.4.2

    TIA,
    Francois De Serres
     
    Francois De Serres, Jul 5, 2005
    #1
    1. Advertising

  2. Francois De Serres wrote:
    > - so, on callback, I create a new thread, after checking that the
    > previous one has returned already (WaitOnSingleObject(mythread)) so we
    > only have one thread involved.


    Uh... to me, this looks like a frighteningly inefficient way of doing
    things. How about using a synchronous queue to post the data to a
    processing thread? That way, you don't have to create an entierly new
    thread each time you receive data in the callback.
     
    Christopher Subich, Jul 5, 2005
    #2
    1. Advertising

  3. Christopher Subich wrote:

    >Francois De Serres wrote:
    >
    >
    >>- so, on callback, I create a new thread, after checking that the
    >>previous one has returned already (WaitOnSingleObject(mythread)) so we
    >>only have one thread involved.
    >>
    >>

    >
    >Uh... to me, this looks like a frighteningly inefficient way of doing
    >things. How about using a synchronous queue to post the data to a
    >processing thread? That way, you don't have to create an entierly new
    >thread each time you receive data in the callback.
    >
    >

    Thanks for the tip, and sorry for frightening you. Maybe you can help as
    well with my issue?

    Francois
     
    Francois De Serres, Jul 5, 2005
    #3
  4. Francois De Serres wrote:
    > PyGILState_STATE gil = PyGILState_Ensure();
    > result = PyEval_CallObject(my_callback, arglist);
    > PyGILState_Release(gil);
    > Py_DECREF(arglist);
    > Py_DECREF(result);


    I think this should be:
    PyGILState_STATE gil = PyGILState_Ensure();
    result = PyEval_CallObject(my_callback, arglist);
    Py_DECREF(arglist);
    Py_DECREF(result);
    PyGILState_Release(gil);

    The DECREFs need to be protected, that is where storage is
    recycled and such, and you still need Python's data structures
    to do that kind of work.

    --Scott David Daniels
     
    Scott David Daniels, Jul 7, 2005
    #4
  5. Francois De Serres

    Tim Roberts Guest

    Scott David Daniels <> wrote:

    >Francois De Serres wrote:
    >> PyGILState_STATE gil = PyGILState_Ensure();
    >> result = PyEval_CallObject(my_callback, arglist);
    >> PyGILState_Release(gil);
    >> Py_DECREF(arglist);
    >> Py_DECREF(result);

    >
    >I think this should be:
    > PyGILState_STATE gil = PyGILState_Ensure();
    > result = PyEval_CallObject(my_callback, arglist);
    > Py_DECREF(arglist);
    > Py_DECREF(result);
    > PyGILState_Release(gil);
    >
    >The DECREFs need to be protected, that is where storage is
    >recycled and such, and you still need Python's data structures
    >to do that kind of work.


    I freely admit to being woefully underinformed about the GIL, but I'm
    wondering if your statement is really true. If the purpose of the GIL is
    simply to make things thread-safe, then I would have guessed that the first
    one was correct. If someone else holds a reference to "arglist", then the
    DECREF is just a nice, atomic decrement. If no one else holds a reference
    to "arglist", then it's quite safe to delete it.

    Is there more to the GIL than I'm assuming?
    --
    - Tim Roberts,
    Providenza & Boekelheide, Inc.
     
    Tim Roberts, Jul 8, 2005
    #5
  6. Francois De Serres

    Gregory Bond Guest

    Tim Roberts wrote:

    >>> PyGILState_STATE gil = PyGILState_Ensure();
    >>> result = PyEval_CallObject(my_callback, arglist);
    >>> PyGILState_Release(gil);
    >>> Py_DECREF(arglist);
    >>> Py_DECREF(result);




    > If someone else holds a reference to "arglist", then the
    > DECREF is just a nice, atomic decrement. If no one else holds a reference
    > to "arglist", then it's quite safe to delete it.


    What if anothere thread takes the GIL as soon as you release it, and is
    attempting to allcate a new opbject, and (at the same time - you might
    be on a multiprocessor!) your PyDECREF removes the last reference and
    starts freeing objects?
     
    Gregory Bond, Jul 8, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gene Heskett

    python hiccups

    Gene Heskett, Jul 14, 2004, in forum: Python
    Replies:
    0
    Views:
    327
    Gene Heskett
    Jul 14, 2004
  2. Tim Spens
    Replies:
    0
    Views:
    737
    Tim Spens
    Jun 26, 2008
  3. Tim Spens
    Replies:
    1
    Views:
    1,086
    Matimus
    Jun 27, 2008
  4. Tom Copeland

    RubyForge migration hiccups

    Tom Copeland, Oct 10, 2007, in forum: Ruby
    Replies:
    2
    Views:
    105
  5. Thomas Thomassen
    Replies:
    2
    Views:
    161
    Thomas Thomassen
    Dec 29, 2010
Loading...

Share This Page