Ctypes and C Infinite Callback Loops

[Thomas Dimson]

Hello,

I have quite a complex issue that is arising with regards to using
ctypes to hook into some legacy code. The legacy code is in infinite
loop - I can not touch this. It does some listening, and periodically
calls a specific callback function.

What I would like to be able to do is spawn a Python thread to handle
this infinite loop, and continue on my merry way. This works to an
extent, however if I try to raise the SystemExit exception (or any
other one) inside of this thread I get an error message of
"AssertionError: cannot join current thread".

I assume there is some issue with the global interpreter lock or that
you can't exit the infinite loop from above Python. Any suggestions on
how I can design this so the thread will be able to issue exits/raise
exceptions just like a regular thread? Is there a way of terminating
this thread from the python interpreter or ctypes.pythonapi?

I have also tried being sneaky by using a pthread in the C code, but I
had issues when I tried to create a new thread state using
ctypes.pythonapi (well, I had issues swapping it in when I get to the
callback). If this is the best solution, how do I create/swap in the
thread state from ctypes?

For some cooked up sample code that simulates this:

main.c (main.o -> main.so )
#include <stdio.h>
void loop( void (*callback)() )
{
while( 1 )
{
callback();
sleep(1);
}
}

void testLoop( void (*callback)() )
{
loop( callback );
}


************************************************
test.py:
import threading,ctypes,time,sys,os

soPath = os.path.join( "/home/tdimson/ctypes/main.so" )

class callLoop( threading.Thread ):
def callback( self ):
sys.exit()

def run( self ):
ctypes.cdll.LoadLibrary( soPath )
mainLib = ctypes.CDLL( soPath )
_callback = ctypes.CFUNCTYPE( None )( self.callback )

mainLib.testLoop( _callback )

loopThread = callLoop()
loopThread.start()

while 1:
print "Not blocking"
time.sleep(10)

********************************************

Then I execute: python test.py and get
Not blocking
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.4/threading.py", line 634, in __exitfunc
t.join()
File "/usr/lib/python2.4/threading.py", line 532, in join
assert self is not currentThread(), "cannot join current thread"
AssertionError: cannot join current thread
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.4/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.4/threading.py", line 634, in __exitfunc
t.join()
File "/usr/lib/python2.4/threading.py", line 532, in join
assert self is not currentThread(), "cannot join current thread"
AssertionError: cannot join current thread


Thanks for even reading this much

-Thomas Dimson

 

[Thomas Dimson]

No... as I recall, you can't /EXIT/ Python from a sub-thread...
Which is what sys.exit() or whatever is trying to do -- shut down the
entire program, not just the thread. The error you get indicates that
the thread doing the shutdown wants to wait for the sub-thread to finish
-- but /it/ IS the sub-thread. Even console interrupts have to be
delivered to the main program.

The only safe way to terminate a thread is to be able to code it
such that /it/ responds to an externally set value (a boolean, read a
message off a Queue, etc.) and for IT to then exit. Based upon the
sample you showed, that would require the C main loop to be checking for
a shutdown signal... If the API to that main loop library doesn't
include a shutdown capability I'd suggest it is a less than complete
library... And the only thing I'd suggest is not using a Python thread,
but instead spawning a separate process that somehow communicates to the
parent process -- and which can be forceably killed using OS specific
capabilities... "kill -9 pid" <G>

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

Thanks for the response, it put me in the right direction (I didn't
realize there was no way of exiting the interpreter directly from a
non-main thread).

If anyone ever has the same problem, the solution I ended up using
went like this:

I created a wrapper around the infinite loop call that had a setjmp in
it, exiting if setjmp was non-zero.

Inside each callback function, I had a try/except statement that
caught all exceptions. If it had an exception, it would set a thread-
specific exception variable to sys.exc_info() and then call a C
function that did a longjmp.

The thread would first call the wrapper to the infinite loop. If the
wrapper returns (because of a longjmp), it would check the thread-
specific exception variable for a non-None value and raise the very
same exception (with the same traceback) if it found it.

A fairly large hack, but it seemed to do the job. Thanks again.