2.4 crashes when try to exit app and mulitple threads active

[Maxwell Hammer]

An application I am developing executes many threads and then has a
"monitor" part that waits for certain events. One of these events causes
the application to have to shutdown. On shutdown the monitor part
notifies the threads of a shutdown, and the threads must cleanup and exit.
When all threads have exited the monitor itself exits.

The traceback below occured on a shutdown. At the time there was only one
thread running, however the same traceback occurs with many threads active.

Some things I've observed:
1 - my application is able to complete successfully,

2 - it appears to be only in python 2.4. - does not occur with 2.3.1.

3 - it is not consistent, there are times when there is no traceback

4 - more likely to occur the longer a thread/application has run.

5 - does occur frequently enough (50% of the time) to be a problem

I do not know how to investigate further. Any help or suggestions on how
to investigate this and resolve would be much appreciated.

Thanks.
---------------------------------------------------------------------------------
Error traceback:
-----------------
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.4/atexit.py", line 22, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.4/threading.py", line 636, in __exitfunc
self._Thread__delete()
File "/usr/lib/python2.4/threading.py", line 522, in __delete
del _active[_get_ident()]
KeyError: 16384
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.4/atexit.py", line 22, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.4/threading.py", line 636, in __exitfunc
self._Thread__delete()
File "/usr/lib/python2.4/threading.py", line 522, in __delete
del _active[_get_ident()]
KeyError: 16384

Software environment:
-----------------------
*********************************************
*** SYSTEM INFORMATION ***
*** (Sat Mar 19 19:15:03 EST 2005) ***
*********************************************
Host or node name : poseidon
Machine type : i686
Operating system : Linux
Kernel version : 2.4.29
Linux Distribution : Slackware 10.1.0
*********************************************
Python 2.4 (#1, Jan 1 2005,21:33:55) [GCC 3.3.4] on linux2
Gnu make 3.80
util-linux 2.12p
modutils 2.4.27
e2fsprogs tune2fs
Linux C Library 2.3.4
Dynamic linker (ldd) 2.3.4
Linux C++ Library 5.0.6
Procps 3.2.3
Net-tools 1.60
Kbd 1.12
Sh-utils 5.2.1
*********************************************
Modules Loaded:
nfs
lockd
sunrpc
ppp_deflate
zlib_inflate
zlib_deflate
bsd_comp
ppp_async
ppp_generic
slhc
emu10k1
ac97_codec
soundcore
xfs
ne2k-pci
8390
crc32
*********************************************
gcc version 3.3.4
Reading specs from
/usr/lib/gcc-lib/i486-slackware-linux/3.3.4/specs Configured with:
.../gcc-3.3.4/configure --prefix=/usr --enable-shared
--enable-threads=posix --enable-__cxa_atexit --disable-checking
--with-gnu-ld --verbose --target=i486-slackware-linux
--host=i486-slackware-linux
Thread model: posix
*********************************************

 

[Peter Hansen]

An application I am developing executes many threads and then has a
"monitor" part that waits for certain events. One of these events causes
the application to have to shutdown. On shutdown the monitor part
notifies the threads of a shutdown, and the threads must cleanup and exit.
When all threads have exited the monitor itself exits.

The traceback below occured on a shutdown. At the time there was only one
thread running, however the same traceback occurs with many threads active.

When you say "only one thread running", did you mean only
one monitor thread in addition to the main thread, or did
you really mean only the main thread was active at this time?

Also, are you doing something odd to terminate threads
here, and could you have screwed things up yourself?
The code in question is at the end of the MainThread's
__exitfunc() method, and normally this would be executing
only after all other threads had terminated, and should
be safe. It appears as though the _active dictionary that
tracks threads has been modified by someone else.

Can you post a small self-contained example that
reproduces the problem?

-Peter

 

[Maxwell Hammer]

When you say "only one thread running", did you mean only
one monitor thread in addition to the main thread, or did
you really mean only the main thread was active at this time?

I meant there was the main app and one thread. The problem
does not occur when there are no threads running and the app
has to exit.

Also, are you doing something odd to terminate threads
here, and could you have screwed things up yourself?

Very likely - it has happened before

The code in question is at the end of the MainThread's
__exitfunc() method, and normally this would be executing
only after all other threads had terminated, and should
be safe. It appears as though the _active dictionary that
tracks threads has been modified by someone else.

That is what it looks like to me. It seems that the interpreter
is trying to delete a non-existant thread

Can you post a small self-contained example that
reproduces the problem?

This will be difficult to do...but perhaps if I give a summary:

def main():
...initialize/configure
...wait for certain events
...based on events launch appropriate thread as follows:

thread.start_new_thread(event_module.run, (StateValues, (job_parms)))
...

def event_module.run(StateValues, (job_parms)):
...initialize/configure
while (1):
Return = os.system(RunString)

if Shutdown flag exists: #thread detects here if needs to exit
save_job(parms)
cleanup_exit(message, exit_code)
if Return == 0: # SUCCESS
break
elif Return == 1: # ERROR
...
elif Return == 7: # ERROR
...

def cleanup_exit(message, exit_code):
remove temp files
log message, etc
return

So at shutdown, any threads running save their state to a job file
then simply do a return.

Hope this helps...also I use command line apps to program like Nedit.

Know of any visual python debuggers? I know there is "idle"...
any others, better ones?
Have you heard of "eric" python ide. Is it good? - it would have debug
capabilities but is dependent on many libraries I think.

thanks a lot.

 

[Peter Hansen]

I meant there was the main app and one thread. The problem
does not occur when there are no threads running and the app
has to exit.

Okay, but make sure you understand that it's not possible
to have "no threads running" unless your application has
already exited. There is *always* at least the main thread.
I think you know that, but it's confusing when you keep
talking as though the main thread doesn't exist.

That is what it looks like to me. It seems that the interpreter
is trying to delete a non-existant thread

More than that, it's the MainThread in threading.py trying
to delete itself at exit, and that should always be possible
unless you've done Something Bad (or if this is indeed some
Python bug).

This will be difficult to do...but perhaps if I give a summary:

def main():
...initialize/configure
...wait for certain events
...based on events launch appropriate thread as follows:

thread.start_new_thread(event_module.run, (StateValues, (job_parms)))
...

Unfortunately, pseudo code like that is of little help.

It does, however, raise a question. You are using the thread.py
module here, yet the exception is being generated as a result
of the MainThread object (that is created when you import
the threading.py module) trying to exit: how can it be that
threading.py is imported when you are using only the thread
module yourself? Other code that you aren't aware of? More
complexity than your snippets suggest?

You really need to reduce this down to a simple piece of code
that reproduces the problem if you hope to solve it. Race
conditions are just about the hardest thing in the world to
troubleshoot, and while you could report a bug here, without
code to reproduce the chances of someone finding and fixing
it are just about nil.

-Peter