debugging ignored exceptions at cleanup

[Michele Simionato]

In some conditions (typically with threads, __del__ methods, etc) the
cleanup mechanism of Python gets in trouble and some exceptions are
not raised but just printed on stderr.
I have an application using Paste and when I run the tests I get some
annoying
ignored exceptions during cleanup. Running the code with the -v option
I get things
like that:

<snip lots of other stuff>
# cleanup[2] sqlalchemy.util
# cleanup[2] sqlalchemy.sql.expression
# cleanup[2] symbol
# cleanup[2] urllib2
# cleanup[2] sqlalchemy.orm.query
# cleanup[2] smweb.config
# cleanup[2] formencode
# cleanup[2] smweb.config.environment
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<function <lambda> at 0x26861b8> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<function <lambda> at 0x2598578> ignored
# cleanup[2] dbhash
# cleanup[2] xmlrpclib
# cleanup[2] mako.pygen
# cleanup[2] time
# cleanup[2] paste.util.import_string
# cleanup sys
# cleanup __builtin__
# cleanup ints: 5096 unfreed ints in 145 out of 171 blocks
# cleanup floats: 43 unfreed floats in 3 out of 4 blocks

As you see, the exceptions happen during the cleanup of
smweb.config.environment,
which the part of code I wrote, but which has no obvious problems and
imports
*lots* of other stuff. Is there some way to hook in the cleanup
mechanism, start
a pdb and see really what is happening? I tried to trace the execution
flow with the
trace module but without success. There is so much code there that I
am unable
to track down the source of the problem. I suspect there is some
__del__ method
somewhere that tries to call a function which has been set to None by
the cleanup
mechanism, possibly from another thread, but I cannot find it. How am
I supposed to
debug such things?

 

[Diez B. Roggisch]

In some conditions (typically with threads, __del__ methods, etc) the
cleanup mechanism of Python gets in trouble and some exceptions are
not raised but just printed on stderr.
I have an application using Paste and when I run the tests I get some
annoying
ignored exceptions during cleanup. Running the code with the -v option
I get things
like that:

<snip lots of other stuff>
# cleanup[2] sqlalchemy.util
# cleanup[2] sqlalchemy.sql.expression
# cleanup[2] symbol
# cleanup[2] urllib2
# cleanup[2] sqlalchemy.orm.query
# cleanup[2] smweb.config
# cleanup[2] formencode
# cleanup[2] smweb.config.environment
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<function <lambda> at 0x26861b8> ignored
Exception exceptions.TypeError: "'NoneType' object is not callable" in
<function <lambda> at 0x2598578> ignored
# cleanup[2] dbhash
# cleanup[2] xmlrpclib
# cleanup[2] mako.pygen
# cleanup[2] time
# cleanup[2] paste.util.import_string
# cleanup sys
# cleanup __builtin__
# cleanup ints: 5096 unfreed ints in 145 out of 171 blocks
# cleanup floats: 43 unfreed floats in 3 out of 4 blocks

As you see, the exceptions happen during the cleanup of
smweb.config.environment,
which the part of code I wrote, but which has no obvious problems and
imports
*lots* of other stuff. Is there some way to hook in the cleanup
mechanism, start
a pdb and see really what is happening? I tried to trace the execution
flow with the
trace module but without success. There is so much code there that I
am unable
to track down the source of the problem. I suspect there is some
__del__ method
somewhere that tries to call a function which has been set to None by
the cleanup
mechanism, possibly from another thread, but I cannot find it. How am
I supposed to
debug such things?

You aren't I guess. AFAIK, you can't do anything about this - if there
is really some cleanup todo, it should be done before exiting the program.

I for once know that the TurboMail lib has had similar problems - and
the "fix" was to check the used names in __del__ before trying to invoke
anything on them. Even *that* might not help in a multi-threaded
environment though, nobody can guarantee that between the check and the
usage of a name the underlying object isn't GCed.

Diez