exceptions from daemon threads which access the global namespace atinterpreter shutdown (how to sque

Discussion in 'Python' started by Ben Cohen, Apr 25, 2010.

  1. Ben Cohen

    Ben Cohen Guest

    I've got an application which makes fairly heavy use of daemon threads to perform 'background' processing and various other long-running tasks that are likely to block.

    I originally thought I could safely fire off a daemon threading.Thread and essentially forget about managing the thread's lifecycle. My thinking was that from the main thread's perspective its safe to fire off a daemon thread to do background twiddling and let the threading machinery manage the thread's lifespan assuming that:
    1. the 'background twiddling' is threadsafe and
    2. the thread can safely 'die' at any point without requiring a shutdown procedure

    Some coding later and I learn this isn't exactly the case, as there as in an additional requirement to correctly use 'daemon' threads -- they can't reference anything in the global namespace else they may raise exceptions at interpreter shutdown. As part of the shutdown procedure, the interpreter sets all global variables to None, a daemon thread may run while this is occuring/after it occured and raise an exception. The exception may get printed if the interpreter catches/prints it before the process exits.

    I garnered this understanding from this problem description -- (although all mistakes in description are my own)


    In this bug report they discuss an interpreter problem which affects non-daemon threads -- I'm not attempting to claim that I'm being affected by an interpreter bug, I reference this link only because it contains good descriptions of the interpreter shutdown process as well as the additional requirements the interpreter shutdown process places on 'daemon' threads
    Here's an example of a variable access from a daemon thread after interpreter shutdown started setting module globals to None:
    In this run, the process exited before even one traceback finished printing but this will be timing dependent, -- sometimes I'll see tracebacks from many backgrounds threads and sometimes just snippets like this. They will typically be AttributeError's or TypeError's resulting from failed attempts to access variables from the global namespace. Here's a longer example from another run:
    My questions concern whether or not its 'safe' to use daemon threads this way -- is the 'no references to globals()' 'requirement' imposed only by the interpreter shutdown process or are there additional reasons for the requirement?

    0. is it possible for me to consider it 'safe' to have code which can trigger the above exceptions during interpreter shutdown?
    1. if it is possible, and it turns out I'm certain what I'm doing is safe, is there a good way to squelch the printing of those stack traces?

    I was hunting around for a way to customize the value that all the globals get set to during interpreter shutdown -- i think it would work for me if I could get the interpreter to replace all globals with an object which:
    1. returns None as usual if accessed from a non-daemon thread
    2. blocks forever if accessed from a daemon thread (since the process is about to exit anyway)

    In my particular case I don't always have control over all of the daemon threads creation or lifespan -- I'm using the paramiko (www.lag.net/paramiko) ssh protocol library which uses daemon thread(s) 'behind the scenes' to simplify its networking code ...

    Looking forward to any comments/advice/alternate explanations or name calling.


    Ben Cohen
    Programmer/Analyst (STS)
    Scripps Institution of Oceanography

    Ben Cohen
    Programmer/Analyst (STS)
    Scripps Institution of Oceanography
    Ben Cohen, Apr 25, 2010
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.