Threading: Method trigger after thred finished

[markus.mj]

Hi,

I am looking for help with following problem. I scripted threaded database query, with session open per thread, and queries delivered through queue. Every open DB session must be closed with "abort" or "commit", however on which event should I trigger the DB disconnect? Ideally it would close the DBas the thread class gets deconstructed, but "__del__" does never fire. Howwould you solve the problem? Here an non working example (bended from IBM developerWorks tutorial) that explains the situation:

<code>
multi_query = ["query1","query2","query3","query4","query5","query6"]
queue = Queue.Queue()

class ThreadSql(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
#Open database connection instance
self.session = DbConnect()

def run(self):
while True:
#grabs query from queue
query = self.queue.get()
#Fire query and print result
print self.session.SQL(query)
#Queue job is done
self.queue.task_done()

# THIS PART IS NOT WORKING
def __del__(self):
#Disconnect Database session and commit or abort transactions
self.session.Disconnect(<abort, commit>)
print "The End"
#---------------------------------

for i in range(5):
t = ThreadUrl(queue)
t.setDaemon(True)
t.start()

#Fill the queue
for single_query in multi_query:
queue.put(single_query)

#Wait until query is empty and finish
queue.join()
</code>

Thank you for any idea!

Markus

 

[Steven D'Aprano]

Hi,

I am looking for help with following problem. I scripted threaded
database query, with session open per thread, and queries delivered
through queue. Every open DB session must be closed with "abort" or
"commit", however on which event should I trigger the DB disconnect?
Ideally it would close the DB as the thread class gets deconstructed,
but "__del__" does never fire.

The __del__ destructor method is not guaranteed to be called in a timely
fashion, if at all. My *guess* is that the main Python environment is
shutting down when the daemon threads get killed, and the __del__ method
never gets a chance to run.

To be honest, I'm not sure why you are using daemon threads. It seems to
me that blocking until the queue is empty makes the use of daemon threads
pointless, but I'm not experienced enough with threads to be sure.

The usual advice is to explicitly call destructors rather than rely on
automatic __del__ methods. Given that, this works for me:


import threading
import Queue
import time

# Fill the queue.
queue = Queue.Queue()
queries = ["query"+str(i) for i in range(10)]
for query in queries:
queue.put(query)

# Make consumer threads.
class ThreadSql(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
# Open database connection instance
self.session = "+++connection+++" # DbConnect()
self._open = True

def run(self):
while self._open:
# Grab a query from queue.
query = self.queue.get()
# And process it.
print self, query
time.sleep(1)
# Mark the queue job as done.
self.queue.task_done()

def close(self):
print "Closing", self
# self.session.Disconnect(<abort, commit>)
self._open = False

threads = [ThreadSql(queue) for _ in range(4)]
for t in threads:
t.setDaemon(True)
t.start()

# Wait until the queue is empty, then close the threads.
queue.join()
for t in threads:
t.close()

 

[Markus]

Hi,

I am looking for help with following problem. I scripted threaded
database query, with session open per thread, and queries delivered
through queue. Every open DB session must be closed with "abort" or
"commit", however on which event should I trigger the DB disconnect?
Ideally it would close the DB as the thread class gets deconstructed,
but "__del__" does never fire.

The __del__ destructor method is not guaranteed to be called in a timely
fashion, if at all. My *guess* is that the main Python environment is
shutting down when the daemon threads get killed, and the __del__ method
never gets a chance to run.

To be honest, I'm not sure why you are using daemon threads. It seems to
me that blocking until the queue is empty makes the use of daemon threads
pointless, but I'm not experienced enough with threads to be sure.

The usual advice is to explicitly call destructors rather than rely on
automatic __del__ methods. Given that, this works for me:

import threading
import Queue
import time

# Fill the queue.
queue = Queue.Queue()
queries = ["query"+str(i) for i in range(10)]
for query in queries:
    queue.put(query)

# Make consumer threads.
class ThreadSql(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.queue = queue
        # Open database connection instance
        self.session = "+++connection+++"  # DbConnect()
        self._open = True

    def run(self):
        while self._open:
            # Grab a query from queue.
            query = self.queue.get()
            # And process it.
            print self, query
            time.sleep(1)
            # Mark the queue job as done.
            self.queue.task_done()

    def close(self):
        print "Closing", self
        # self.session.Disconnect(<abort, commit>)
        self._open = False

threads = [ThreadSql(queue) for _ in range(4)]
for t in threads:
    t.setDaemon(True)
    t.start()

# Wait until the queue is empty, then close the threads.
queue.join()
for t in threads:
    t.close()

Hi Steven,

great point with the explicit call of destructor.
I did a quick test and it is behaving exactly like I need.

Thank you very much!
Markus