thread vs GC

[Paul Rubin]

Another of those "how can I kill a thread" questions.

Let's say I have an app that needs a random prime number for something
every now and then, based on user interaction. I have a function
makeprime() which creates such a prime, but it's pretty slow, so I
don't want to wait for it when the user clicks for a new prime. I want
to generate some primes in the background and cache a few of them. Let's
say I want to make an iterator object to wrap all this:

class primegen:
def __init__(self):
self.q = Queue.Queue(5) # cache up to 5 primes
def background_generator():
while True:
self.q.put(makeprime()) # sleeps if 5 primes are already cached
threading.Thread(target=background_generator).start()

def next(self):
return self.q.read()
def __iter__(self): return self

Now I can make a caching prime stream with

g = primegen()

and it's all rather elegant. But maybe g will sit in some other
object or for some other reason I end up making a bunch of these
generators. The generators get cleaned up by GC when they go out of
scope, but the threads they spawn are left hanging around.

Adding a __del__ method that sets a flag (actually threading.Event)
which the background_generator function checks doesn't seem to do
the job.

Any suggestions for the cleanest way to get rid of the thread? I can
think of some awful kludges that I'd rather not resort to.

Extra points if it works in both Python and Jython, i.e. it isn't GIL
dependent.

Thanks for any ideas.

 

[Chris Lambacher]

You probably have one or more of several problems.


If a __del__ method exists, the object might be stuck in the
gc.garbage list. In general you are better off explicitly stopping it
my adding a method that is called to set your threading.Event.

Likely you will still be stuck waiting on your queue and unable to
check the status of threading.Event. Try setting a timeout on the
blocking Queue.put call and then checking the status of the event. If
it is set break out, otherwise go try the put again. Depending on how
makeprime works, you may have to cache the value so you can keep
trying to put the number in the queue.

class primegen:
def __init__(self):
self.q = Queue.Queue(5) # cache up to 5 primes
self.e = threading.Event()
def background_generator():
n = makeprime()
while True:
try:
self.q.put(makeprime(),True, 1) # sleeps if 5 primes are
already cached
except Queue.Full:
pass
else:
n = makeprime()
if self.e.isSet():
return
threading.Thread(target=background_generator).start()

def next(self):
return self.q.read()
def __iter__(self): return self

def stop(self):
self.e.set()

-Chris

 

[Jeff Epler]

I suspect that getting the threads to die will be tricky, and as written
the thread holds a reference to the 'primegen' instance (this part can
be cured, but it still doesn't ever make the thread exit).

Instead of figuring out how to get this to clean itself up, why not make
sure you only make one 'primegen' instance, using one of the various
singleton patterns? There may still be something "uncollectable", but
at least there will only be one.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCn8i3Jd01MZaTXX0RAkgsAJ9qbfhTdHhwYw5vEW/j7aGZolDudQCeJB9Y
R1UmxZjNS1xqLmtntCM9nvU=
=OYRC
-----END PGP SIGNATURE-----

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Any suggestions for the cleanest way to get rid of the thread?

As Jeff explains, it is rather unlikely that GC will collect
primegen objects, since the generating thread holds self as
a local variable.

You should make background_generator have explicit q and
event arguments, and you should signal the event in __del__.
However, this won't terminate the thread, since it still
hangs in .put. So it might be easiest to .read() in __del__
first.

Regards,
Martin