Garbage collector and threads

[Nicolas Fleury]

Hi everyone,
I just discovered, correct me if I'm wrong, that as long as a
threading.Thread is running, its destructor will not be called, because
it is referring to himself. So if I have something like:

class MyThread(threading.Thread):
def __init__(self):
self.cancelEvent = threading.Event()
threading.Thread.__init__(self)
def __del__(self):
self.cancel()
def run(self):
self.cancelEvent.wait()
def cancel(self):
self.cancelEvent.set()

I must call cancel from the outside if I want the destructor to be
called (note that I don't want deamon threads). I can make a wrapper
over threading.Thread to have the behaviour I want: have a thread with
__del__ called when it is not referred by another thread. But my
question is, and I don't have an overall vision of the issue at all,
should that be the default behaviour anyway?

Regards,
Nicolas

 

[Aahz]

I just discovered, correct me if I'm wrong, that as long as a
threading.Thread is running, its destructor will not be called, because
it is referring to himself. So if I have something like:

class MyThread(threading.Thread):
def __init__(self):
self.cancelEvent = threading.Event()
threading.Thread.__init__(self)
def __del__(self):
self.cancel()
def run(self):
self.cancelEvent.wait()
def cancel(self):
self.cancelEvent.set()

I must call cancel from the outside if I want the destructor to be
called (note that I don't want deamon threads). I can make a wrapper
over threading.Thread to have the behaviour I want: have a thread with
__del__ called when it is not referred by another thread. But my
question is, and I don't have an overall vision of the issue at all,
should that be the default behaviour anyway?

Yes, Don't Do That. Do not rely on finalizers for external resources
(such as threads).

Note that your understanding of Python's memory management could use some
improvement: GC *never* runs on objects containing __del__

 

[Nicolas Fleury]

Yes, Don't Do That. Do not rely on finalizers for external resources
(such as threads).

Why? Isn't a kind of RAII applied to Python?

Note that your understanding of Python's memory management could use some
improvement: GC *never* runs on objects containing __del__

Well, I guess you're right. What I mean is the reasons behind the call
to __del__, whatever it is ref-counting or something else. Wouldn't a
class like CancellableThread, with a sub-class implemented cancel method
called when the object is not referred in any other thread be useful?

Regards,
Nicolas

 

[Aahz]

Why? Isn't a kind of RAII applied to Python?

What's "RAII"?

Well, I guess you're right. What I mean is the reasons behind the call
to __del__, whatever it is ref-counting or something else. Wouldn't a
class like CancellableThread, with a sub-class implemented cancel method
called when the object is not referred in any other thread be useful?

Perhaps, but threading needs references to created threads in order to
implement its functionality. You might try building your own thread
constructs using ``thread`` directly.

 

[Michael Hudson]

What's "RAII"?

Resource Aquisition Is Initialization. Works in C++. Doesn't really
in Python.

Cheers,
mwh

 

[Nicolas Fleury]

What's "RAII"?

"Resource Acquisition Is Initialization"
It's a term frequently used in C++ to describe the use of constructor
and destructor for resource allocation/deallocation.

Perhaps, but threading needs references to created threads in order to
implement its functionality. You might try building your own thread
constructs using ``thread`` directly.

I still don't know how to implement it. I can do something very simple
like:

class Canceller:
def __init__(self, thread):
self.thread = thread
def __del__(self):
self.thread.cancel()
self.thread.join()

but I can't merge the concept with a Thread class, because the self
passed to run would be a reference to the object... I don't know if a
language extension would be worth the effort or if there's a cool
solution I'm missing, but what is sure is that I write a lot of code to
cancel threads in finalizers of parent objects. I don't want deamon
threads so that everything is stopped cleanly, but at the same time it
is error-prone in a GUI application, because I risk to have the process
still running after a quit.

I just want a way to garantee that all my threads are cancelled when the
main thread is finished.

Regards,
Nicolas

 

[Aahz]

"Resource Acquisition Is Initialization"
It's a term frequently used in C++ to describe the use of constructor
and destructor for resource allocation/deallocation.

As Michael Hudson said, it doesn't work that way in Python, because all
Python objects are global (and I mean truly global here, not just module
global). Essentially, all objects are heap objects that can be traced
by crawling through Python's internals.

I still don't know how to implement it. I can do something very simple
like:

class Canceller:
def __init__(self, thread):
self.thread = thread
def __del__(self):
self.thread.cancel()
self.thread.join()

but I can't merge the concept with a Thread class, because the self
passed to run would be a reference to the object... I don't know if a
language extension would be worth the effort or if there's a cool
solution I'm missing, but what is sure is that I write a lot of code to
cancel threads in finalizers of parent objects. I don't want deamon
threads so that everything is stopped cleanly, but at the same time it
is error-prone in a GUI application, because I risk to have the process
still running after a quit.

Right. As I said, you have to use the ``thread`` module directly; you
can't use ``threading``.

I just want a way to garantee that all my threads are cancelled when the
main thread is finished.

So hold references to them and call the ``cancel()`` method yourself.

 

[Nicolas Fleury]

As Michael Hudson said, it doesn't work that way in Python, because all
Python objects are global (and I mean truly global here, not just module
global). Essentially, all objects are heap objects that can be traced
by crawling through Python's internals.

I understand that. RAII is often used when speaking of scoped
variables. In my case I was talking about global scope in main thread.

Right. As I said, you have to use the ``thread`` module directly; you
can't use ``threading``.

But even if I use "thread", I have no way to put the functionality in
one class if I want access the members in the running thread.

So hold references to them and call the ``cancel()`` method yourself.

That's a possible solution. Make a custom Thread class with a cancel
method and with all instances registered somewhere. At the end of the
main thread, I call something to cancel all these threads. There's
however some problems with that solution:
- Parent threads need to be cancelled before their children threads.
- It only works at the end of the main thread; it cannot be used at the
middle of an appplication.
- The cancellations are not automatic (I guess there's a way to do it by
overriding something like exit, anyway...)

All these problems are solved by using RAII with the Canceller class I
shown in a previous message, so I prefer using this solution. In fact,
what I would really need, is a mechanism, and I don't know if there's
already one, to disable ref-counting of a thread object in its
corresponding thread, so that I could provide a single class for
cancellable threads.

Regards,
Nicolas

 

[Aahz]

But even if I use "thread", I have no way to put the functionality in
one class if I want access the members in the running thread.

Huh? I don't understand what you mean.

That's a possible solution. Make a custom Thread class with a cancel
method and with all instances registered somewhere. At the end of the
main thread, I call something to cancel all these threads. There's
however some problems with that solution:
- Parent threads need to be cancelled before their children threads.

Why's that a problem? Just hold references to the parent threads in the
main thread and cascade the cancels down.

- It only works at the end of the main thread; it cannot be used at the
middle of an appplication.

Why?

 

[Nicolas Fleury]

Huh? I don't understand what you mean.

Well, suppose I want to have a class with the same interface of
threading.Thread, but with two additions:
class CancellableThread:
def __del__(self): self.cancel()

The first addition is that the finalizer use a sub-class implemented
mechanism for cancellation of the thread.
The second addition is that, as soon as the object is not referred
anymore in the parent thread, the thread object is finalized.

Note that it doesn't need to be the finalizer, what is necessary is
something to be called when the object is no more referred in the parent
thread.

I'm saying that I don't know of any way to do that with a single class,
because I expect the sub-class to define something like:
def run(self)
And the thread object will be referred in the running thread by "self",
so I'm stuck without any way to know when the thread object is not
referred in the main thread.

I can do something like:
class Canceller:
def __init__(self, thread): self.thread = thread
def __del__(self): self.thread.cancel()

so that I encapsulate all threads with "Cancellers" in the parent
thread, but then I have to use two objects to the job that could
possibly be done by one.

Why's that a problem? Just hold references to the parent threads in the
main thread and cascade the cancels down.

You're right, it works. At the moment I wrote that I expected mix of a
CancellableThread class implemented with this mechanism with other
threads classes to cause loss of link between threads. If that happens
all threads that are not CancellableThreads (like the main thread) must
call the function to cancel remaining CancellableThreads started
directly by them.

Why?

Because the goal is to cancel threads that are no more referred in the
main thread. Since there's no way to know which they are, there's no
way to do it at the middle of an application. You have to do it at the
end of the main thread, as for other threads, where you can assumed that
cancellable threads should be cancelled and joined.

I don't know if my explications of what I'm searching are more clear. I
have the feeling it's impossible to do it without a language extension
and at the same time that it would be less error-prone than
threading.Thread as deamons or not. Maybe I could write a PEP just for
the exercise of it...

Regards,
Nicolas