Searching equivalent to C++ RAII or deterministic destructors

[Ulrich Eckhardt]

Hi!

I'm currently converting my bioware to handle Python code and I have
stumbled across a problem...

Simple scenario: I have a handle to a resource. This handle allows me to
manipulate the resource in various ways and it also represents ownership.
Now, when I put this into a class, instances to that class can be shared,
using Python's reference counting. What I'm missing is a way to
automatically release the resource, something which I would do in the
destructor in C++.

Any ideas how to solve this?

Uli

 

[Bearophile]

Ulrich Eckhardt:

a way to automatically release the resource, something
which I would do in the destructor in C++.

Is this helpful?
http://effbot.org/pyref/with.htm

Bye,
bearophile

 

[Ulrich Eckhardt]

Ulrich Eckhardt:

Is this helpful?
http://effbot.org/pyref/with.htm

Yes, it aims in the same direction. However, I'm not sure this applies to my
case. The point is that the resource handle is not just used locally in a
restricted scope but it is allocated and stored. The 'with' is something
that makes sense in the context of mutex locking, where you have a
well-defined critical section. What I need is something similar to open(),
which returs a file. When the last reference to that object goes out of
scope, the underlying file object is closed.

Uli

 

[Peter Otten]

Yes, it aims in the same direction. However, I'm not sure this applies to
my case. The point is that the resource handle is not just used locally in
a restricted scope but it is allocated and stored. The 'with' is something
that makes sense in the context of mutex locking, where you have a
well-defined critical section.

Isn't that exactly what RAII does?

What I need is something similar to open(),
which returs a file. When the last reference to that object goes out of
scope, the underlying file object is closed.

You can go ahead and implement a __del__() method. It will often work in
CPython, but you get no guarantees, especially when you have reference
cycles and with other Python implementations that don't use refcounting.

Peter

 

[Dave Angel]

Hi!

I'm currently converting my bioware to handle Python code and I have
stumbled across a problem...

Simple scenario: I have a handle to a resource. This handle allows me to
manipulate the resource in various ways and it also represents ownership.
Now, when I put this into a class, instances to that class can be shared,
using Python's reference counting. What I'm missing is a way to
automatically release the resource, something which I would do in the
destructor in C++.

Any ideas how to solve this?

Uli

As someone else pointed out, 'with' is the first line of defense. It
makes the stuff you could already do with try/except/finally much easier
to get right.

Look also at 'del' a command in the language which explicitly deletes an
object.

But I'm guessing you want something that automatically deletes objects
whenever the last reference disappears. That's an implementation
detail, not a language guarantee. In particular CPython does what you
want, by using reference counting. That's the only Python I've used, so
it's only hearsay when I say that other implementations, (maybe Cython
or Jython) do not all work the same way.

In CPython, any object whose *last* reference goes away, will get
immediately deleted. So if you avoid circular references, it should do
just what you want. Orphaned circular references are caught by the
garbage collector, which runs periodically, but is non-deterministic.

Two more caveats. Exceptions tend to hang onto stuff in their near
vicinity, and I can't tell you the algorithm for what happens. But an
explicit del in the except clause can probably handle that.

Finally, closures can hang onto stuff. So if you have nested functions,
or lambda functions, it's possible they're going to keep things longer
than you realize.

 

[Peter Otten]

But I'm guessing you want something that automatically deletes objects
whenever the last reference disappears. That's an implementation
detail, not a language guarantee. In particular CPython does what you
want, by using reference counting. That's the only Python I've used, so
it's only hearsay when I say that other implementations, (maybe Cython
or Jython) do not all work the same way.

Here are some examples from Kubuntu 9.04's zoo of python implementations:

$ cat del.py
import sys
print sys.version

class A(object):
def __init__(self, x):
self.x = x
def __del__(self):
print "releasing A(%r)" % self.x

def f():
a = A("local in function")
f()

a = A("global (one ref)")

c = A("global (cycle)")
c.a = c
del c

b = A("global (no refs)")
del b

print "about to quit"

$ python del.py
2.6.2 (release26-maint, Apr 19 2009, 01:58:18)
[GCC 4.3.3]
releasing A('local in function')
releasing A('global (no refs)')
about to quit
releasing A('global (one ref)')

$ jython del.py
2.2.1
about to quit

$ ipy del.py
2.4.0 (IronPython 1.1.1 (1.1.1) on .NET 2.0.50727.42)
about to quit
releasing A('global (no refs)')
releasing A('global (cycle)')
releasing A('local in function')

Unhandled Exception: System.ArgumentException: I/O operation on closed file
at IronPython.Runtime.PythonFile.ThrowIfClosed () [0x00000]
at IronPython.Runtime.PythonFile.Write (System.String s) [0x00000]
$

IronPython sometimes segfaulted.

Peter

 

[Carl Banks]

Yes, it aims in the same direction. However, I'm not sure this applies to my
case. The point is that the resource handle is not just used locally in a
restricted scope but it is allocated and stored. The 'with' is something
that makes sense in the context of mutex locking, where you have a
well-defined critical section. What I need is something similar to open(),
which returs a file. When the last reference to that object goes out of
scope, the underlying file object is closed.

On CPython you can do it with a __del__ attribute.

Warning: objects with a __del__ attribute prevent reference cycle
detection, which can potentially lead to memory (and resource) leaks.
So you must be careful to avoid creating reference loops with that
object.

Note that file objects have a close method; you can explicitly close
it at any time. Your object should follow that example, and define a
close (or release, or whatever) method. I'd recommend making an
effort to call it and to rely on __del__ as little as possible.


Carl Banks

 

[Roel Schroeven]

Peter Otten schreef:

Isn't that exactly what RAII does?

RAII also works if the resource handle is stored, for example, in a data
member of an object. If that object is destroyed (because it goes out of
scope, or because it is deleted), the resource is automatically
destroyed too.

The way RAII works is actually the one thing from C++ that I miss in Python.

--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov

Roel Schroeven

 

[ryles]

You can go ahead and implement a __del__() method. It will often work in

CPython, but you get no guarantees, especially when you have reference
cycles and with other Python implementations that don't use refcounting.

And for resources whose lifetime is greater than your process (e.g. a
file), you can also use the atexit module to help ensure they are
cleaned/reclaimed at shutdown. One way to do this is to maintain a
weak dictionary (from the weakref module) to your objects and install
a handler which iterates this. This is useful for not only dealing
with reference cycles, but for objects which may exist at the time the
interpreter exits. These may not be deleted.

http://docs.python.org/reference/datamodel.html#object.__del__

 

[Ulrich Eckhardt]

Thanks to all that answered, in particular I wasn't aware of the existence
of the __del__ function.

For completeness' sake, I think I have found another way to not really solve
but at least circumvent the problem: weak references. If I understand
correctly, those would allow me to pass out handles to the resources and,
if some code decides it is time, release the resources and render all the
weak references invalid. At least I don't suffer resource leaks but rather
get meaningful errors that way, which is enough for my case.

cheers!

Uli

 

[Jack Diederich]

On CPython you can do it with a __del__ attribute.

Warning: objects with a __del__ attribute prevent reference cycle
detection, which can potentially lead to memory (and resource) leaks.
So you must be careful to avoid creating reference loops with that
object.

WARNING-er: As Carl points out, adding a __del__ method actually makes
it /less/ likely that your object will be cleaned up. __del__ is only
useful if you have a complicated tear-down procedure. Since you come
from C++ RAII land (my old haunt) your objects probably only allocate
one resource each, or worse: a bunch of objects that allocate one
resource each. In that case __del__ will always hurt you.

The C++ temptation is to match every __init__ with a __del__. A
better rule of thumb is to only add a __del__ method after confirming
with someone else that it would be useful. Better still is to ask
them by postal courier. For best results that someone should be your
grandmother.

Note that file objects have a close method; you can explicitly close
it at any time.  Your object should follow that example, and define a
close (or release, or whatever) method.  I'd recommend making an
effort to call it and to rely on __del__ as little as possible.

This. If you care enough about a resource to write a __del__ method
you care enough to clean it up explicitly. 'with' blocks are very
nice for that.


-jack

 

[Aahz]

Yes. As far as the object is concerned, "obj = None" and "del obj" are
exactly identical. In both cases, there is one less binding to the name.

The difference between the two is only whether the name lives on in the
namespace.

A local variable is (usually) just a name in the local() namespace.

OTOH, Python's ``del`` applies to targets generally, not just names:

L = [1,2,3]
del L[1]
L

[1, 3]

 

[Carl Banks]

WARNING-er: As Carl points out, adding a __del__ method actually makes
it /less/ likely that your object will be cleaned up.

No I wasn't pointing that out, and I don't really agree with it. (You
can't really say it is "more likely" or "less likely" without
considering the skill and patience of the programmer, and there are
many programmers who are up to the task.)

__del__ can succeed with care, and if you are willing to give up a
certain amount of flexibility and portability

I don't recommend going that way. For one thing RAII isn't quite so
wonderful in Python's dynamic environment as it is in C++'s static
universe, and people seem to expect more from it than they get.
Second, relying on implementation-dependent behavior is not a good
thing and can set you up for problems later, something few people
expect to happen to them but it does more often than they think.
(Search comp.lang.python for a thread started by Warren DeLano for an
example of someone who got bit hard by such a change he should have
seen coming.) But still I leave that for them to risk for themselves.


[snip]

 If you care enough about a resource to write a __del__ method
you care enough to clean it up explicitly.  'with' blocks are very
nice for that.

The OP already said with blocks won't suffice since the resources are
long-lived objects that aren't limited to a single scope.


Carl Banks