Deterministic destruction and RAII idioms in Python

[plahey]

I have been dabbling in Python for a while now. One of the things that
really appeals to me is that I can seem to be able to use C++-style
RAII idioms to deal with resource management issues.

For those that have no idea what I am talking about (I learn a lot
reading posts on subjects in which I am clueless), consider the
following code snippet:

for line in file(name):
...print line,

This is nice and clean because I don't have to worry about cleaning
up after myself. If I can't rely on the destructor for the file
object to close the file, I must write the code like this:

file_obj = file(name)
for line in file_obj:
...print line,
file_obj.close()

not nearly as nice. Depending on the type of work you do, this can be
no problem or a major headache when exceptions are thrown into the mix.

Python objects have destructors so it seems that the original intent
was to support deterministic destruction (destructors are not very
useful in most GC'ed languages since you never know when or even if
they will be called). The problem is that other implementations of
Python (Jython and IronPython) do not support deterministic
destruction. So we are left with a problem: is deterministic
destruction an implementation detail of CPython that can go away at
anytime, or is it an official property of the language.

Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream" Python or am
I out of luck here?

 

[Terry Reedy]

The problem is that other implementations of
Python (Jython and IronPython) do not support deterministic
destruction. So we are left with a problem: is deterministic
destruction an implementation detail of CPython that can go away at
anytime, or is it an official property of the language.

It is a reliable implementation detail of CPython that will not go away
anytime soon. For the language def, see the Reference Manual section of
the del command.

Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream" Python or am
I out of luck here?

Depends on what you mean by 'mainstream'.

Terry Jan Reedy

 

[Giovanni Bajo]

Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream" Python or am
I out of luck here?

Most people rely on that. I do that *all* the time, and I would really dislike
if CPython lose this feature. Anyway, if you want to make your code portable to
other Python implementations, you'll have to implement your RAII idioms in a
slightly different way. Specifically, Python supports a try/finally construct
that it is useful to properly release resources. Eg. the following code is
"correct" even with IronPython:

f = file(name):
try:
for L in f:
print L
finally:
f.close()

Anyway, there is some work being done to add some RAII constructs to the
language that are compatibile across Python implementations. See the new "with"
statement. http://www.python.org/peps/pep-0343.html. This would allow to write
generic "functors" for RAII using generators, such as:

@contextmanager
def opened(filename):
f = open(filename)
try:
yield f
finally:
f.close()

These building blocks can then be used in the new 'with' statement such as:

with opened("foo.txt") as f:
for L in f:
print L

Of course, there's more to this (eg, 'with' statement can be used also with
normal objects as long as they support new __enter__ and __exit__ special
method), so go to the link above for more details and more juicy examples that
I'm sure you'll like.

 

[Paul Rubin]

Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream" Python or am
I out of luck here?

IMO you shouldn't rely on it. I believe it already doesn't work in
Jython, which uses the regular Java GC system. See PEP 343 for a new
(forthcoming) Python mechanism to do what you're looking for:

http://www.python.org/peps/pep-0343.html

You shouldn't think of reference counting as being like C++
destructors. A destructor definitely runs when the C++ object goes
out of scope. Ref counting frees only the object if there are no
other references when the scope exits. So if you want to rely on the
object getting gc'd when you expect, you have to carefully manage the
references, which defeats the point of the language doing it for you
automatically.

 

[plahey]

Thanks to all who replied. This has definitely given me something to
chew on.

I looked at pep-0343, it looks interesting. It is not what I really
want (deterministic destruction) but it is a lot more than most GC'ed
languages give me (Java, I am looking at you... ).

As far as my comment about "mainstream" Python, I have always taken
CPython as "Python". I guess this will have to change as Jython and
IronPython bring interesting things to the table, even if they do take
some things away in the process.

Cheers!

 

[Paul Rubin]

I looked at pep-0343, it looks interesting. It is not what I really
want (deterministic destruction)

I think it's better.

As far as my comment about "mainstream" Python, I have always taken
CPython as "Python". I guess this will have to change as Jython and
IronPython bring interesting things to the table, even if they do take
some things away in the process.

Hopefully PyPy will become "mainstream". How it handles GC, I'm not sure.

 

[plahey]

Hi Paul,

I think it's better.

Is there something specific you have in mind that makes you say that?

I am not a python expert, so I probably do not understand all the
implications of the proposal. What I got out of it is summarized by:

with EXPR as VAR:
BLOCK

The translation of the above statement is:

abc = (EXPR).__context__()
exc = (None, None, None)
VAR = abc.__enter__()
try:
try:
BLOCK
except:
exc = sys.exc_info()
raise
finally:
abc.__exit__(*exc)

Which, looks like you have a constructor (__enter__) and a destructor
(__exit__), which you can always count on being called. I am not sure
how that is better than C++-style RAII. Now, in the interest of full
disclosure, there is this __context__ method which I don't really
follow... maybe there is something important there I should know about?

At first glance, it seems to me that the syntax gets slightly ackward
as you encapsulate more resources. From the proposal:

with opened(filename, "w") as f:
with stdout_redirected(f):
print "Hello world"

which is not as nice as simply creating two objects and having them
destroyed in the reverse order that they were created in when they go
out of scope. Is it big deal... no. I am definitely going to have to
play with this stuff a while before I really know its strengths and
weakness relative to C++-style RAII.

If you have some examples, however, to support your feeling that it is
better than C++-style RAII, I would love to see them. I am always
eager to learn new stuff!

Again, I am not criticizing Python or the proposal. I like Python and
I like the capabilities that the proposal implies (to the extent that I
understand them ).

As far as PyPy goes, thanks for reminding me of that. I had forgotten
about that one.

Cheers!

 

[Paul Rubin]

Is there something specific you have in mind that makes you say that?...
Which, looks like you have a constructor (__enter__) and a destructor
(__exit__), which you can always count on being called. I am not sure
how that is better than C++-style RAII.

I didn't understand then. I thought by "deterministic destruction"
you meant relying on what CPython does now, which is reference
counting with destruction when the last reference is released.

PEP 343 is closer to C++-style RAII. I haven't studied it enough to
say what the differences really are. I do remember thinking C++ RAII
was a nuisance because you had to create classes with destructors to
do what other languages did with try/finally or the equivalent.

So to be clear, I think PEP 343 is better than simply expecting the
CPython reference counting system take care of releasing things when
the scope exits. I'm not much of a C++ user and I don't claim
anything about PEP 343 vs C++.

 

[plahey]

Hi Paul,

I didn't understand then. I thought by "deterministic destruction"
you meant relying on what CPython does now, which is reference
counting with destruction when the last reference is released.

I effectively do mean that. In C++ it is guaranteed that a destructor
will be called on an object when it goes out of scope, no matter how
that happens (normal exit, exception, etc...). C++-style RAII just
refers to wrapping up resources in classes so that you don't need to
worry about explicitly freeing them.

I do remember thinking C++ RAII
was a nuisance because you had to create classes with destructors to
do what other languages did with try/finally or the equivalent.

It really depends how often you have to deal with getting/releasing
resouces. You can define a class once and then just deal the resource
in your code with a single line (creating the object). It can also
simplify things when you are dealing with the simultaneous management
of multiple resources. In any case, this is now getting off topic.

Thanks for your input! It was fun.

 

[Carl Friedrich Bolz]

I think it's better.




Hopefully PyPy will become "mainstream". How it handles GC, I'm not sure.

Right now PyPy gives (at compile-time) the choice between two GCs: naive
reference counting and the conservative Boehm garbage collector
(http://research.hp.com/personal/Hans_Boehm/gc/). Currently we are
working on supporting other garbage collection strategies as well, like
mark-and-sweep, a copying collector or some sort of sophisticated
generational collector.

Cheers,

Carl Friedrich Bolz