is there anybody using __del__ correctly??

[Michele Simionato]

The title is provocative, of course

However, I was browsing through our codebase here at work and I
noticed a few
usages of __del__ as a resource finalizer (i.e. __del__ just calls a
close
method).

I consider this practice an error, since with __del__
you are never sure that the resource will be released
(http://docs.python.org/ref/customization.html#l2h-175) and anyway
this
should be done with try .. finally or the 'with' statement.

So I got into thinking: "yes, using __del__ as a resource finalizer is
wrong,
but then what are good use cases for it? Let's look at the standard
library and see what people use __del__ for".

So I did, and to my dismay 95% of the __del__ methods in the standard
library
are just calling a close method!!

In particular this happens in the following modules: (Python 2.5 on
Ubuntu):

zipfile, wave, urllib, close, tarfile, sunau, shelve, httplib, gzip,
fileinput, dumbdbm, audiodev, aifc, bsddb.dbshelve, tempfile, socket,
platform, ... (I got tired after this point)

I see one good use case for __del__ in wsgiref.validate: here __del__
prints a warning if the resource is *not* closed explicitely.
Something
similar happens in subprocess and popen2, where __del__ updates the
list
of active processes.

So I am beginning to wonder if there exists good use cases for
__del__,
apart for debugging/checking purposes. Can you provide some?
Yes, you may take that as a challenge


Michele Simionato


P.S. BTW, I should mention that if you search comp.lang.python for
__del__
you will find hundreds of people who were bitten by __del__, so I
usually give
advices such as "you should never __del__ in your code". If I am wrong
in
giving this advice, please let me know!

P.P.S. Here and there I hear rumors about deprecating __del__ and
nothing
happens, are there any news about that? Expecially concerning Py3k?

 

[greg]

So I did, and to my dismay 95% of the __del__ methods in the standard
library are just calling a close method!

You can't conclude that this is wrong just from
looking at the __del__ method itself. You need to
consider whether there is any way the thing being
closed could be referenced from somewhere else.

If that's so, the __del__ method may be performing
a useful service by closing it more promptly than
it would otherwise be.

 

[Raymond Hettinger]

[Michele Simionato]

Here and there I hear rumors about deprecating __del__ and
nothing
happens, are there any news about that? Expecially concerning Py3k?

I was writing a Py3K PEP advocating the elimination of __del__
because:

* 'with closing()' and try/finally are the preferred ways of calling
close()

* it is easy to accidently keep or introduce a reference that blocks
the __del__ logic

* under-the-hood, the implementation of __del__ clashes badly with GC
logic and is a maintenance nightmare

* the body of a __del__ method may unintentionally resurrect an object
that was in the process of being deleted

For the PEP to have a chance, I neede to make build-outs to the
weakref module so that existing use cases for __del__ can be easily
migrated. That hasn't been done yet, so the campaign to eliminate
__del__ is stalled.

I should mention that if you search comp.lang.python
for __del__ you will find hundreds of people who were
bitten by __del__, so I usually give advices such as
"you should never __del__ in your code"

Good advice. Explicit finalization is almost always preferable.



Raymond Hettinger

 

[Steven Bethard]

[Michele Simionato]

Here and there I hear rumors about deprecating __del__ and
nothing
happens, are there any news about that? Expecially concerning Py3k?

I was writing a Py3K PEP advocating the elimination of __del__
because:

* 'with closing()' and try/finally are the preferred ways of calling
close()

* it is easy to accidently keep or introduce a reference that blocks
the __del__ logic

* under-the-hood, the implementation of __del__ clashes badly with GC
logic and is a maintenance nightmare

* the body of a __del__ method may unintentionally resurrect an object
that was in the process of being deleted

For the PEP to have a chance, I neede to make build-outs to the
weakref module so that existing use cases for __del__ can be easily
migrated. That hasn't been done yet, so the campaign to eliminate
__del__ is stalled.

There were also a few recipes posted during this discussion that wrap
weakrefs up a bit nicer so it's easier to use them in place of __del__:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/519635
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/519610
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/519621

Since they're using weakrefs, you shouldn't have to worry about some of
the weirdness of __del__.

STeVe

 

[Michele Simionato]

There were also a few recipes posted during this discussion that wrap
weakrefs up a bit nicer so it's easier to use them in place of __del__:

http://aspn.activestate.com/ASPN/Co...estate.com/ASPN/Cookbook/Python/Recipe/519621

Since they're using weakrefs, you shouldn't have to worry about some of
the weirdness of __del__.

STeVe

Then I will mention my own contribution to the cookbook,
which does not use weak references, but just atexit:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/523007

 

[Michele Simionato]

There were also a few recipes posted during this discussion that wrap
weakrefs up a bit nicer so it's easier to use them in place of __del__:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/519635

I knew about your recipe and I thought it was clever, *very*
clever. I have been thinking about it a bit more today and
here is an alternative, which does not require metaclasses,
does not require descriptors, and does not require the user to
declare the attributes she will use in the __finalize__ method.
Warning: it is no more tested than the test case in your recipe:

import weakref

_get = object.__getattribute__
_set = object.__setattr__
_del = object.__delattr__

def getinnerobj(self):
return _get(self, '*innerobj*')

class Impostor(object):
"It tries very hard to impersonate the inner object"
def __init__(self, obj):
_set(self, '*innerobj*', obj)
def __getattribute__(self, name):
return getattr(getinnerobj(self), name)
def __setattr__(self, name, value):
_set(getinnerobj(self), name, value)
def __delattr__(self, name):
_del(getinnerobj(self), name)

_refset = set()

class Finalized(object):
def __new__(cls, *args, **kw):
self = super(Finalized, cls).__new__(cls, *args, **kw)
self.__init__(*args, **kw)
def finalize(ref, refset=_refset):
refset.remove(ref)
cls.__finalize__(self)
fake = Impostor(self)
_refset.add(weakref.ref(fake, finalize))
return fake
def __finalize__(self):
pass

Hope you will enjoy it,

Michele Simionato

 

[Kay Schluehr]

P.S. BTW, I should mention that if you search comp.lang.python for
__del__
you will find hundreds of people who were bitten by __del__, so I
usually give
advices such as "you should never __del__ in your code". If I am wrong
in
giving this advice, please let me know!

I wish we had constructive proofs of cycle freedom and appropriate
warnings - much more then __del__ freedom [1].

Well, we discussed a retro computing example yesterday where some guy
needed to close a generator and release resources, still working with
Python 2.4.

http://groups.google.com/group/comp.lang.python/browse_frm/thread/3f789096bded17e7/?hl=en#

We did it both wrong initially and constructed a reference cycle. Now
I keep my fingers crossed that my most recent solution using __del__
works, because I like it

[1] While I'm using Python on and off I feel subconsciously attracted
by the latest Haskell mania and start to dream alike. Maybe I should
visit a shrink or a shaman?

 

[Steven Bethard]

I knew about your recipe and I thought it was clever, *very*
clever. I have been thinking about it a bit more today and
here is an alternative, which does not require metaclasses,
does not require descriptors, and does not require the user to
declare the attributes she will use in the __finalize__ method.
Warning: it is no more tested than the test case in your recipe:

import weakref

_get = object.__getattribute__
_set = object.__setattr__
_del = object.__delattr__

def getinnerobj(self):
return _get(self, '*innerobj*')

class Impostor(object):
"It tries very hard to impersonate the inner object"
def __init__(self, obj):
_set(self, '*innerobj*', obj)
def __getattribute__(self, name):
return getattr(getinnerobj(self), name)
def __setattr__(self, name, value):
_set(getinnerobj(self), name, value)
def __delattr__(self, name):
_del(getinnerobj(self), name)

_refset = set()

class Finalized(object):
def __new__(cls, *args, **kw):
self = super(Finalized, cls).__new__(cls, *args, **kw)
self.__init__(*args, **kw)
def finalize(ref, refset=_refset):
refset.remove(ref)
cls.__finalize__(self)
fake = Impostor(self)
_refset.add(weakref.ref(fake, finalize))
return fake
def __finalize__(self):
pass

The problem here is that __getattribute__ doesn't work for things like
__getitem__. So the fake object can't properly delegate these::
... def __getitem__(self, key):
... return key
...

>>> c = C()
>>> c[1]

Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
TypeError: 'Impostor' object is unindexable

AFAIK, with new-style classes, there is no way to do this kind of
delegation properly.

STeVe

 

[Michele Simionato]

The problem here is that __getattribute__ doesn't work for things like
__getitem__.

Yeah, it is the old issue of special methods being special, i.e.
len(x)
is not exactly x.__len__(). I am not sure if this can be changed in
Py3k,
it is certainly annoying in some (rare) circumstances, as
here.
One workaround is to add all special methods to the Impostor
class:

SPECIALMETHODS = ['__%s__' % name for name in
'''
abs add and call concat contains delitem delslice div eq floordiv ge
getitem
getslice gt iadd iand iconcat idiv ifloordiv ilshift imod imul index
inv invert
ior ipow irepeat irshift isub iter itruediv ixor le len lshift lt mod
mul ne neg
not or pos pow repeat repr rshift setitem setslice str sub truediv
xor
'''.split()]

def add_special_method(cls,
name): def
meth(self,
*args):
return getattr(self, name)
(*args)
meth.__name__ =
name
setattr(cls, name,
meth)
for name in
SPECIALMETHODS:
add_special_method(Impostor,
name)

In this way the Impostor can emulate even the special methods. Here is
an
example:

....
pass
....

<__main__.C object at
0x102a390>

Notice that the impostor is calling the __str__ method of C, so it
really
looks like a C object.
Moreover

<class
'__main__.C'>

and

True

so the Impostor is doing a damn pretty good job of imposture for C
objects.
Of course it does what it can, and it cannot impersonate C objects
completely:<class
'__main__.Impostor'>

so code using checks like ``type(c) is C`` would break and the
approach here
cannot be considered more than a hack. Still a cool hack, I would
say

Michele Simionato

 

[Steven Bethard]

SPECIALMETHODS = ['__%s__' % name for name in
'''
abs add and call concat contains delitem delslice div eq floordiv ge
getitem
getslice gt iadd iand iconcat idiv ifloordiv ilshift imod imul index
inv invert
ior ipow irepeat irshift isub iter itruediv ixor le len lshift lt mod
mul ne neg
not or pos pow repeat repr rshift setitem setslice str sub truediv
xor
'''.split()]

def add_special_method(cls, name):
def meth(self, *args):
return getattr(self, name)(*args)
meth.__name__ = name
setattr(cls, name, meth)
for name in SPECIALMETHODS:
add_special_method(Impostor, name)

In this way the Impostor can emulate even the special methods. Here is
an example:

...
pass
...

<__main__.C object at 0x102a390>

Notice that the impostor is calling the __str__ method of C, so it
really looks like a C object. Moreover
<class '__main__.C'>

and
True

so the Impostor is doing a damn pretty good job of imposture for C
objects. Of course it does what it can, and it cannot impersonate C
objects completely:
<class '__main__.Impostor'>

so code using checks like ``type(c) is C`` would break and the
approach here cannot be considered more than a hack.

That's fun. =)

I don't really think it's too much of a hack either. ``type(c) is C`` is
expected to fail for proxy classes, which is why everyone keeps getting
told to use ``isinstance(c, C)`` instead. In Py3K, when ``isinstance``
becomes overloadable, it's going to be even more of a mistake to write
``type(c) is C``.

That said, I doubt I'd want to slow down all attribute access on my
class just to do some cleanup, when it's probably better to just tell
everyone to use a ``with`` block. ;-)

STeVe

 

[Michele Simionato]

I doubt I'd want to slow down all attribute access on my
class just to do some cleanup, when it's probably better to just tell
everyone to use a ``with`` block. ;-)

Amen. The point is that finding an easy upgrade path for current code
using __del__ does not look easy at all. Maybe we should be bold,
remove __del__ in Py3K, say to everybody to use 'with', and force
people to rewrite their code, ignoring the complaints from the
big frameworks guys (anyway, they are already complaining


Michele Simionato