Re: Weakref.ref callbacks and eliminating __del__ methods

Discussion in 'Python' started by Mike C. Fletcher, Jan 24, 2005.

  1. Tim Peters wrote:

    >[Mike C. Fletcher]
    >>I'm looking at rewriting parts of Twisted and TwistedSNMP to eliminate
    >>__del__ methods (and the memory leaks they create).

    >A worthy goal!

    Well, as of now it seems to have eliminated the last leaks in
    TwistedSNMP, and that's likely going to eliminate the last of our leaks
    in our products, so yes, I suppose it was :) .

    >A callback is strongly referenced from the weakref(s) containing it.


    >It would really help to give a minimal example of self-contained
    >executable code. I'm not sure how you're using this code, and
    >guessing takes too long.

    Sorry about that. I was expecting a theoretical problem, not a
    practical one, so didn't think to provide a practical example.

    >The "2" there is important, just because this is an interactive
    >session. The point of it was to stop `_` from holding a reference to
    >the created weakref. In your code
    > weakref.ref( self, self.close )
    >the weakref itself becomes trash immediately after it's created, so
    >the weakref is thrown away entirely (including its strong reference to
    >self.close, and its weak reference to self) right after that line
    >ends. Of course callbacks aren't *supposed* to be called when the
    >weakref itself goes away, they're supposed to be called when the thing
    >being weakly referenced goes away. So this all looks vanilla and
    >inevitable to me -- the callback is never invoked because the weakref
    >itself goes away long before self goes away.

    Alex already scooped you on this "smack yourself on the head, Mikey". I
    was assuming the weakref object was a proxy for a reference in an
    internal structure; I was expecting a list of weak references that was
    nothing but a series of functions to call after finalising the object.
    My bad.

    >I'm pretty sure it's just because there's nothing here to keep the
    >weakref itself alive after it's created. You could try, e.g.,
    > self.wr = weakref.ref( self, self.close )
    >to keep it alive, but things get fuzzy then if self becomes part of
    >cyclic trash.

    Yes, so it requires an external storage mechanism and code to clean that
    up... ick. __del__ looks cleaner and cleaner by comparison. You'd
    think I would have noticed the huge tables of weakref objects in
    PyDispatcher as I was working in there...

    >>I can work around it in this particular case by defining a __del__ on
    >>the Closer, but that just fixes this particular instance (and leaves
    >>just as many __del__'s hanging around). I'm wondering if there's a
    >>ready recipe that can *always* replace a __del__'s operation?

    >In the presence of cycles it gets tricky; there's a lot of
    >more-or-less recent words (less than a year old <wink>) about that on
    >python-dev. It gets complicated quickly, and it seems nobody can make
    >more time to think about it.

    Sigh, guess I should stop listening to rumours just because they are
    saying something about that for which I yearn.

    >>I know I heard a rumour somewhere about Uncle Timmy wanting to eliminate
    >>__del__ in 2.5 or thereabouts,

    >Python 3 at the earliest. That's the earliest everything nobody can
    >make time for lands <0.5 wink>.

    Well, we darn well better solve it by then! Don't want memory leaks
    when we're hard-wired into someone's brain.

    >There are unresolved issues about how to get all this stuff to work
    >sanely. The driving principle behind cyclic-trash weakref endcases
    >right now is "do anything defensible that won't segfault".
    >I'll note that one fairly obvious pattern works very well for weakrefs
    >and __del__ methods (mutatis mutandis): don't put the __del__ method
    >in self, put it in a dead-simple object hanging *off* of self. Like
    >the simple:
    >class BTreeCloser:
    > def __init__(self, btree):
    > self.btree = btree
    > def __del__(self):
    > if self.btree:
    > self.btree.close()
    > self.btree = None

    Yes, this was the "work around" I'd implemented (well, with __call__
    instead and del just calling it). Of course, that didn't actually
    eliminate any __del__ methods, but oh well, if we're not losing those
    until 3.x it doesn't really matter :) .

    The fix for the Deferred in Twisted got a little hideous (there the
    __del__ is poking around into lots of different attributes of the
    Deferred. To hack that I created a descriptor class that forwards a
    variable to the held object and wrapped each of the needed variables in
    one of those... I'm going to have to find something a little less
    baroque if I'm going to get it into Twisted... really, it doesn't look
    like any of it's necessary, it's just logging an error if the Deferred
    ended on a Failure. Something should be done though, leaking memory
    from a core object in the framework is just a PITA.

    >When a weakref and its weak referent are both in cyclic trash, Python
    >currently says "the order in which they die is undefined, so we'll
    >pretend the weakref dies first", and then, as at the start of this
    >msg, the callback is never invoked.

    So external storage of a weakref is basically a requirement to make it
    useful. Hmm.

    >The problem this avoids is that
    >the callback may itself be part of cyclic trash too, in which case
    >it's possible for the callback to resurrect anything else in cyclic
    >trash, including the weakly-referenced object associated with the
    >callback. But by hypothesis in this case, that object is also in
    >cyclic trash, and horrible things can happen if a partially destructed
    >object gets resurrected.

    Reasonable I suppose.

    >So the real rule right now is that, if you want a guarantee that a
    >callback on a weakly referenced object gets invoked, you have to write
    >your code to guarantee that the referent becomes trash before its
    >weakref becomes trash. (And note that if you do, the callback
    >necessarily will be alive too, because a weakref does in fact hold a
    >strong reference to its callback.)

    Guess I'll go back to the sub-object approach for all things and merely
    dream of a far off weakref utopia free of all __del__ methods.

    Thanks, Tim,

    Mike C. Fletcher
    Designer, VR Plumber, Coder
    Mike C. Fletcher, Jan 24, 2005
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mike C. Fletcher
    Mike C. Fletcher
    Jan 24, 2005
  2. Replies:
  3. Replies:
    peter koch
    Apr 30, 2008
  4. George Sakkis
    George Sakkis
    Jun 12, 2008
  5. Justin C
    Justin C
    Oct 7, 2013

Share This Page