Re: Weakref.ref callbacks and eliminating __del__ methods

Discussion in 'Python' started by Mike C. Fletcher, Jan 24, 2005.

  1. Tim Peters wrote:

    >[Mike C. Fletcher]
    >
    >
    >>I'm looking at rewriting parts of Twisted and TwistedSNMP to eliminate
    >>__del__ methods (and the memory leaks they create).
    >>
    >>

    >A worthy goal!
    >
    >

    Well, as of now it seems to have eliminated the last leaks in
    TwistedSNMP, and that's likely going to eliminate the last of our leaks
    in our products, so yes, I suppose it was :) .

    >A callback is strongly referenced from the weakref(s) containing it.
    >
    >

    Thanks.

    >It would really help to give a minimal example of self-contained
    >executable code. I'm not sure how you're using this code, and
    >guessing takes too long.
    >
    >

    Sorry about that. I was expecting a theoretical problem, not a
    practical one, so didn't think to provide a practical example.

    >The "2" there is important, just because this is an interactive
    >session. The point of it was to stop `_` from holding a reference to
    >the created weakref. In your code
    >
    > weakref.ref( self, self.close )
    >
    >the weakref itself becomes trash immediately after it's created, so
    >the weakref is thrown away entirely (including its strong reference to
    >self.close, and its weak reference to self) right after that line
    >ends. Of course callbacks aren't *supposed* to be called when the
    >weakref itself goes away, they're supposed to be called when the thing
    >being weakly referenced goes away. So this all looks vanilla and
    >inevitable to me -- the callback is never invoked because the weakref
    >itself goes away long before self goes away.
    >
    >

    Alex already scooped you on this "smack yourself on the head, Mikey". I
    was assuming the weakref object was a proxy for a reference in an
    internal structure; I was expecting a list of weak references that was
    nothing but a series of functions to call after finalising the object.
    My bad.

    >I'm pretty sure it's just because there's nothing here to keep the
    >weakref itself alive after it's created. You could try, e.g.,
    >
    > self.wr = weakref.ref( self, self.close )
    >
    >to keep it alive, but things get fuzzy then if self becomes part of
    >cyclic trash.
    >
    >

    Yes, so it requires an external storage mechanism and code to clean that
    up... ick. __del__ looks cleaner and cleaner by comparison. You'd
    think I would have noticed the huge tables of weakref objects in
    PyDispatcher as I was working in there...

    >>I can work around it in this particular case by defining a __del__ on
    >>the Closer, but that just fixes this particular instance (and leaves
    >>just as many __del__'s hanging around). I'm wondering if there's a
    >>ready recipe that can *always* replace a __del__'s operation?
    >>
    >>

    >
    >In the presence of cycles it gets tricky; there's a lot of
    >more-or-less recent words (less than a year old <wink>) about that on
    >python-dev. It gets complicated quickly, and it seems nobody can make
    >more time to think about it.
    >
    >

    Sigh, guess I should stop listening to rumours just because they are
    saying something about that for which I yearn.

    >>I know I heard a rumour somewhere about Uncle Timmy wanting to eliminate
    >>__del__ in 2.5 or thereabouts,
    >>
    >>

    >
    >Python 3 at the earliest. That's the earliest everything nobody can
    >make time for lands <0.5 wink>.
    >
    >

    Well, we darn well better solve it by then! Don't want memory leaks
    when we're hard-wired into someone's brain.

    >There are unresolved issues about how to get all this stuff to work
    >sanely. The driving principle behind cyclic-trash weakref endcases
    >right now is "do anything defensible that won't segfault".
    >
    >I'll note that one fairly obvious pattern works very well for weakrefs
    >and __del__ methods (mutatis mutandis): don't put the __del__ method
    >in self, put it in a dead-simple object hanging *off* of self. Like
    >the simple:
    >
    >class BTreeCloser:
    > def __init__(self, btree):
    > self.btree = btree
    >
    > def __del__(self):
    > if self.btree:
    > self.btree.close()
    > self.btree = None
    >
    >

    Yes, this was the "work around" I'd implemented (well, with __call__
    instead and del just calling it). Of course, that didn't actually
    eliminate any __del__ methods, but oh well, if we're not losing those
    until 3.x it doesn't really matter :) .

    The fix for the Deferred in Twisted got a little hideous (there the
    __del__ is poking around into lots of different attributes of the
    Deferred. To hack that I created a descriptor class that forwards a
    variable to the held object and wrapped each of the needed variables in
    one of those... I'm going to have to find something a little less
    baroque if I'm going to get it into Twisted... really, it doesn't look
    like any of it's necessary, it's just logging an error if the Deferred
    ended on a Failure. Something should be done though, leaking memory
    from a core object in the framework is just a PITA.

    >When a weakref and its weak referent are both in cyclic trash, Python
    >currently says "the order in which they die is undefined, so we'll
    >pretend the weakref dies first", and then, as at the start of this
    >msg, the callback is never invoked.
    >

    So external storage of a weakref is basically a requirement to make it
    useful. Hmm.

    >The problem this avoids is that
    >the callback may itself be part of cyclic trash too, in which case
    >it's possible for the callback to resurrect anything else in cyclic
    >trash, including the weakly-referenced object associated with the
    >callback. But by hypothesis in this case, that object is also in
    >cyclic trash, and horrible things can happen if a partially destructed
    >object gets resurrected.
    >
    >

    Reasonable I suppose.

    >So the real rule right now is that, if you want a guarantee that a
    >callback on a weakly referenced object gets invoked, you have to write
    >your code to guarantee that the referent becomes trash before its
    >weakref becomes trash. (And note that if you do, the callback
    >necessarily will be alive too, because a weakref does in fact hold a
    >strong reference to its callback.)
    >
    >

    Guess I'll go back to the sub-object approach for all things and merely
    dream of a far off weakref utopia free of all __del__ methods.

    Thanks, Tim,
    Mike

    ________________________________________________
    Mike C. Fletcher
    Designer, VR Plumber, Coder
    http://www.vrplumber.com
    http://blog.vrplumber.com
    Mike C. Fletcher, Jan 24, 2005
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mike C. Fletcher
    Replies:
    2
    Views:
    291
    Mike C. Fletcher
    Jan 24, 2005
  2. Replies:
    0
    Views:
    324
  3. Replies:
    22
    Views:
    729
    peter koch
    Apr 30, 2008
  4. George Sakkis
    Replies:
    6
    Views:
    291
    George Sakkis
    Jun 12, 2008
  5. Justin C
    Replies:
    1
    Views:
    169
    Justin C
    Oct 7, 2013
Loading...

Share This Page