Possible to assure no "cyclic"/"uncollectible" memory leaks?

Discussion in 'Python' started by Joe Peterson, Dec 2, 2006.

  1. Joe Peterson

    Joe Peterson Guest

    I've been doing a lot of searching on the topic of one of Python's more
    disturbing issues (at least to me): the fact that if a __del__ finalizer
    is defined and a cyclic (circular) reference is made, the garbage
    collector cannot clean it up.

    First of all, it seems that it's best to avoid using __del__. So far, I
    have never used it in my Python programming. So I am safe there. Or am
    I? Also, to my knowledge, I have never created a cyclic reference, but
    we do not typically create bugs intentionally either (and there are
    certainly times when it is an OK thing to do).

    Still, it's not comforting to know that it is possible to create a
    situation that would create a memory leak using a language that is
    supposed to relieve us of that worry. I understand the problem, but it
    would be nice to know that as a programmer, I could be assured that
    Python would always deal with memory management and that memory leaks
    were not something I had to think about.

    So here's a question: if I write Python software and never use __del__,
    can I guarantee that there is no way to create a memory leak? What
    about system libraries - do any of them use __del__, and if so, are they
    written in such a way that it is not possible to create a cyclic reference?

    Thanks, Joe
    Joe Peterson, Dec 2, 2006
    #1
    1. Advertising

  2. Joe Peterson

    Klaas Guest

    Joe Peterson wrote:
    > I've been doing a lot of searching on the topic of one of Python's more
    > disturbing issues (at least to me): the fact that if a __del__ finalizer
    > is defined and a cyclic (circular) reference is made, the garbage
    > collector cannot clean it up.


    It is a somewhat fundamental limitation of GCs, if you want to support:

    1. __del__ that can resurrect objects and is deterministically called
    when objects are destroyed
    2. the "view" of alive objects by __del__ methods is consistent
    3. no crashing

    If there is a cycle of objects containing __del__ methods, there is
    clearly no way of knowing a safe order of invoking them.

    > First of all, it seems that it's best to avoid using __del__. So far, I
    > have never used it in my Python programming. So I am safe there. Or am
    > I? Also, to my knowledge, I have never created a cyclic reference, but
    > we do not typically create bugs intentionally either (and there are
    > certainly times when it is an OK thing to do).


    It is good practice to avoid __del__ unless there is a compelling
    reason to do so. weakref resource management is much safer. Note that
    it is pretty much impossible to avoid creating reference cycles--they
    have a tendency to sneak into unsuspecting places (for instance, bound
    methods can be a subtle source of cycles).

    > Still, it's not comforting to know that it is possible to create a
    > situation that would create a memory leak using a language that is
    > supposed to relieve us of that worry. I understand the problem, but it
    > would be nice to know that as a programmer, I could be assured that
    > Python would always deal with memory management and that memory leaks
    > were not something I had to think about.


    It is unrealistic to ever be completely relieved of such worry, since
    it is always possible to accidently hold on to a strong reference to
    data that should actually be "garbage". But your question is perhaps
    precluding these kinds of memory leak. In that case, it is a matter of
    providing to the programmer sufficiently-fine-grained abstractions such
    that the compiler can reason about their safety. For instance, an
    included weakref-based resource cleanup scheme has been discussed and
    would cover many of the current uses of __del__. It would also be nice
    to remove some of the hidden "gotchas" that are inherent in CPython,
    like the integer and float object freelist (not necessarily removing
    those features, but providing some mechanism for reclaiming them when
    they get out of hand).

    These things can reduce the possibility of a problem, but (IMO) can
    never completely obviate it.

    > So here's a question: if I write Python software and never use __del__,
    > can I guarantee that there is no way to create a memory leak? What
    > about system libraries - do any of them use __del__, and if so, are they
    > written in such a way that it is not possible to create a cyclic reference?


    It is always possible to create a cyclic reference by monkeypatching a
    class. Here are the stdlib modules which use __del__:
    $ find -name \*.py | xargs grep __del__ | grep -v test
    ../Mac/Demo/sound/morselib.py: def __del__(self):
    ../Lib/telnetlib.py: def __del__(self):
    ../Lib/plat-mac/EasyDialogs.py: def __del__(self):
    ../Lib/plat-mac/FrameWork.py: def __del__(self):
    ../Lib/plat-mac/MiniAEFrame.py: def __del__(self):
    ../Lib/plat-mac/Audio_mac.py: def __del__(self):
    ../Lib/plat-mac/videoreader.py: def __del__(self):
    ../Lib/fileinput.py: def __del__(self):
    ../Lib/subprocess.py: def __del__(self):
    ../Lib/gzip.py: def __del__(self):
    ../Lib/wave.py: def __del__(self):
    ../Lib/wave.py: def __del__(self):
    ../Lib/popen2.py: def __del__(self):
    ../Lib/lib-tk/Tkdnd.py: def __del__(self):
    ../Lib/lib-tk/tkFont.py: def __del__(self):
    ../Lib/lib-tk/Tkinter.py: def __del__(self):
    ../Lib/lib-tk/Tkinter.py: def __del__(self):
    ../Lib/urllib.py: def __del__(self):
    ../Lib/tempfile.py: # __del__ is called.
    ../Lib/tempfile.py: def __del__(self):
    ../Lib/tarfile.py: def __del__(self):
    ../Lib/socket.py: def __del__(self):
    ../Lib/zipfile.py: fp = None # Set here since
    __del__ checks it
    ../Lib/zipfile.py: def __del__(self):
    ../Lib/httplib.py: def __del__(self):
    ../Lib/bsddb/dbshelve.py: def __del__(self):
    ../Lib/bsddb/dbshelve.py: def __del__(self):
    ../Lib/bsddb/__init__.py: def __del__(self):
    ../Lib/bsddb/dbtables.py: def __del__(self):
    ../Lib/idlelib/MultiCall.py: def __del__(self):
    ../Lib/idlelib/MultiCall.py: def __del__(self):
    ../Lib/idlelib/MultiCall.py: def __del__(self):
    ../Lib/sunau.py: def __del__(self):
    ../Lib/sunau.py: def __del__(self):
    ../Lib/poplib.py: #__del__ = quit
    ../Lib/_threading_local.py: def __del__(self):
    ../Lib/aifc.py: def __del__(self):
    ../Lib/dumbdbm.py: # gets called. One place _commit() gets called is
    from __del__(),
    ../Lib/dumbdbm.py: # be called from __del__(). Therefore we must
    never reference a
    ../Lib/dumbdbm.py: __del__ = close
    ../Lib/wsgiref/validate.py: def __del__(self):
    ../Lib/shelve.py: def __del__(self):
    ../Lib/cgi.py: terminates, try defining a __del__ method in a
    derived class
    ../Lib/platform.py: __del__ = close
    ../Lib/audiodev.py: def __del__(self):
    ../Lib/audiodev.py: def __del__(self):

    -Mike
    Klaas, Dec 2, 2006
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    14
    Views:
    546
    Christos TZOTZIOY Georgiou
    Apr 28, 2005
  2. Federica
    Replies:
    3
    Views:
    759
  3. moerchendiser2k3

    Python leaks in cyclic garbage collection

    moerchendiser2k3, Feb 19, 2011, in forum: Python
    Replies:
    6
    Views:
    240
  4. thisis
    Replies:
    5
    Views:
    250
    thisis
    Oct 29, 2006
  5. Mark Probert
    Replies:
    4
    Views:
    125
    Ville Mattila
    Feb 15, 2005
Loading...

Share This Page