Memory leak issue with complex data structure

Discussion in 'Python' started by Alan Franzoni, Jul 4, 2007.

  1. Hello,
    I've got a complex data structure (graph-like). Each of my nodes may
    reference zero or more other nodes, internally managed by a set() object.

    I have a root node which is not referenced by any other node. So, I
    created a "clear()" method which is called on all children (by calling
    their clear() method" and then clears the set with the references of the
    node itself.

    I have a serious "leak" issue; even though I clear all those sets and I
    delete all the references I can have to the current namespace, memory is
    not freed.

    I have tried the gc module and it couldn't help (I can't determine where
    the references are) and I tried using weakref.ref as well to connect
    object, but then I have problems (objects have no more references to
    them beside their parents, hence they get deleted).

    I understand it's impossibile to tell what's my problem without seeing
    the actual code (which I can't post) but I'm asking if there's any
    tool/debugger I could employ in order to track what's happening.

    Thanks!

    --
    Alan Franzoni <>
    -
    Togli .xyz dalla mia email per contattarmi.
    Remove .xyz from my address in order to contact me.
    -
    GPG Key Fingerprint (Key ID = FE068F3E):
    5C77 9DC3 BD5B 3A28 E7BC 921A 0255 42AA FE06 8F3E
    Alan Franzoni, Jul 4, 2007
    #1
    1. Advertising

  2. Alan Franzoni wrote:
    > I have a root node which is not referenced by any other node. So, I
    > created a "clear()" method which is called on all children (by calling
    > their clear() method" and then clears the set with the references of the
    > node itself.


    Using the .clear() method on sets (or dictionaries) does not reduce the
    memory that the underlying hash tables use, it only removes references
    to the keys (and values) that the sets (and dictionaries) contain(s).
    If your sets used to be large, I could have sworn that you could force a
    resize down by adding some items to the set, but I seem to be mistaken:

    #memory use is 3440k
    s = set(xrange(1000000))
    #memory use is 31,864k
    s.clear()
    #memory use is 15,460k
    s.add(0)
    s.remove(0)
    #memory use is 15,460k

    Then again, this may be an artifact of my operating system memory
    freeing semantics.

    As is usually the case, read the C source from svn.python.org/view (or
    some other means) for more information.

    - Josiah
    Josiah Carlson, Jul 4, 2007
    #2
    1. Advertising

  3. Alan Franzoni <> writes:

    > I have a serious "leak" issue; even though I clear all those sets
    > and I delete all the references I can have to the current namespace,
    > memory is not freed.


    Maybe the memory is freed (marked as available for further use by
    Python), just not released to the operating system.[1] To test against
    that, try to allocate more Python structures and see if they reuse the
    freed memory or if they allocate even more memory. Even better, run
    code like this:

    while 1:
    ... populate your data structures ...
    clear()

    If this causes Python to allocate more and more memory, it means you
    have a real leak. If not, it means that the GC is working fine, but
    it's not possible to release the memory to the OS.


    [1]
    Not giving freed memory back to the system is not (necessarily) a
    Python bug; the same thing happens in C and is a consequence of
    managed memory being assigned to the process as a contiguous block.
    Hrvoje Niksic, Jul 5, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jeff

    complex data structure

    Jeff, Jun 26, 2004, in forum: Perl
    Replies:
    5
    Views:
    507
  2. Waqqas Farooq

    complex data structure in httpcookie

    Waqqas Farooq, Jun 30, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    336
    avnrao
    Jun 30, 2004
  3. spar
    Replies:
    4
    Views:
    310
    Alex Martelli
    Oct 11, 2004
  4. Sigmathaar
    Replies:
    3
    Views:
    320
    Bob Hairgrove
    Dec 6, 2005
  5. Ram
    Replies:
    3
    Views:
    416
    Barry Schwarz
    Mar 24, 2009
Loading...

Share This Page