Need advice on finding memory leak

Discussion in 'Python' started by mark.engelberg@gmail.com, Oct 16, 2005.

  1. Guest

    I am having trouble identifying the source of a memory leak in a
    Windows Python program. The basic gist is as follows:
    1. Generate a directed graph (approx. 1000 nodes).
    2. Write the graph to a file.
    3. Use the os.system command to invoke another program which processes
    the graph file (graphViz), and generates a gif image of the graph.
    4. Use another os.system command to delete the intermediate file, which
    is no longer needed.
    5. Append to a global list variable some summary information about the
    graph (a few floats, a few ints, and a couple of short strings).
    6. Repeat the above steps several thousand times.
    7. Sort the global summary list.
    8. Write the summary information out to a file.

    When running this program, Python consumes all memory after about a
    thousand graphs.

    I have confirmed the following:
    1. I am closing all files that I open.
    2. Although within the directed graph, nodes refer back and forth to
    each other, the main object which manages the directed graph is not
    part of anything cyclical. It's reference count is in fact going to
    zero when the graph goes out of scope.
    3. After running gc.collect(), I have confirmed that the nodes are not
    in the garbage list. I interpret this to mean that although their may
    be cyclical references among them, the collector was able to determine
    that they were unreachable as a group, and successfully collected them.
    4. I am generating all these graphs in a for loop (using xrange), not
    through a recursive process that would cause the stack to grow.

    So really the only thing that should be growing here is the global list
    that maintains summary information, and since I'm only adding little
    bits of data, it's hard to imagine how this could be exhausting 2GB of
    memory after just a thousand iterations.

    Can anyone help me think of other possible sources of a leak? Are
    there any classic "python-gotchas" I'm missing here, such as something
    leaking from calling os.system so many times? Any other suggestions as
    to what strategies I could employ to track down the leak?

    Thanks,

    Mark
    , Oct 16, 2005
    #1
    1. Advertising

  2. In message <>,
    writes
    >I am having trouble identifying the source of a memory leak in a
    >Windows Python program. The basic gist is as follows:


    Perhaps Python Memory Validator can help you?

    http://www.softwareverify.com/beta.php

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
    Stephen Kellett, Oct 16, 2005
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?TGlvcg==?=

    Desperately need help finding connection leak

    =?Utf-8?B?TGlvcg==?=, Oct 30, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    4,647
    David Browne
    Oct 30, 2004
  2. s.subbarayan

    Dynamic memory allocation and memory leak...

    s.subbarayan, Mar 18, 2005, in forum: C Programming
    Replies:
    10
    Views:
    679
    Eric Sosman
    Mar 22, 2005
  3. rupert.thurner

    finding memory leak in edgewall trac 0.11

    rupert.thurner, Jan 19, 2008, in forum: Python
    Replies:
    10
    Views:
    475
    rupert.thurner
    Feb 2, 2008
  4. cham
    Replies:
    5
    Views:
    756
  5. Mark Probert
    Replies:
    4
    Views:
    317
    Mark Probert
    Feb 9, 2005
Loading...

Share This Page