Re: python memory analysis

Discussion in 'Python' started by Alexander Hoffmann, Sep 6, 2004.

  1. On Monday 06 September 2004 17:05, Miki Tebeka wrote:
    Hi Miki,
    Thanks for your reply !

    > Hello Alexander,
    >
    > > I wrote a deamon like program in python and after a while I realized that
    > > it consumes more and more memory over time. So my question is: How to
    > > debug memory leaks in python ?

    >
    > Try the gc module.


    You probably mean to utilize gc.get_objects and build some kind of statics
    upon it, don't you ?
    Ok, so this means there is indeed no tool which will help me with that ? :-(

    -alex
     
    Alexander Hoffmann, Sep 6, 2004
    #1
    1. Advertising

  2. On Mon, 06 Sep 2004 16:51:09 +0200, Alexander Hoffmann wrote:
    > You probably mean to utilize gc.get_objects and build some kind of statics
    > upon it, don't you ?
    > Ok, so this means there is indeed no tool which will help me with that ? :-(
    >
    > -alex


    So far, every time I've had a memory leak in Python, simply running the
    process until it leaks for a while, then looking at gc.get_objects(), has
    made it immediately clear what was being leaked. (Regretfully, I have
    never been able to isolate a test case for any of these leaks.) You
    usually see a bajillion instances of some class and that has always been
    enough of a clue.

    Due to how Python works, I'm not sure it is even possible to create a tool
    that does any better than that. It is meaningful to talk about a C
    function that "leaks memory" by allocating and never deallocating, but it
    is much trickier to assign "blame" in Python in a way that could ever have
    meaning to a tool. If the object is left lying around it is already
    because the computer believes somebody has a reference to it; how is the
    computer to "know" this reference isn't "real"? (Assuming no bugs in
    Python of course, but even so you're left with the same problem.)
     
    Jeremy Bowers, Sep 6, 2004
    #2
    1. Advertising

  3. Hello Jeremy,
    Thanks for your help !

    Indeed a first invocation of gc.get_objects already gave me an idea about what
    might go wrong. It seems like the garbage collector does not remove objects a
    and b if a references b even though no one else references either a or b.

    I will continue my analysis.

    Thanks,
    Alex

    On Monday 06 September 2004 17:38, Jeremy Bowers wrote:
    > On Mon, 06 Sep 2004 16:51:09 +0200, Alexander Hoffmann wrote:
    > > You probably mean to utilize gc.get_objects and build some kind of
    > > statics upon it, don't you ?
    > > Ok, so this means there is indeed no tool which will help me with that ?
    > > :-(
    > >
    > > -alex

    >
    > So far, every time I've had a memory leak in Python, simply running the
    > process until it leaks for a while, then looking at gc.get_objects(), has
    > made it immediately clear what was being leaked. (Regretfully, I have
    > never been able to isolate a test case for any of these leaks.) You
    > usually see a bajillion instances of some class and that has always been
    > enough of a clue.
    >
    > Due to how Python works, I'm not sure it is even possible to create a tool
    > that does any better than that. It is meaningful to talk about a C
    > function that "leaks memory" by allocating and never deallocating, but it
    > is much trickier to assign "blame" in Python in a way that could ever have
    > meaning to a tool. If the object is left lying around it is already
    > because the computer believes somebody has a reference to it; how is the
    > computer to "know" this reference isn't "real"? (Assuming no bugs in
    > Python of course, but even so you're left with the same problem.)
     
    Alexander Hoffmann, Sep 6, 2004
    #3
  4. Alexander Hoffmann <> wrote in
    news::

    > Hello Jeremy,
    > Thanks for your help !
    >
    > Indeed a first invocation of gc.get_objects already gave me an idea
    > about what might go wrong. It seems like the garbage collector does
    > not remove objects a and b if a references b even though no one else
    > references either a or b.


    the GC cant remove cycles with objects that contain a __del__ method.
    non cycles are removed by reference counting anayway.

    either dont make circles or don't use __del__, maybe the weakref module can
    be of some use too.

    chris


    > I will continue my analysis.
    >
    > Thanks,
    > Alex
    >
    > On Monday 06 September 2004 17:38, Jeremy Bowers wrote:
    >> On Mon, 06 Sep 2004 16:51:09 +0200, Alexander Hoffmann wrote:
    >> > You probably mean to utilize gc.get_objects and build some kind of
    >> > statics upon it, don't you ?
    >> > Ok, so this means there is indeed no tool which will help me with
    >> > that ?
    >> > :-(
    >> >
    >> > -alex

    >>
    >> So far, every time I've had a memory leak in Python, simply running
    >> the process until it leaks for a while, then looking at
    >> gc.get_objects(), has made it immediately clear what was being
    >> leaked. (Regretfully, I have never been able to isolate a test case
    >> for any of these leaks.) You usually see a bajillion instances of
    >> some class and that has always been enough of a clue.
    >>
    >> Due to how Python works, I'm not sure it is even possible to create a
    >> tool that does any better than that. It is meaningful to talk about a
    >> C function that "leaks memory" by allocating and never deallocating,
    >> but it is much trickier to assign "blame" in Python in a way that
    >> could ever have meaning to a tool. If the object is left lying around
    >> it is already because the computer believes somebody has a reference
    >> to it; how is the computer to "know" this reference isn't "real"?
    >> (Assuming no bugs in Python of course, but even so you're left with
    >> the same problem.)

    >
    >




    --
    Chris <>
     
    Chris Liechti, Sep 6, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. qazmlp
    Replies:
    3
    Views:
    670
    Robert Olofsson
    Jan 7, 2004
  2. Alexander Hoffmann

    python memory analysis

    Alexander Hoffmann, Sep 6, 2004, in forum: Python
    Replies:
    1
    Views:
    321
    Alex Martelli
    Sep 6, 2004
  3. Moritz Wissenbach

    Memory Profiler for Heap Analysis

    Moritz Wissenbach, Jun 13, 2007, in forum: Java
    Replies:
    14
    Views:
    1,878
    Moritz Wissenbach
    Jun 20, 2007
  4. ssubbarayan
    Replies:
    5
    Views:
    2,336
    Dave Hansen
    Nov 3, 2009
  5. JTP PR
    Replies:
    0
    Views:
    294
    JTP PR
    Jun 16, 2010
Loading...

Share This Page