python memory analysis

Discussion in 'Python' started by Alexander Hoffmann, Sep 6, 2004.

  1. Hello,

    I wrote a deamon like program in python and after a while I realized that it
    consumes more and more memory over time. So my question is: How to debug
    memory leaks in python ?
    Of course I found the built-in profiler but it only helps to make
    performance analysis. Is there a way (or a tools) to view all the instances
    that are kept im memory by the interpreter ?


    Thanks a lot,
    Alex
     
    Alexander Hoffmann, Sep 6, 2004
    #1
    1. Advertising

  2. Alexander Hoffmann <> wrote:

    > Hello,
    >
    > I wrote a deamon like program in python and after a while I realized that it
    > consumes more and more memory over time. So my question is: How to debug
    > memory leaks in python ?
    > Of course I found the built-in profiler but it only helps to make
    > performance analysis. Is there a way (or a tools) to view all the instances
    > that are kept im memory by the interpreter ?


    It's actually quite a problem, sigh. gc.get_objects() gives you a list
    of all objects tracked by the garbage collector (pretty close to, "all
    the objects" period) but that's just the beginning, now you have to find
    out what each object IS, why is it around (functions get_referents and
    get_referrers can help), and hardest of all how much memory that
    particular object IS actually consuming.

    How to estimate the latter is a headache. E.g. consider a list of
    integers. A list with a 1,000,000 items which are all worth 1438397
    presumably has a single copy of that particular int shared by all slots
    (but that's not certain... you'd need to check for that) so you're
    paying for a million slots plus one int (and there is no clear doc as to
    how much that means in bytes on a given platform, btw). If the items
    are all different then you're paying for a million slots plus a million
    ints (unless some of those ints are actually also used and still needed
    elsewhere).

    Still I really wish I had a call gc.memory_howmuch(objs) returning a
    tuple of two ints X <= Y, telling me that the memory taken up by all the
    objs together, the memory I could get back for other purposes if all
    references to all of the objs went away, is somewhere between X and Y
    bytes -- i.e. if all of the objs went away I'd save at least X bytes but
    surely no more than Y. How small one could make Y-X, how tight the
    bounds could be, may be harder to gauge, but it may depend on how much
    one wants to invest in digging deep down...


    Alex
     
    Alex Martelli, Sep 6, 2004
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. qazmlp
    Replies:
    3
    Views:
    670
    Robert Olofsson
    Jan 7, 2004
  2. Alexander Hoffmann

    Re: python memory analysis

    Alexander Hoffmann, Sep 6, 2004, in forum: Python
    Replies:
    3
    Views:
    595
    Chris Liechti
    Sep 6, 2004
  3. Moritz Wissenbach

    Memory Profiler for Heap Analysis

    Moritz Wissenbach, Jun 13, 2007, in forum: Java
    Replies:
    14
    Views:
    1,877
    Moritz Wissenbach
    Jun 20, 2007
  4. ssubbarayan
    Replies:
    5
    Views:
    2,336
    Dave Hansen
    Nov 3, 2009
  5. JTP PR
    Replies:
    0
    Views:
    294
    JTP PR
    Jun 16, 2010
Loading...

Share This Page