Re: How to measure the memory cost in Python?

Discussion in 'Python' started by Chris Rebert, May 1, 2009.

  1. Chris Rebert

    Chris Rebert Guest

    On Fri, May 1, 2009 at 1:03 AM, Li Wang <> wrote:
    > Hi everyone:
    >
    > I want to measure the actual memory cost of a particular step in my program
    > (Python program), does anyone know if there is some function in Python could
    > help me to do this job? Or should I seek other tools to help me?


    See Gabriel's answer in your other thread "Measure the memory cost in Python".
    And please don't post duplicate threads asking essentially the same
    question, especially when you were given a perfectly acceptable answer
    and not that much time has past since you started the other thread.

    Cheers,
    Chris
    --
    http://blog.rebertia.com
     
    Chris Rebert, May 1, 2009
    #1
    1. Advertising

  2. Chris Rebert

    CTO Guest

    Not OP, but I'd actually like to know if there's an answer to this one
    that doesn't involve platform-specific tools.
     
    CTO, May 1, 2009
    #2
    1. Advertising

  3. Chris Rebert

    Chris Rebert Guest

    On Fri, May 1, 2009 at 7:54 AM, CTO <> wrote:
    > Not OP, but I'd actually like to know if there's an answer to this one
    > that doesn't involve platform-specific tools.


    sys.getsizeof() [a suggested solution] isn't platform-specific.

    Cheers,
    Chris
    --
    http://blog.rebertia.com
     
    Chris Rebert, May 1, 2009
    #3
  4. Chris Rebert

    Jean Guest

    On May 1, 7:54 am, CTO <> wrote:
    > Not OP, but I'd actually like to know if there's an answer to this one
    > that doesn't involve platform-specific tools.


    Depending on what you need and the O/S you are using, this recipe may
    help

    <http://code.activestate.com/recipes/286222/>

    That recipe also appeared in the 2nd edition of the Python Cookbook,
    see

    <http://books.google.com/books?
    id=Q0s6Vgb98CQC&printsec=frontcover&dq=editions:ISBN0596001673#PPA334,M1>

    /Jean Brouwers
     
    Jean, May 1, 2009
    #4
  5. Chris Rebert

    CTO Guest

    > sys.getsizeof() [a suggested solution] isn't platform-specific.

    So, to answer the OP's question, you'd just do something like

    def get_totalsize(obj):
    total_size = sys.getsizeof(obj)
    for value in vars(obj).values():
    try: total_size += get_total_size(value)
    except: total_size += sys.getsizeof(value)
    return totalSize

    def get_current_size(env):
    size = 0
    for value in env.values():
    try: size += get_total_size(value)
    except: pass
    return size

    get_current_size(vars())

    and discount the weight of the interpreter?
     
    CTO, May 1, 2009
    #5
  6. Chris Rebert

    Jean Guest

    On May 1, 10:56 am, CTO <> wrote:
    > > sys.getsizeof() [a suggested solution] isn't platform-specific.

    >
    > So, to answer the OP's question, you'd just do something like
    >
    > def get_totalsize(obj):
    >         total_size = sys.getsizeof(obj)
    >         for value in vars(obj).values():
    >                 try: total_size += get_total_size(value)
    >                 except: total_size += sys.getsizeof(value)
    >         return totalSize
    >
    > def get_current_size(env):
    >         size = 0
    >         for value in env.values():
    >                 try: size += get_total_size(value)
    >                 except: pass
    >         return size
    >
    > get_current_size(vars())
    >
    > and discount the weight of the interpreter?


    Keep in mind, sys.getsizeof(obj) returns only the size of the given
    object. Any referenced objects are not included. You can get the
    latter from gc.get_referents(obj).

    /Jean Brouwers

    PS) The asizeof(obj) function from this recipe <http://
    code.activestate.com/recipes/546530> does size the object plus its
    references, recursively.
     
    Jean, May 1, 2009
    #6
  7. Chris Rebert

    Jean Guest

    On May 1, 12:50 pm, Jean <> wrote:
    > On May 1, 10:56 am, CTO <> wrote:
    >
    >
    >
    > > > sys.getsizeof() [a suggested solution] isn't platform-specific.

    >
    > > So, to answer the OP's question, you'd just do something like

    >
    > > def get_totalsize(obj):
    > >         total_size = sys.getsizeof(obj)
    > >         for value in vars(obj).values():
    > >                 try: total_size += get_total_size(value)
    > >                 except: total_size += sys.getsizeof(value)
    > >         return totalSize

    >
    > > def get_current_size(env):
    > >         size = 0
    > >         for value in env.values():
    > >                 try: size += get_total_size(value)
    > >                 except: pass
    > >         return size

    >
    > > get_current_size(vars())

    >
    > > and discount the weight of the interpreter?

    >
    > Keep in mind, sys.getsizeof(obj) returns only the size of the given
    > object.  Any referenced objects are not included.  You can get the
    > latter from gc.get_referents(obj).
    >
    > /Jean Brouwers
    >
    > PS) The asizeof(obj) function from this recipe <http://
    > code.activestate.com/recipes/546530> does size the object plus its
    > references, recursively.


    Correction, the last sentence should be: The asizeof(obj) ... plus its
    referents, recursively.
     
    Jean, May 2, 2009
    #7
  8. Chris Rebert

    CTO Guest

    > > PS) The asizeof(obj) function from this recipe <http://
    > > code.activestate.com/recipes/546530> does size the object plus its
    > > references, recursively.

    >
    > Correction, the last sentence should be: The asizeof(obj) ... plus its
    > referents, recursively.


    I will admit, I have *no idea* what that code is doing, but in looking
    through the gc module documentation, I'm seeing the gc.get_objects
    function. Would it be equivalent to what the OP is asking to track the
    size of every element returned by that?
     
    CTO, May 2, 2009
    #8
  9. Chris Rebert

    Aahz Guest

    In article <>,
    CTO <> wrote:
    >
    >I will admit, I have *no idea* what that code is doing, but in looking
    >through the gc module documentation, I'm seeing the gc.get_objects
    >function. Would it be equivalent to what the OP is asking to track the
    >size of every element returned by that?


    Perhaps. gc.get_objects() only returns objects tracked by GC (i.e.
    containers). You would need to also check all the object references
    held by the containers.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Typing is cheap. Thinking is expensive." --Roy Smith
     
    Aahz, May 3, 2009
    #9
  10. Chris Rebert

    CTO Guest

    Alright, it's pretty obvious that I have a lot to learn before I'll be
    able
    to intelligently address this problem, but if somebody could point me
    at
    something that would help me figure out the terminology at least I'd
    really
    appreciate it. From what you're saying, it sounds like a combination
    of the
    above approaches would do what I'm asking- ie, get all the containers,
    then
    get the contents of each container- but I don't see why that would
    work
    unless gc tracks some kind of global container for small values,
    which, as
    I (poorly) understand it, are tracked separately?

    Thanks again,
    Geremy Condra
     
    CTO, May 3, 2009
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tom L

    how to measure memory consumption

    Tom L, Apr 13, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    478
    bruce barker
    Apr 14, 2004
  2. Guadala Harry

    Measure Memory Used By ASP.NET Application

    Guadala Harry, Oct 15, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    590
    Kevin Spencer
    Oct 15, 2004
  3. Rune Strand

    Measure memory usage in Python

    Rune Strand, Apr 30, 2006, in forum: Python
    Replies:
    3
    Views:
    2,382
    gene tani
    May 1, 2006
  4. Neagu, Adrian
    Replies:
    3
    Views:
    409
    Heikki Toivonen
    Sep 21, 2006
  5. Fredrik Lundh
    Replies:
    1
    Views:
    368
    AdrianNg3
    Sep 20, 2006
Loading...

Share This Page