Memory footpring of python objects

Discussion in 'Python' started by BlueBird, Apr 22, 2009.

  1. BlueBird

    BlueBird Guest

    Hi,

    I have a program that manages several thousands instances of one
    object. To reduce memory
    consumption, I want of course that specific object to have the
    smallest memory footpring possible.

    I have a few ideas that I want to experiment with, like using
    __slots__, using a tuple or using a dict. My
    question is: how do I know the memory footprint of a given python
    object ? I could not find any
    builtin functions for this in the documentation.

    cheers,

    Philippe
    BlueBird, Apr 22, 2009
    #1
    1. Advertising

  2. BlueBird

    Chris Rebert Guest

    On Wed, Apr 22, 2009 at 2:24 AM, BlueBird <> wrote:
    >
    > Hi,
    >
    > I have a program that manages several thousands instances of one
    > object. To reduce memory
    > consumption, I want of course that specific object to have the
    > smallest memory footpring possible.
    >
    > I have a few ideas that I want to experiment with, like using
    > __slots__, using a tuple or using a dict. My
    > question is: how do I know the memory footprint of a given python
    > object ? I could not find any
    > builtin functions for this in the documentation.


    sys.getsizeof() - http://docs.python.org/library/sys.html#sys.getsizeof

    Cheers,
    Chris
    --
    I have a blog:
    http://blog.rebertia.com
    Chris Rebert, Apr 22, 2009
    #2
    1. Advertising

  3. On Wed, 22 Apr 2009 02:30:32 -0700, Chris Rebert wrote:

    > On Wed, Apr 22, 2009 at 2:24 AM, BlueBird <> wrote:
    >>
    >> Hi,
    >>
    >> I have a program that manages several thousands instances of one
    >> object. To reduce memory
    >> consumption, I want of course that specific object to have the smallest
    >> memory footpring possible.
    >>
    >> I have a few ideas that I want to experiment with, like using
    >> __slots__, using a tuple or using a dict. My question is: how do I know
    >> the memory footprint of a given python object ? I could not find any
    >> builtin functions for this in the documentation.

    >
    > sys.getsizeof() - http://docs.python.org/library/sys.html#sys.getsizeof


    Only in Python 2.6.

    But if you search Activestate, there's a recipe to do the same thing. Ah,
    here it is:

    http://code.activestate.com/recipes/546530/



    --
    Steven
    Steven D'Aprano, Apr 22, 2009
    #3
  4. BlueBird wrote:

    > I have a program that manages several thousands instances of one
    > object. To reduce memory
    > consumption, I want of course that specific object to have the
    > smallest memory footpring possible.


    Have you thought of using something like the flyweight pattern and a
    compact data representation like a numpy array?
    Marco Mariani, Apr 22, 2009
    #4
  5. BlueBird

    BlueBird Guest

    On 22 avr, 11:56, Steven D'Aprano
    <> wrote:
    > On Wed, 22 Apr 2009 02:30:32 -0700, Chris Rebert wrote:
    > > On Wed, Apr 22, 2009 at 2:24 AM, BlueBird <> wrote:

    >
    > >> Hi,

    >
    > >> I have a program that manages several thousands instances of one
    > >> object. To reduce memory
    > >> consumption, I want of course that specific object to have the smallest
    > >> memory footpring possible.

    >
    > >> I have a few ideas that I want to experiment with, like using
    > >> __slots__, using a tuple or using a dict. My question is: how do I know
    > >> the memory footprint of a given python object ? I could not find any
    > >> builtin functions for this in the documentation.

    >
    > > sys.getsizeof() -http://docs.python.org/library/sys.html#sys.getsizeof

    >
    > Only in Python 2.6.
    >
    > But if you search Activestate, there's a recipe to do the same thing. Ah,
    > here it is:
    >
    > http://code.activestate.com/recipes/546530/


    Thanks. "There is always a tool that makes you realise how you are
    less smart than you thought" - Andrew Morton, about git.

    I had a bit of hard time getting through the different sizes reported
    and the vocabulary. From what I understood, basicsize is the size of
    the object alone, without the size of his references. And asizeof( x,
    code=False, limit=100 ) will give me the memory footprint of a new
    instance of an object.

    In my case, I just want an object with named attributes.

    My two attempts for comparison:
    import sys, asizeof

    def print_sizeof( v_name, v ):
    print 'Size of %s: %d, %d' % (v_name,
    asizeof.flatsize(v),
    asizeof.asizeof(v, code=False, limit=100) )

    def test_impl( v ):
    assert v.a == 1
    assert v.b == 'bbb'
    assert v.c == False

    ### The reference, a dictionnary
    ref_dict = {'a':1, 'b':'bbb', 'c':False }
    print_sizeof( 'ref_dict', ref_dict )

    ### A modified dictionnary
    class AttrDict (dict):
    """Dictionary allowing attribute access to its elements if they
    are valid attribute names and not already existing methods."""

    def __getattr__ (self, name):
    return self[name]

    attrDict = AttrDict( ref_dict )
    test_impl( attrDict )
    print_sizeof( 'attrDict', attrDict )

    ### Using __slots__
    class ObjWithSlots(object):
    __slots__ = [ 'a', 'b', 'c' ]

    objWithSlots = ObjWithSlots()
    for k in ref_dict:
    setattr( ObjWithSlots, k, ref_dict[k] )
    test_impl( objWithSlots )
    print_sizeof( 'ObjectWithSlots', objWithSlots )

    is giving me:
    Size of ref_dict: 140, 304
    Size of attrDict: 140,
    304
    Size of ObjectWithSlots: 36,
    144

    So, it looks like I could shrink down the structure to 144 bytes in a
    simple case.

    Marco, my objects have varying length data structures (strings) so I
    don't think that would be possible. And it would clearly be overkill
    in my case.
    BlueBird, Apr 22, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Peter Markowsky

    Memory Address' of Python Objects

    Peter Markowsky, Jun 29, 2003, in forum: Python
    Replies:
    1
    Views:
    3,974
    =?ISO-8859-1?Q?Gerhard_H=E4ring?=
    Jun 29, 2003
  2. Neagu, Adrian
    Replies:
    3
    Views:
    388
    Heikki Toivonen
    Sep 21, 2006
  3. Fredrik Lundh
    Replies:
    1
    Views:
    349
    AdrianNg3
    Sep 20, 2006
  4. 7stud
    Replies:
    11
    Views:
    689
    Dennis Lee Bieber
    Mar 20, 2007
  5. dmtr
    Replies:
    18
    Views:
    766
    Nobody
    Aug 7, 2010
Loading...

Share This Page