__slots__ vs __dict__

Discussion in 'Python' started by Jean Brouwers, May 12, 2004.

  1. Classes using __slots__ seem to be quite a bit smaller and faster
    to instantiate than regular Python classes using __dict__.

    Below are the results for the __slots__ and __dict__ version of a
    specific class with 16 attributes. Each line in the tables shows the
    number of instances created so far, the total memory usage in Bytes,
    the CPU time in secs, the average size per instance in Bytes and the
    average CPU time per instance in micseconds.

    Instances of this particular class with __slots__ are almost 6x
    smaller and nearly 3x faster to create than intances of the __dict__
    version. Results for other classes will vary, obviously.

    Comments?

    /Jean Brouwers
    ProphICy Semiconductor, Inc.


    PS) The tests were run on a dual 2.4 GHz Xeon system with RedHat
    8.0 and Python 2.3.2. The test script is attached but keep in mind
    that it only has been tested on Linux. It will not work elsewhere
    due to the implementation of the memory() function.


    testing __slots__ version ...
    4096 insts so far: 3.0e+05 B 0.030 sec 73.0 B/i 7.3 usec/i
    8192 insts so far: 8.8e+05 B 0.070 sec 107.5 B/i 8.5 usec/i
    16384 insts so far: 1.5e+06 B 0.150 sec 92.2 B/i 9.2 usec/i
    32768 insts so far: 3.3e+06 B 0.280 sec 101.0 B/i 8.5 usec/i
    65536 insts so far: 6.6e+06 B 0.560 sec 101.2 B/i 8.5 usec/i
    131072 insts so far: 1.4e+07 B 1.200 sec 103.4 B/i 9.2 usec/i
    262144 insts so far: 2.7e+07 B 2.480 sec 103.4 B/i 9.5 usec/i
    524288 insts so far: 5.5e+07 B 5.630 sec 104.0 B/i 10.7 usec/i
    1048576 insts so far: 1.1e+08 B 13.980 sec 104.0 B/i 13.3 usec/i
    1050000 insts total: 1.1e+08 B 14.000 sec 103.9 B/i 13.3 usec/i


    testing __dict__ version ...
    4096 insts so far: 2.4e+06 B 0.050 sec 595.0 B/i 12.2 usec/i
    8192 insts so far: 4.6e+06 B 0.090 sec 564.5 B/i 11.0 usec/i
    16384 insts so far: 9.5e+06 B 0.180 sec 581.8 B/i 11.0 usec/i
    32768 insts so far: 1.9e+07 B 0.370 sec 582.2 B/i 11.3 usec/i
    65536 insts so far: 3.8e+07 B 0.830 sec 582.6 B/i 12.7 usec/i
    131072 insts so far: 7.6e+07 B 1.760 sec 582.7 B/i 13.4 usec/i
    262144 insts so far: 1.5e+08 B 4.510 sec 582.8 B/i 17.2 usec/i
    524288 insts so far: 3.1e+08 B 12.820 sec 582.8 B/i 24.5 usec/i
    1048576 insts so far: 6.1e+08 B 38.370 sec 583.1 B/i 36.6 usec/i
    1050000 insts total: 6.1e+08 B 38.380 sec 583.1 B/i 36.6 usec/i


    -------------------------------slots.py-------------------------------
    <pre>

    from time import clock as time_clock
    def cputime(since=0.0):
    '''Return CPU in secs.
    '''
    return time_clock() - since


    import os
    _proc_status = '/proc/%d/status' % os.getpid() # Linux only
    _scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
    'KB': 1024.0, 'MB': 1024.0*1024.0}

    def _VmB(VmKey):
    global _scale
    try: # get the /proc/<pid>/status pseudo file
    t = open(_proc_status)
    v = [v for v in t.readlines() if v.startswith(VmKey)]
    t.close()
    # convert Vm value to bytes
    if len(v) == 1:
    t = v[0].split() # e.g. 'VmRSS: 9999 kB'
    if len(t) == 3: ## and t[0] == VmKey:
    return float(t[1]) * _scale.get(t[2], 0.0)
    except:
    pass
    return 0.0

    def memory(since=0.0):
    '''Return process memory usage in bytes.
    '''
    return _VmB('VmSize:') - since

    def stacksize(since=0.0):
    '''Return process stack size in bytes.
    '''
    return _VmB('VmStk:') - since



    def slots(**kwds):
    '''Return the slots names as sequence.
    '''
    return tuple(kwds.keys())

    # __slots__ version
    class SlotsClass(object):
    __slots__ = slots(_attr1= False,
    _attr2= None,
    _attr3= None,
    _attr4= None,
    _attr5= None,
    _attr6= None,
    _attr7= 0,
    _attr8= None,
    _attr9= None,
    _attr10=None,
    _attr11=None,
    _attr12=None,
    _attr13=None,
    _attr14=None,
    _attr15=None,
    _attr16=None)

    def __init__(self, tuple4, parent):
    self._attr1 = False
    self._attr2 = None
    self._attr3 = None
    self._attr4 = None
    self._attr5 = None
    self._attr6 = None
    if parent:
    self._attr7 = parent._attr7 + 1
    self._attr8 = parent._attr8
    self._attr9 = parent._attr9
    self._attr10 = parent
    self._attr11 = parent._attr11
    self._attr12 = parent._attr12
    else:
    self._attr7 = 0
    self._attr8 = None
    self._attr9 = None
    self._attr10 = None
    self._attr11 = self
    self._attr12 = None
    self._attr13, self._attr14, self._attr15, self._attr16 = tuple4


    # __dict__ version
    class DictClass(object):
    _attr1 = None
    _attr2 = None
    _attr3 = None
    _attr4 = None
    _attr5 = None
    _attr6 = None
    _attr7 = 0
    _attr8 = None
    _attr9 = None
    _attr10 = None
    _attr11 = None
    _attr12 = None
    _attr13 = None
    _attr14 = None
    _attr15 = None
    _attr16 = None

    def __init__(self, tuple4, parent):
    if parent:
    self._attr7 = parent._attr7 + 1
    self._attr8 = parent._attr8
    self._attr9 = parent._attr9
    self._attr10 = parent
    self._attr11 = parent._attr11
    self._attr12 = parent._attr12
    else:
    self._attr11 = self
    self._attr13, self._attr14, self._attr15, self._attr16 = tuple4


    if __name__ == '__main__':

    import sys

    def report(txt, n, b0, c0):
    c = cputime(c0);
    b = memory(b0)
    print "%8d insts %s: %8.1e B %7.3f sec %6.1f B/i %6.1f usec/i" \
    % (n, txt, b, c, b/n, 1.0e6*c/n)

    if not sys.platform.startswith('linux'):
    raise NotImplementedError, "%r not supported" % sys.platform

    if 'dict' in sys.argv[1:]:
    print 'testing __dict__ version ...'
    testClass = DictClass
    else:
    print 'testing __slots__ version ...'
    testClass = SlotsClass

    t4 = ('', 0, 0, [])
    b0 = memory()
    c0 = cputime()
    p = testClass(t4, None)
    n, m = 1, 4096
    # generate 1+ M instances
    while n < 1050000: # 1048576:
    p = testClass(t4, p)
    n += 1
    if n >= m: # occasionally print stats
    m += m
    report('so far', n, b0, c0)
    report(' total', n, b0, c0)

    </pre>
    Jean Brouwers, May 12, 2004
    #1
    1. Advertising

  2. Jean Brouwers <> writes:

    > Classes using __slots__ seem to be quite a bit smaller and faster
    > to instantiate than regular Python classes using __dict__.

    [snip]

    Yes, but instances usually will not have a Class.__dict__ attribute anymore.
    Thus the following code throws an exception on binding new attributes on a
    per-instance basis:

    # start
    class Foo (object):
    __slots__ = "_test"

    def __init__ (self):
    self._test = None

    if __name__ == "__main__":
    f = Foo ()
    f.testvar = "test"
    return
    # end

    Rebinding __slots__ in __setattr__ fails with a bus error on my system (it
    should not be possible anyways, because __slots__ is a tuple):

    # start
    class Foo (object):
    __slots__ = "_test", "_test2"

    def __init__ (self):
    self._test = 1
    self._test2 = 2

    def __setattr__ (self, name, value):
    # just test a simple rebinding
    self.__slots__ = self.__slots__
    return

    if __name__ == "__main__":
    f = Foo ()
    f.testvar = "test"
    return
    # end

    --- gdb backtrace

    (gdb) r foo.py
    ....
    Program received signal SIGBUS, Bus error.
    0x08070050 in PyDict_GetItem ()
    (gdb) bt
    #0 0x08070050 in PyDict_GetItem ()
    #1 0x08080ab7 in _PyType_Lookup ()
    .....


    Regards
    Marcus

    --
    We don't understand the software, and sometimes we don't understand the
    hardware, but we can *see* the blinking lights!
    Marcus von Appen, May 12, 2004
    #2
    1. Advertising

  3. On Thu, May 13, 2004 at 12:07:54AM +0200, Marcus von Appen wrote:
    > Jean Brouwers <> writes:
    >
    > > Classes using __slots__ seem to be quite a bit smaller and faster
    > > to instantiate than regular Python classes using __dict__.

    > [snip]
    >
    > Yes, but instances usually will not have a Class.__dict__ attribute anymore.
    > Thus the following code throws an exception on binding new attributes on a
    > per-instance basis:
    >
    > # start
    > class Foo (object):
    > __slots__ = "_test"
    >
    > def __init__ (self):
    > self._test = None
    >
    > if __name__ == "__main__":
    > f = Foo ()
    > f.testvar = "test"
    > return
    > # end


    This code is a syntax error for me -- the final return isn't in a function.

    > Rebinding __slots__ in __setattr__ fails with a bus error on my system (it
    > should not be possible anyways, because __slots__ is a tuple):
    >
    > # start
    > class Foo (object):
    > __slots__ = "_test", "_test2"
    >
    > def __init__ (self):
    > self._test = 1
    > self._test2 = 2
    >
    > def __setattr__ (self, name, value):
    > # just test a simple rebinding
    > self.__slots__ = self.__slots__
    > return
    >
    > if __name__ == "__main__":
    > f = Foo ()
    > f.testvar = "test"
    > return
    > # end


    Also a syntax error. I also don't get a bus error, just infinite recursion
    (because assigning to self.__slots__ calls __setattr__, but __slots__ isn't
    an attribute of the instance). What version of Python, and what platform?
    I'm guessing you're on something like FreeBSD where Python has had trouble
    coping gracefully with infinite recursion.

    -Andrew.
    Andrew Bennetts, May 12, 2004
    #3
  4. Andrew Bennetts <> writes:

    [...]
    >> # start
    >> class Foo (object):
    >> __slots__ = "_test"
    >>
    >> def __init__ (self):
    >> self._test = None
    >>
    >> if __name__ == "__main__":
    >> f = Foo ()
    >> f.testvar = "test"
    >> return
    >> # end

    >
    > This code is a syntax error for me -- the final return isn't in a function.


    Sorry for that messy code - the return is absolutely wrong (no idea, why I put
    it there after typing the example in).

    >> Rebinding __slots__ in __setattr__ fails with a bus error on my system (it
    >> should not be possible anyways, because __slots__ is a tuple):
    >>
    >> # start
    >> class Foo (object):
    >> __slots__ = "_test", "_test2"
    >>
    >> def __init__ (self):
    >> self._test = 1
    >> self._test2 = 2
    >>
    >> def __setattr__ (self, name, value):
    >> # just test a simple rebinding
    >> self.__slots__ = self.__slots__
    >> return
    >>
    >> if __name__ == "__main__":
    >> f = Foo ()
    >> f.testvar = "test"
    >> return
    >> # end

    >
    > Also a syntax error. I also don't get a bus error, just infinite recursion
    > (because assigning to self.__slots__ calls __setattr__, but __slots__ isn't
    > an attribute of the instance). What version of Python, and what platform?


    Typed in a wrong line again here. Change it to something appropriate like

    def __setattr__ (self, name, value):
    # just test a simple rebinding
    self.__slots__.__add__ (tuple (name))
    return

    > I'm guessing you're on something like FreeBSD where Python has had trouble
    > coping gracefully with infinite recursion.


    Right, but this was caused by typing in wrong code. Sorry for that.

    Regards
    Marcus

    --
    We don't understand the software, and sometimes we don't understand the
    hardware, but we can *see* the blinking lights!
    Marcus von Appen, May 13, 2004
    #4
  5. On Thu, May 13, 2004 at 06:55:39AM +0200, Marcus von Appen wrote:
    > Andrew Bennetts <> writes:
    >
    > [...]
    > >>
    > >> # start
    > >> class Foo (object):
    > >> __slots__ = "_test", "_test2"
    > >>
    > >> def __init__ (self):
    > >> self._test = 1
    > >> self._test2 = 2
    > >>
    > >> def __setattr__ (self, name, value):
    > >> # just test a simple rebinding
    > >> self.__slots__ = self.__slots__
    > >> return
    > >>
    > >> if __name__ == "__main__":
    > >> f = Foo ()
    > >> f.testvar = "test"
    > >> return
    > >> # end

    > >
    > > Also a syntax error. I also don't get a bus error, just infinite recursion
    > > (because assigning to self.__slots__ calls __setattr__, but __slots__ isn't
    > > an attribute of the instance). What version of Python, and what platform?

    >
    > Typed in a wrong line again here. Change it to something appropriate like
    >
    > def __setattr__ (self, name, value):
    > # just test a simple rebinding
    > self.__slots__.__add__ (tuple (name))
    > return


    Now your example runs just fine (i.e. the script terminates normally, and
    nothing happens). I still don't see any error. (And why do you keep
    putting a redundant return at the end of your functions?)

    Are you sure this is the same code that you get a crash with? What version
    of Python are you using? (I've tested with 2.2.3 and 2.3.3).

    -Andrew.
    Andrew Bennetts, May 13, 2004
    #5
  6. Andrew Bennetts <> writes:

    [...]

    > Now your example runs just fine (i.e. the script terminates normally, and
    > nothing happens). I still don't see any error.


    I thought you would inspect the objects yourself. Let me show you another (now
    working) example, which will explain it:

    ------
    class DictClass (object):
    test = "DictTest"

    def __init__ (self):
    pass

    class SlotClass (object):
    __slots__ = "test"

    def __init__ (self):
    self.test = "SlotTest"

    def print_obj (obj):
    # inspect .test and show the object attributes
    print obj.test
    print dir (obj)

    # bind new attribute and inspect the object attributes
    obj.new_test = "Test"
    print obj.new_test
    print dir (obj)
    print "------"
    return

    if __name__ == "__main__":
    dict_obj = DictClass ()
    slot_obj = SlotClass ()

    print_obj (dict_obj)
    print_obj (slot_obj)

    ---

    You will get something like the following output:

    ----------------
    DictTest
    ['__class__', '__delattr__', '__dict__', ....., '__weakref__', 'test']

    Test
    ['__class__', '__delattr__', '__dict__', ...., '__weakref__', 'new_test',
    'test']

    ------

    SlotTest
    ['__class__', '__delattr__', ...., '__slots__', '__str__', 'test']

    Traceback (most recent call last):
    File "foo.py", line 31, in ?
    print_obj (slot_obj)
    File "foo.py", line 19, in print_obj
    obj.new_test = "Test"
    AttributeError: 'SlotClass' object has no attribute 'new_test'
    ----------------

    As you will notice, a __slot__ed object/class has no __weakref__ nor __dict__
    attribute.
    Thus binding new object attributes will fail with an AttributeError.

    So let's try it again with using __setattr__ in SlotClass:

    class SlotClass (object):
    test = "SlotTest"
    test2 = "SlotTest2" # just for making a tuple creation easier
    __slots__ = test, test2

    def __init__ (self):
    pass

    def __setattr__ (self, name, value):
    # create a dict here to add its key to the __slot__ tuple
    self.__slots__.__add__ (tuple ((name)))
    return


    def print_obj (obj):
    # inspect .test and show the object attributes
    print obj.test
    print dir (obj)

    # bind new attribute and inspect the object attributes
    obj.new_test = "Test"
    print obj.new_test
    print dir (obj)
    print "------"
    return

    if __name__ == "__main__":

    #dict_obj = DictClass ()
    slot_obj = SlotClass ()

    #print_obj (dict_obj)
    print_obj (slot_obj)

    ----


    As you will see, you get an AttributeError again.tuple ((name))was not
    concatenated to __slots__.
    I think that should explain enough about the advantage and disadvantage of
    __slots__ here.

    (And why do you keep
    > putting a redundant return at the end of your functions?)


    I'm used to it :).

    > Are you sure this is the same code that you get a crash with? What version
    > of Python are you using? (I've tested with 2.2.3 and 2.3.3).


    No, I put messy code without _really_ thinking about it in my first post.
    I did not realize that self.__slots__ = self._slots in __setattr__ will end
    up in a recursion...

    Regards
    Marcus

    --
    We don't understand the software, and sometimes we don't understand the
    hardware, but we can *see* the blinking lights!
    Marcus von Appen, May 13, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jp Calderone
    Replies:
    1
    Views:
    512
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Jul 6, 2003
  2. simon place

    PEP idea. ( removing __slots__ )

    simon place, Jul 5, 2003, in forum: Python
    Replies:
    6
    Views:
    321
    Ulrich Petri
    Jul 7, 2003
  3. Replies:
    1
    Views:
    326
    Alex Martelli
    Nov 6, 2003
  4. Derek Fountain

    When is a __dict__ not a __dict__?

    Derek Fountain, Apr 21, 2004, in forum: Python
    Replies:
    1
    Views:
    322
    John Roth
    Apr 21, 2004
  5. Hornberger, Chris

    RE: __slots__ vs __dict__

    Hornberger, Chris, May 12, 2004, in forum: Python
    Replies:
    1
    Views:
    415
    Jean Brouwers
    May 12, 2004
Loading...

Share This Page