problems with shelve(), collections.defaultdict, self

Discussion in 'Python' started by 7stud, Feb 11, 2012.

  1. 7stud

    7stud Guest

    The following code demonstrates that a collections.defaultdict is
    shelve worthy:


    import shelve
    import collections as c


    dd = c.defaultdict(int)
    dd["Joe"] = 3
    print(dd)

    my_shelve = shelve.open('data.shelve')
    my_shelve['strike_record'] = dd
    my_shelve.close()

    my_shelve = shelve.open('data.shelve')
    data = my_shelve['strike_record']
    my_shelve.close()

    dd.clear()
    dd.update(data)
    print(dd)

    --output:--
    defaultdict(<class 'int'>, {'Joe': 3})
    defaultdict(<class 'int'>, {'Joe': 3})


    And the following code demonstrates that a class that inherits from
    dict can shelve itself:

    import collections as c
    import shelve

    class Dog(dict):
    def __init__(self):
    super().__init__(Joe=1)
    print('****', self)

    def save(self):
    my_shelve = shelve.open('data22.shelve')
    my_shelve['x'] = self
    my_shelve.close()

    def load(self):
    my_shelve = shelve.open('data22.shelve')
    data = my_shelve['x']
    my_shelve.close()

    print(data)


    d = Dog()
    d.save()
    d.load()

    --output:--
    **** {'Joe': 1}
    {'Joe': 1}


    But I cannot get a class that inherits from collections.defaultdict to
    shelve itself:


    import collections as c
    import shelve

    class Dog(c.defaultdict):
    def __init__(self):
    super().__init__(int, Joe=0)
    print('****', self)

    def save(self):
    my_shelve = shelve.open('data22.shelve')
    my_shelve['dd'] = self
    my_shelve.close()

    def load(self):
    my_shelve = shelve.open('data22.shelve')
    data = my_shelve['dd']
    my_shelve.close()

    print(data)


    d = Dog()
    d.save()
    d.load()

    --output:--

    **** defaultdict(<class 'int'>, {'Joe': 30})
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
    python3.2/shelve.py", line 111, in __getitem__
    value = self.cache[key]
    KeyError: 'dd'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "3.py", line 95, in <module>
    d.load()
    File "3.py", line 87, in load
    data = my_shelve['dd']
    File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
    python3.2/shelve.py", line 114, in __getitem__
    value = Unpickler(f).load()
    TypeError: __init__() takes exactly 1 positional argument (2 given)



    I deleted all *.shelve.db files between program runs. I can't figure
    out what I'm doing wrong.
     
    7stud, Feb 11, 2012
    #1
    1. Advertising

  2. 7stud

    7stud Guest

    On Feb 10, 7:48 pm, 7stud <> wrote:
    >
    > But I cannot get a class that inherits from collections.defaultdict to
    > shelve itself:
    >
    > import collections as c
    > import shelve
    >
    > class Dog(c.defaultdict):
    >     def __init__(self):
    >         super().__init__(int, Joe=0)
    >         print('****', self)


    Whoops. I changed:

    super().__init__(int, Joe=0)

    to:

    super().__init__(int, Joe=30)

    hence this output..

    > --output:--
    >
    > **** defaultdict(<class 'int'>, {'Joe': 30})
     
    7stud, Feb 11, 2012
    #2
    1. Advertising

  3. 7stud

    7stud Guest

    On Feb 10, 7:52 pm, 7stud <> wrote:

    I don't know if this helps, but I notice when I initially do this:

    shelve.open('data22')

    the file is saved as 'data22.db'. But on subsequent calls to
    shelve.open(), if I use the file name 'data22.db', I get a different
    error:

    --output:--

    **** defaultdict(<class 'int'>, {'Joe': 30})
    Traceback (most recent call last):
    File "3.py", line 95, in <module>
    d.load()
    File "3.py", line 86, in load
    my_shelve = shelve.open('data22.db')
    File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
    python3.2/shelve.py", line 232, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
    File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
    python3.2/shelve.py", line 216, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol,
    writeback)
    File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
    python3.2/dbm/__init__.py", line 83, in open
    raise error[0]("db type could not be determined")
    dbm.error: db type could not be determined



    The code that produced that error:



    import collections as c
    import shelve

    class Dog(c.defaultdict):
    def __init__(self):
    super().__init__(int, Joe=30)
    print('****', self)

    def save(self):
    my_shelve = shelve.open('data22')
    my_shelve['dd'] = self
    my_shelve.close()

    def load(self):
    my_shelve = shelve.open('data22.db')
    data = my_shelve['dd']
    my_shelve.close()

    print(data)


    d = Dog()
    d.save()
    d.load()


    I'm using python 3.2.2.
     
    7stud, Feb 11, 2012
    #3
  4. 7stud

    Ian Kelly Guest

    On Fri, Feb 10, 2012 at 7:48 PM, 7stud <> wrote:
    > But I cannot get a class that inherits from collections.defaultdict to
    > shelve itself:
    >
    >
    > import collections as c
    > import shelve
    >
    > class Dog(c.defaultdict):
    >    def __init__(self):
    >        super().__init__(int, Joe=0)
    >        print('****', self)
    >
    >    def save(self):
    >        my_shelve = shelve.open('data22.shelve')
    >        my_shelve['dd'] = self
    >        my_shelve.close()
    >
    >    def load(self):
    >        my_shelve = shelve.open('data22.shelve')
    >        data = my_shelve['dd']
    >        my_shelve.close()
    >
    >        print(data)
    >
    >
    > d = Dog()
    > d.save()
    > d.load()
    >
    > --output:--
    >
    > **** defaultdict(<class 'int'>, {'Joe': 30})
    > Traceback (most recent call last):
    >  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
    > python3.2/shelve.py", line 111, in __getitem__
    >    value = self.cache[key]
    > KeyError: 'dd'
    >
    > During handling of the above exception, another exception occurred:
    >
    > Traceback (most recent call last):
    >  File "3.py", line 95, in <module>
    >    d.load()
    >  File "3.py", line 87, in load
    >    data = my_shelve['dd']
    >  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/
    > python3.2/shelve.py", line 114, in __getitem__
    >    value = Unpickler(f).load()
    > TypeError: __init__() takes exactly 1 positional argument (2 given)
    >
    >
    >
    > I deleted all *.shelve.db files between program runs.  I can't figure
    > out what I'm doing wrong.


    The problem is that defaultdict defines a custom __reduce__ method
    which is used by the pickle protocol to determine how the object
    should be reconstructed. It uses this to reconstruct the defaultdict
    with the same default factory, by calling the type with a single
    argument of the default factory. Since your subclass's __init__
    method takes no arguments, this results in the error you see.

    There are a couple of ways you could fix this. The first would be to
    change the signature of the __init__ method to take an optional
    argument accepting the default factory instead of hard-coding it, like
    this:

    def __init__(self, default_factory=int):
    super().__init__(default_factory, Joe=0)

    The other would be to just fix the __reduce__ method to not pass the
    default factory to your initializer, since it is hard-coded. That
    would look like this:

    def __reduce__(self):
    return (type(self), ())

    Cheers,
    Ian
     
    Ian Kelly, Feb 11, 2012
    #4
  5. 7stud

    Ian Kelly Guest

    On Sat, Feb 11, 2012 at 10:46 AM, Ian Kelly <> wrote:
    > The problem is that defaultdict defines a custom __reduce__ method
    > which is used by the pickle protocol to determine how the object
    > should be reconstructed.  It uses this to reconstruct the defaultdict
    > with the same default factory, by calling the type with a single
    > argument of the default factory.  Since your subclass's __init__
    > method takes no arguments, this results in the error you see.
    >
    > There are a couple of ways you could fix this.  The first would be to
    > change the signature of the __init__ method to take an optional
    > argument accepting the default factory instead of hard-coding it, like
    > this:
    >
    >    def __init__(self, default_factory=int):
    >        super().__init__(default_factory, Joe=0)
    >
    > The other would be to just fix the __reduce__ method to not pass the
    > default factory to your initializer, since it is hard-coded.  That
    > would look like this:
    >
    >    def __reduce__(self):
    >        return (type(self), ())


    It occurs to me that there's also an option 3: you don't really need a
    defaultdict to do what you're trying to do here. You just need a dict
    with a custom __missing__ method. That could look something like
    this:

    class Dog(dict):

    def __missing__(self):
    return 0

    And then you don't have to worry about the weird pickle behavior of
    defaultdict at all.

    Cheers,
    Ian
     
    Ian Kelly, Feb 11, 2012
    #5
  6. 7stud

    Ian Kelly Guest

    On Sat, Feb 11, 2012 at 10:54 AM, Ian Kelly <> wrote:
    > class Dog(dict):
    >
    >    def __missing__(self):
    >        return 0


    Sorry, that should have been:

    class Dog(dict):

    def __missing__(self, key):
    return 0

    Cheers,
    Ian
     
    Ian Kelly, Feb 11, 2012
    #6
  7. 7stud

    7stud Guest

    On Feb 11, 10:56 am, Ian Kelly <> wrote:
    > On Sat, Feb 11, 2012 at 10:54 AM, Ian Kelly <> wrote:
    > > class Dog(dict):

    >
    > >    def __missing__(self):
    > >        return 0

    >
    > Sorry, that should have been:
    >
    > class Dog(dict):
    >
    >     def __missing__(self, key):
    >         return 0
    >
    > Cheers,
    > Ian


    Thanks Ian!
     
    7stud, Feb 11, 2012
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    2
    Views:
    345
  2. metaperl.com
    Replies:
    2
    Views:
    432
    Duncan Booth
    Nov 6, 2007
  3. Matthew Wilson
    Replies:
    8
    Views:
    361
  4. Pete Emerson
    Replies:
    11
    Views:
    1,030
    Steven D'Aprano
    Mar 8, 2010
  5. Steven W. Orr

    Question about collections.defaultdict

    Steven W. Orr, Mar 26, 2012, in forum: Python
    Replies:
    0
    Views:
    225
    Steven W. Orr
    Mar 26, 2012
Loading...

Share This Page