shelve and nested dictionaries

[Matthew Schibler]

I'm a newbie to Python, with some experience using perl (where I used
nested arrays and hashes extensively). I am building a script in
python for a MUD I play, and I want to use the shelve module to store
persistent information between script executions. The following code
does not work for me,

import shelve, sys, os, string
db = shelve.open(os.path.abspath(os.path.dirname(sys.argv[0])) + '/' +
'sandbox.dat', 'c')
db['JustSomeVariable'] = 'apple'
db['subdb'] = {}
db['subdb']['anotherdict'] = {}
db['subdb']['anotherdict']['bleh'] = 'hello world'
db.close()

of course, that's just a working example but it illustrates the
problem i'm having. I think shelve objects act like dictionaries in a
way, at least they seem to have dictionary keys beneath them. And I
don't seem to have this problem when I use a normal dictionary as
opposed to shelve for nesting other dictionaries.

So i'm now confused, i've hit a brick wall and i'm not sure how to
solve this problem.

Can anyone explain what i'm doing wrong?

Thanks

 

[Fredrik Lundh]

I'm a newbie to Python, with some experience using perl (where I used
nested arrays and hashes extensively). I am building a script in
python for a MUD I play, and I want to use the shelve module to store
persistent information between script executions. The following code
does not work for me,

import shelve, sys, os, string
db = shelve.open(os.path.abspath(os.path.dirname(sys.argv[0])) + '/' +
'sandbox.dat', 'c')
db['JustSomeVariable'] = 'apple'
db['subdb'] = {}
db['subdb']['anotherdict'] = {}
db['subdb']['anotherdict']['bleh'] = 'hello world'
db.close()

of course, that's just a working example but it illustrates the
problem i'm having. I think shelve objects act like dictionaries in a
way, at least they seem to have dictionary keys beneath them.

the shelve module only tracks changes to the shelf itself (i.e.
db[key]), not changes to to mutable objects stored in the shelve).

to change a mutable object, you have to fetch it, modify it, and then
write it back:

value = db[key]
... update value ...
db[key] = value

in Python 2.3 and later, the shelve can help you with this, to some
extent; from the help page:

To avoid the problem with mutable entries, you may pass
the keyword argument writeback=True in the call to
shelve.open. When you use:

d = shelve.open(filename, writeback=True)

then d keeps a cache of all entries you access, and writes
them all back to the persistent mapping when you call
d.close(). This ensures that such usage as
d[key].append(anitem) works as intended.

However, using keyword argument writeback=True may consume
vast amount of memory for the cache, and it may make
d.close() very slow, if you access many of d's entries
after opening it in this way: d has no way to check which
of the entries you access are mutable and/or which ones
you actually mutate, so it must cache, and write back at
close, all of the entries that you access. You can call
d.sync() to write back all the entries in the cache, and
empty the cache (d.sync() also synchronizes the persistent
dictionary on disk, if feasible).

</F>