Saving / Restoring data in a program

[Bob van der Poel]

I've got an application that needs to store (internally) a bunch of data
consisting of lists and some dicts. None of the items are very large,
and the entire set (for a single save) is probably less that 1K of data,
or a total of several hundred individual objects (mostly short, 4 to 16
element, lists of integers).

Right now, I'm doing something like:

name = 'Store1'
saved[name] = {
'NAMES' = names[:],
'CHORDS' = stuff.copy(),
.... }

and, then when I want to restore I:

s=saved[name]
names=s['NAMES']
stuff = s['CHORDS']
...

this is turning into a bit of a mess. It's error prone since I have to
make sure that I restore everything I save and I have to make sure I
usethe same spellings. And, I have to be aware of the data types so I
can decide if [:] or copy() or even copy.deepcopy() is needed.

So, is there better way? I looked at pickle and it appears to me that
this will only work if I store to disk, which I don't really want to do.

Guess what I'd like is something like:

saved[name] = wonderfulSave(names, stuff, foo,
more_of_my_items/lists)

and then have it all restored with:

wonderfulRestore(saved[name])

Anything like that around???

I should mention that the items are part of a class. So, I'm really doing:

saved[name] = self.foo[:] ... etc.

I'm thinking that if I were to put all the stuff into a master-class,
then I could do something like:

saved[name]= copy.deepcopy(self.masterclass)

but, then I'd have to make sure that I use the 'masterclass' in all my
references to the data ... and that might just create more confusions
(well, more typing) and slowdowns.

BTW, whoever said that creating proper data types BEFORE writing a
program is a good thing was correct. Just wish I'd listened better

 

[Lonnie Princehouse]

It's not entirely clear what you're trying to do, but it sounds like
you want to save and restore the state of a subset of local namespace.

First off, rethink this whole scheme. Consider writing a class for
your saved data. Much more Pythonic.

But if you're determined, consider this snippet:

# make some variables
a = 5
b = 10
c = 'foo'

# Names of variables to save
save_vars = ['a','b','c']

# Save them by first dumping their values into a list with
# a list comprehension
saved['store1'] = [(key,locals()[key]) for key in save_vars]

# Restore variables later by injecting them into the
# local namespace dictionary
# (probably considered very bad form!)
for key,value in saved['store1']:
locals()[key] = value


On a side note, it _is_ possible to use pickle without a proper file.
In your case, it doesn't sound like there's any reason to do this, but
it does come in handy upon occasion. Pickle only wants something that
behaves like a file, so you can use a StringIO instead:

import pickle
from StringIO import StringIO

foo = "something that needs to be pickled"

fake_file = StringIO()

# Pickle it
pickle.dump(foo, fake_file)

# Unpickle it
fake_file.seek(0)
bar = pickle.load(fake_file)

I've got an application that needs to store (internally) a bunch of data
consisting of lists and some dicts. None of the items are very large,
and the entire set (for a single save) is probably less that 1K of data,
or a total of several hundred individual objects (mostly short, 4 to 16
element, lists of integers).

Right now, I'm doing something like:

name = 'Store1'
saved[name] = {
'NAMES' = names[:],
'CHORDS' = stuff.copy(),
.... }

and, then when I want to restore I:

s=saved[name]
names=s['NAMES']
stuff = s['CHORDS']
...

this is turning into a bit of a mess. It's error prone since I have to
make sure that I restore everything I save and I have to make sure I
usethe same spellings. And, I have to be aware of the data types so I
can decide if [:] or copy() or even copy.deepcopy() is needed.

So, is there better way? I looked at pickle and it appears to me that
this will only work if I store to disk, which I don't really want to do.

Guess what I'd like is something like:

saved[name] = wonderfulSave(names, stuff, foo,
more_of_my_items/lists)

and then have it all restored with:

wonderfulRestore(saved[name])

Anything like that around???

I should mention that the items are part of a class. So, I'm really doing:

saved[name] = self.foo[:] ... etc.

I'm thinking that if I were to put all the stuff into a master-class,
then I could do something like:

saved[name]= copy.deepcopy(self.masterclass)

but, then I'd have to make sure that I use the 'masterclass' in all my
references to the data ... and that might just create more confusions
(well, more typing) and slowdowns.

BTW, whoever said that cre''.join([chr(ord(c)+23) for c in 'UXWWRN)VJPRLYNJLNOJ[V\x17LXV'])

ating proper data types BEFORE writing a

 

[Bob van der Poel]

It's not entirely clear what you're trying to do, but it sounds like
you want to save and restore the state of a subset of local namespace.

First off, rethink this whole scheme. Consider writing a class for
your saved data. Much more Pythonic.

Sorry about being unclear. I tired to make my problem simplier than it
is, and this probably just led to less clearity. Let me try again...

My progarm, MMA, creates midi accompaniment tracks. (BTW, an alpha
version of this program is available with demos, docs, etc. on my web
site.) It lets the user define a "Groove" which is a collection of style
details. These details include things like the pattern to use for drums,
piano chords, etc. As well, the groove includes the octave for each
voice, timing options, volumes, etc.

A user creates a song by listing the chords and setting the grooves. So,
he might have the first 4 bars of a song with a groove called "SWING",
the next 8 with "RHUMBA", etc.

What my program needs to do is to save the pattern, etc. in a storage
space and, when needed, restore the settings from that storage. One
complication is that the current settings don't need to be saved. So, it
isn't as simple as having a pointer to the "SWING" slot. Instead, I
copy the data from the slot into the current space (overwritting the
current settings). And all this data from the current space can be saved
into a storage space.

Now, I do have all this working. I just don't like the way it has been
implimented. As I tried (poorly) to explain in my previous post, I put
the various details into storage with code like this... consider the
variables needed to maintain a piano chord track (this stuff is
duplicated for drum tracks, etc. And, yes, this is all object based).

Again, trying to make it bit clearer, we have a number of variables.
I've noted them here with static values, but they are created dynamically.

self.octave = (4,0,1,5)
self.timeadjust = (5,5,5,5)
self.pattern = { vols=(90,80,99,100), start=(0, 25, 50, 75),
len=(4,4,4,4) }
..and a lot more...

Now, at some point we want to save all this so it can be recalled later.
And, note, we have to save copies of the data, not references. I'm using
a dict with keys the same as the groove names. So, for the SWING groove,
the data is saved in self.grooves['SWING']. So, we end up with code:

self.grooves["SWING"] = {
'OCTAVE':self.octave[:]
'TIME':self.timeadjust[:]
'PATS':self.pattern.copy()
... etc }

And, later when we need to restore:

g=self.grooves["SWING"]
self.octave=g['OCTAVE']
self.timeadjust=g['TIME']
self.pattern=g['PATS']

My concerns with all this are mainly maintainace. Everytime I add or
modify something I have make sure I add them to the save/restore code.
And this includes having the right "copy" code (is this a list which can
be copied with [:], or a dict which needs ".copy", etc). Plus I have to
make sure I duplicate things properly in the save and restore sections.
And the spellings of the storage slots ('OCTAVE', 'PATS') has to be the
same.

But if you're determined, consider this snippet:

Really not a matter of being determined to do it my way But, I am
determined to keep it working...

# make some variables
a = 5
b = 10
c = 'foo'

# Names of variables to save
save_vars = ['a','b','c']

# Save them by first dumping their values into a list with
# a list comprehension
saved['store1'] = [(key,locals()[key]) for key in save_vars]

If they were all simple items like an integer, then this would be fine.
But, I have a mix of simple variables, lists and dicts.

On a side note, it _is_ possible to use pickle without a proper file.
In your case, it doesn't sound like there's any reason to do this, but
it does come in handy upon occasion. Pickle only wants something that
behaves like a file, so you can use a StringIO instead:

Yes, I was thinking of using pickle, but you're right that there doesn't
seem to be much benefit. But, I will file away the thought of using
StringIO for a future project. Thanks.

Thinking a bit more about all this it might make the most sense to pack
all the variables which need saving into a class. What do you think
about something like this:

class mutablevariables:
def __init__(self):
self.octave = [0,0,0,0]
self.pats = ...

Now, when I need to reference the octave, I would end up with:

o = self.mutablevariables.octave[offset]

which isn't that much different from what I'm doing right now which is:

o = self.octave[offset]

And I can always set a temp variable to make the refs a bit shorts.

And then I can save the whole works with:

self.grooves["SWING"] = copy.deepcopy(self.mutablevariables)

Is this what you mean by "Consider writing a class for your saved data.
Much more Pythonic."

Thanks!

 

[Miki Tebeka]

Hello Bob,

I've got an application that needs to store (internally) a bunch of data
consisting of lists and some dicts. None of the items are very large,
and the entire set (for a single save) is probably less that 1K of data,
or a total of several hundred individual objects (mostly short, 4 to 16
element, lists of integers).

So, is there better way? I looked at pickle and it appears to me that
this will only work if I store to disk, which I don't really want to do.

Have you looked at pickle's "dumps" and "loads" which store data in
strings?

Guess what I'd like is something like:

saved[name] = wonderfulSave(names, stuff, foo,
more_of_my_items/lists)

and then have it all restored with:

wonderfulRestore(saved[name])

Anything like that around???

As suggested in earlier post, use a class to store all the attributes
and then save it.
class Groove:
def __init__(self, names, stuff, foo, *extra):
self.names = names
self.stuff = stuff
self.foo = foo
self.extra = extra

And then save a instance of it using pickle.

I should mention that the items are part of a class. So, I'm really doing:

saved[name] = self.foo[:] ... etc.

Another way is to create a dictionary with subset of __dict__
save = {}
for attr in ["names", "stuff", "foo", "extra"]:
save[attr] = self.__dict__[attr]
saved_attrs = dumps(save)

And save this dictionary with pickle.
Later on...
args = loads(saved_attrs)
self.__dict__.update(args)

BTW, whoever said that creating proper data types BEFORE writing a
program is a good thing was correct. Just wish I'd listened better

The important thing is to learn from your mistakes:
"I've missed more than 9,000 shots in my career. I've lost more than
300 games. Twenty-six times I've been trusted to take the game-winning
shot -- and missed. I've failed over and over and over again in my
life. And that is why I succeed."
Michael Jordan

HTH.
Miki

 

[darrell]

I like files that people can read and edit.
So I write valid python to a file to save config data.
Then import it later.

fp=open("config",'w')
print >>fp, "config={"
for k,v in myDict.items():
print >>fp, "%s:%s,"%(`k`,`v`)
print >>fp,"}"

pprint is good for this also.
repr works but you get very long lines.

With extra effort, XML would be good for this also.

--Darrell