Backing up objects in db file

[Chris Stiles]

Hi --

I have a program that stores all its data as pickled objects inside a
(berkeley) db file.

Having been bitten a couple of times when the format changed across linux
distribution upgrades, I'd like to write a simple program that dumps the
objects to some kind of text format, which can then be reconstituted into
objects later. It can't just dump out the pickle format, because that could
change between python versions.

There seem to be a few solutions in this space that nearly work - two
incomplete YAML implementations, and a similiar maturity XML thing. Does
anyone know of a solution that will both work, and is actively maintained ?

 

[Skip Montanaro]

Chris> I have a program that stores all its data as pickled objects
Chris> inside a (berkeley) db file.

Chris> Having been bitten a couple of times when the format changed
Chris> across linux distribution upgrades, I'd like to write a simple
Chris> program that dumps the objects to some kind of text format, which
Chris> can then be reconstituted into objects later. It can't just dump
Chris> out the pickle format, because that could change between python
Chris> versions.

Take a look at db2pickle.py and pickle2db.py in the Tools/scripts directory
of the source distribution. I have no idea if they are included in the
Windows installer. Here's the doc string from db2pickle.py:

Synopsis: db2pickle.py [-h|-g|-b|-r|-a] dbfile [ picklefile ]

Convert the database file given on the command line to a pickle
representation. The optional flags indicate the type of the database
(hash, btree, recno). The default is hash. If a pickle file is named it
is opened for write access (deleting any existing data). If no pickle
file is named, the pickle output is written to standard output.

Looking at that I can see I need to update the doc string. The flags mean:

-h - open as bsddb hash file
-g - open as gdbm file
-b - open as bsddb btree file
-r - open as bsddb recno file
-a - open using anydbm

Skip

 

[Miki Tebeka]

Hello Chris,

There seem to be a few solutions in this space that nearly work - two
incomplete YAML implementations, and a similiar maturity XML thing. Does
anyone know of a solution that will both work, and is actively maintained ?

Maybe use Zope DB?

HTH.
Miki

 

[Chris Stiles]

Take a look at db2pickle.py and pickle2db.py in the Tools/scripts directory
of the source distribution. I have no idea if they are included in the
Windows installer. Here's the doc string from db2pickle.py:

Skip - this would be fine if only the db file changed between releases of
libdb. Unfortunately, it appears the pickle format also sometimes changes, at
least this was my experience. So it's not a portable text format.

 

[Chris Stiles]

Maybe use Zope DB?

Does that store objects using a portable text format ? I thought it used
Gadfly.

 

[Miki Tebeka]

Hello Chris,

Does that store objects using a portable text format ? I thought it used

Don't know. I just know Zope has an object DB.

Miki

 

[Chris Stiles]

Hello Chris,

Don't know. I just know Zope has an object DB.

Sure, but what i'm looking to do is dump out objects in a portable text format
which I can then reimport into python

 

[Skip Montanaro]

Chris> Skip - this would be fine if only the db file changed between
Chris> releases of libdb. Unfortunately, it appears the pickle format
Chris> also sometimes changes, at least this was my experience. So it's
Chris> not a portable text format.

I've yet to see a situation where an older pickle file couldn't be read by a
newer pickle version. One goal of the pickle format was precisely that it
should be version-independent. You might not be able to go the other way
(dump in newer, load in older), but even then you should be able to force
the dump to be made in the plain text (version 0) format.

The one situation I can think of where an older pickle would be unreadable
with a new version of pickle is if you dumped an instance in the older
version whose class could not be found in the newer version.

Skip

 

[Chris Stiles]

I've yet to see a situation where an older pickle file couldn't be read by a
newer pickle version. One goal of the pickle format was precisely that it
should be version-independent. You might not be able to go the other way
(dump in newer, load in older), but even then you should be able to force
the dump to be made in the plain text (version 0) format.

Right, so I used the plain text format. But a couple of times (when going
from 2.X -> 2.X), i ran into pickle VM problems when reading things back. I
didn't know enough about the format to debug the thing and wasn't aware that
there was a definitive statement that the pickle format MUST remain backward
compatible in this way.