handeling very large dictionaries

[mclovin]

Hello all,

I need to have a dictionary of about 8 gigs (well the data it is
processing is around 4gb). so naturally i am running into memory
errors.

So i looked around and found bsddb which acts like a dictionary object
only offloads the data from the RAM to the HDD, however that only
supports strings.

my dictionaries hold my own class objects

Is there something like it that is more flexible?

 

[Aahz]

I need to have a dictionary of about 8 gigs (well the data it is
processing is around 4gb). so naturally i am running into memory
errors.

So i looked around and found bsddb which acts like a dictionary object
only offloads the data from the RAM to the HDD, however that only
supports strings.

Look at the pickle module. Personally, I'd use SQLite instead of bsddb.
You might also look into a full-blown ORM such as SQLAlchemy.

 

[alex23]

Is there something like it that is more flexible?

Have you seen the stdlib module 'shelve'? http://docs.python.org/library/shelve.html

It creates a persistent file-based dictionary, which can hold any type
of object as long as it can be pickled.

I really like the 3rd party module 'shove': http://pypi.python.org/pypi/shove

It's similar to shelve but provides many more backend options,
including dbs, svn and Amazon S3. Very handy.

 

[Joseph Turian]

You could also try using a key-value store.
I am using pytc, a Python API for Tokyo Cabinet. It seems to figure
out quite nicely when to go to disk, and when to use memory. But I
have not done extensive tests.

Here is some example code for using pytc:
http://github.com/turian/pytc-example/tree/master

Joseph