i have a big dictionary...:)

[gabor farkas]

hi,

i have a big dictionary...

well...
i have a 6mb textfile,
from which i build a dictionary the following way:
the first word of a line is the key, the rest is the value.
and i fill all the lines into the dictionary (it contains around
100*1000 lines)

now when it's in memory, it uses around 60megabytes of memory (measured
from "ps aux | grep python"

how could i lower this memory requirement?

is there a simple database-format, something small.... so i don't have
to keep the whole dictionary in memory?

thanks,
gabor

 

[Maciej Dziardziel]

is there a simple database-format, something small.... so i don't have
to keep the whole dictionary in memory?

take a look at sqlite.

--
Maciej "Fiedzia" Dziardziel (fiedzia (at) fiedzia (dot) prv (dot) pl)
www.fiedzia.prv.pl

A corpse is a corpse, of course, of course, and no-one can talk to a
corpse, of course. That is, of course, unless the corpse is the famous
Mr. Dead!

 

[Kamilche]

Have you considered keeping only the key in memory, plus an offset of
its location in the text file?

So it would look like:
key offset
key1 14
key2 275
key3 64004

Then as you need to access individual lines, hop to the right spot in
the file based on the offset, and read the line.