i have a big dictionary...:)

Discussion in 'Python' started by gabor farkas, Jun 26, 2004.

  1. gabor farkas

    gabor farkas Guest

    hi,

    i have a big dictionary...

    well...
    i have a 6mb textfile,
    from which i build a dictionary the following way:
    the first word of a line is the key, the rest is the value.
    and i fill all the lines into the dictionary (it contains around
    100*1000 lines)

    now when it's in memory, it uses around 60megabytes of memory (measured
    from "ps aux | grep python" :)

    how could i lower this memory requirement?

    is there a simple database-format, something small.... so i don't have
    to keep the whole dictionary in memory?

    thanks,
    gabor
    gabor farkas, Jun 26, 2004
    #1
    1. Advertising

  2. gabor farkas wrote:

    > is there a simple database-format, something small.... so i don't have
    > to keep the whole dictionary in memory?


    take a look at sqlite.

    --
    Maciej "Fiedzia" Dziardziel (fiedzia (at) fiedzia (dot) prv (dot) pl)
    www.fiedzia.prv.pl

    A corpse is a corpse, of course, of course, and no-one can talk to a
    corpse, of course. That is, of course, unless the corpse is the famous
    Mr. Dead!
    Maciej Dziardziel, Jun 26, 2004
    #2
    1. Advertising

  3. gabor farkas

    Kamilche Guest

    gabor farkas <> wrote in message news:<>...

    Have you considered keeping only the key in memory, plus an offset of
    its location in the text file?

    So it would look like:
    key offset
    key1 14
    key2 275
    key3 64004

    Then as you need to access individual lines, hop to the right spot in
    the file based on the offset, and read the line.
    Kamilche, Jun 27, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    11
    Views:
    706
    Roedy Green
    Sep 18, 2005
  2. Shaguf
    Replies:
    0
    Views:
    343
    Shaguf
    Dec 24, 2008
  3. Shaguf
    Replies:
    0
    Views:
    442
    Shaguf
    Dec 26, 2008
  4. Shaguf
    Replies:
    0
    Views:
    229
    Shaguf
    Dec 26, 2008
  5. Shaguf
    Replies:
    0
    Views:
    210
    Shaguf
    Dec 24, 2008
Loading...

Share This Page