DBM scalability

Discussion in 'Python' started by George Sakkis, Oct 21, 2005.

  1. I'm trying to create a dbm database with around 4.5 million entries but the existing dbm modules
    (dbhash, gdbm) don't seem to cut it. What happens is that the more entries are added, the more time
    per new entry is required, so the complexity seems to be much worse than linear. Is this to be
    expected and if so, should I expect better performance (i.e. linear or almost linear) from a real
    database, e.g. sqlite ?

    George
    George Sakkis, Oct 21, 2005
    #1
    1. Advertising

  2. George Sakkis

    Paul Rubin Guest

    "George Sakkis" <> writes:
    > I'm trying to create a dbm database with around 4.5 million entries
    > but the existing dbm modules (dbhash, gdbm) don't seem to cut
    > it. What happens is that the more entries are added, the more time
    > per new entry is required, so the complexity seems to be much worse
    > than linear. Is this to be expected


    No, not expected. See if you're using something like db.keys() which
    tries to read all the keys from the db into memory, or anything like that.
    Paul Rubin, Oct 21, 2005
    #2
    1. Advertising

  3. "Paul Rubin" <http://> wrote:

    > "George Sakkis" <> writes:
    > > I'm trying to create a dbm database with around 4.5 million entries
    > > but the existing dbm modules (dbhash, gdbm) don't seem to cut
    > > it. What happens is that the more entries are added, the more time
    > > per new entry is required, so the complexity seems to be much worse
    > > than linear. Is this to be expected

    >
    > No, not expected. See if you're using something like db.keys() which
    > tries to read all the keys from the db into memory, or anything like that.


    It turns out it doesn't have to do with python or the dbm modules. The same program on a different
    box and platform runs linearly, so I guess it has to do with the OS and/or the hard disk
    configuration.

    George
    George Sakkis, Oct 22, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matt Johnson
    Replies:
    0
    Views:
    1,267
    Matt Johnson
    Aug 8, 2003
  2. Topher
    Replies:
    1
    Views:
    1,122
    Gunnar Hjalmarsson
    Jan 21, 2004
  3. Andrew Chalk

    dbm under Windows

    Andrew Chalk, Sep 5, 2003, in forum: Python
    Replies:
    0
    Views:
    364
    Andrew Chalk
    Sep 5, 2003
  4. Guest

    Best dbm to use?

    Guest, Sep 7, 2005, in forum: Python
    Replies:
    1
    Views:
    269
    Ivan Voras
    Sep 7, 2005
  5. Guest
    Replies:
    0
    Views:
    508
    Guest
    Oct 13, 2005
Loading...

Share This Page