Re: best way to handle this in Python

Discussion in 'Python' started by Ian Kelly, Jul 20, 2012.

  1. Ian Kelly

    Ian Kelly Guest

    On Fri, Jul 20, 2012 at 4:34 AM, Rita <> wrote:
    > Thats an interesting data structure Dennis. I will actually be running this
    > type of query many times preferable in an ad-hoc environment. That makes it
    > tough for sqlite3 since there will be several hundred thousand tuples.


    Several hundred thousand is not an enormous number. I think you're
    underestimating sqlite3. I just tried a test with one million tuples,
    six colors per tuple (six million rows altogether). Each row contains
    a primary key, a timestamp, a color, and a count, with an index on the
    timestamp column. Building the database from scratch took about a
    minute; adding the index took about another minute. Incremental
    updates would of course be much faster. Queries like "select * from
    data where timestamp between 500000 and 600000" return instantly (from
    a user perspective).

    Cheers,
    Ian
    Ian Kelly, Jul 20, 2012
    #1
    1. Advertising

  2. On Fri, 20 Jul 2012 12:14:30 -0600, Ian Kelly wrote:

    > On Fri, Jul 20, 2012 at 4:34 AM, Rita <> wrote:
    >> Thats an interesting data structure Dennis. I will actually be running
    >> this type of query many times preferable in an ad-hoc environment. That
    >> makes it tough for sqlite3 since there will be several hundred thousand
    >> tuples.

    >
    > Several hundred thousand is not an enormous number. I think you're
    > underestimating sqlite3.



    A common trap, and not just for sqlite. I frequently have to remind
    people -- including myself -- that what is a lot of data for you may not
    be a lot of data for your computer.




    --
    Steven
    Steven D'Aprano, Jul 21, 2012
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ravikanth[MVP]
    Replies:
    6
    Views:
    3,884
    Aemca
    Jul 18, 2003
  2. Thomas Scheiderich

    Best way to handle documents in ASP.NET

    Thomas Scheiderich, May 20, 2004, in forum: ASP .Net
    Replies:
    11
    Views:
    2,483
    Jim Corey
    May 20, 2004
  3. Dave Angel

    Re: best way to handle this in Python

    Dave Angel, Jul 20, 2012, in forum: Python
    Replies:
    0
    Views:
    159
    Dave Angel
    Jul 20, 2012
  4. Dennis Lee Bieber

    Re: best way to handle this in Python

    Dennis Lee Bieber, Jul 20, 2012, in forum: Python
    Replies:
    0
    Views:
    167
    Dennis Lee Bieber
    Jul 20, 2012
  5. Dennis Lee Bieber

    Re: best way to handle this in Python

    Dennis Lee Bieber, Jul 20, 2012, in forum: Python
    Replies:
    0
    Views:
    147
    Dennis Lee Bieber
    Jul 20, 2012
Loading...

Share This Page