Python - what is the fastest database ?

Discussion in 'Python' started by martijn@gamecreators.nl, Feb 28, 2005.

  1. Guest

    H!,

    I'm testing things with Python with databases.
    But I have one big question.

    What is the 'fastest' database for the internet in combination with
    Python ?

    - with +/- 15 GB data.
    - fast internet SELECT query's.

    Python use bsddb but could he handle 15 GB fast ?

    A other question is:
    How is it possible that google (super big database) is super fast?
    What type database do they use / software ?

    Thanks Very Much,
    GC-Martijn
    , Feb 28, 2005
    #1
    1. Advertising

  2. PA Guest

    PA, Feb 28, 2005
    #2
    1. Advertising

  3. Peter Hansen Guest

    PA wrote:
    > On Feb 28, 2005, at 13:10, wrote:
    >> What type database do they use / software ?

    >
    > Hmmm... they don't use a "database" in the traditional sense of the term.
    >
    > http://www-db.stanford.edu/~backrub/google.html


    Section "4.3 Crawling the Web" in that discusses the role Python
    plays in Google, at least in the form it (Google) had when this
    paper was written.

    -Peter
    Peter Hansen, Feb 28, 2005
    #3
  4. Guest

    It depends on what you mean by database.

    If you want really fast I/O, try pytables.
    "PyTables is a hierarchical database package designed to efficiently
    manage very large amounts of data."

    http://pytables.sourceforge.net/html/WelcomePage.html

    some more comments from the webpage:

    # High performance I/O: On modern systems, and for large amounts of
    data, tables and array objects can be read and written at a speed only
    limited by the performance of the underlying I/O subsystem. Moreover,
    if your data is compressible, even faster than your I/O maximum
    throughput (!).

    # Support of files bigger than 2 GB: So that you won't be limited if
    you want to deal with very large datasets. In fact, PyTables support
    full 64-bit file addressing even on 32-bit platforms (provided that the
    underlying filesystem does so too, of course).

    # Architecture-independent: PyTables has been carefully coded (as HDF5
    itself) with little-endian/big-endian byte orderings issues in mind .
    So, you can write a file in a big-endian machine (like a Sparc or MIPS)
    and read it in other little-endian (like Intel or Alpha) without
    problems.

    # Portability: PyTables has been ported to many architectures, namely
    Linux, Windows, MacOSX, FreeBSD, Solaris, IRIX and probably works in
    many more. Moreover, it runs just fine also in 64 bit plaforms (like
    AMD64, Intel64, UltraSparc or MIPS RXX000 processors).
    , Feb 28, 2005
    #4
  5. Terry Reedy Guest

    <> wrote in message
    news:...
    >> How is it possible that google (super big database) is super fast?

    > What type database do they use / software ?


    On the hardware side, Google's secret is massively parallel cluster
    computing, coupled with proprietary software for splitting tasks and
    joining results. They have perhaps 200,000 CPUs. A query might be given
    to hundreds of them for a fraction of a second.

    Terry J. Reedy
    Terry Reedy, Feb 28, 2005
    #5
  6. km Guest

    Hi all,

    Google has specially designed file system 'Goolgle File System' too.

    KM
    -------------------------------------------------------------------------
    On Mon, Feb 28, 2005 at 01:43:53PM -0500, Terry Reedy wrote:
    >
    > <> wrote in message
    > news:...
    > >> How is it possible that google (super big database) is super fast?

    > > What type database do they use / software ?

    >
    > On the hardware side, Google's secret is massively parallel cluster
    > computing, coupled with proprietary software for splitting tasks and
    > joining results. They have perhaps 200,000 CPUs. A query might be given
    > to hundreds of them for a fraction of a second.
    >
    > Terry J. Reedy
    >
    >
    >
    > --
    km, Mar 1, 2005
    #6
  7. yasso

    Joined:
    Apr 2, 2009
    Messages:
    1
    Fastest Database ? Try Lexst

    Actually google doesn't use "database",


    Lexst-database cluster actually is not a database, their technology is partitioning the data to numerous servers, then the searching tasks are distributed to all servers. so Lexst claims they can search billions of rows while keep the performance. you can see lexst.com here.
    Last edited: Jul 3, 2009
    yasso, Apr 2, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    396
  2. samuraisam
    Replies:
    4
    Views:
    1,060
    Carl Friedrich Bolz
    Feb 20, 2008
  3. per
    Replies:
    10
    Views:
    1,045
    Aaron Brady
    Jun 19, 2009
  4. Harry Haller
    Replies:
    1
    Views:
    209
  5. Chris Rebert
    Replies:
    0
    Views:
    270
    Chris Rebert
    Oct 28, 2012
Loading...

Share This Page