Python - what is the fastest database ?

[martijn]

H!,

I'm testing things with Python with databases.
But I have one big question.

What is the 'fastest' database for the internet in combination with
Python ?

- with +/- 15 GB data.
- fast internet SELECT query's.

Python use bsddb but could he handle 15 GB fast ?

A other question is:
How is it possible that google (super big database) is super fast?
What type database do they use / software ?

Thanks Very Much,
GC-Martijn

 

[PA]

What type database do they use / software ?

Hmmm... they don't use a "database" in the traditional sense of the
term.

http://www-db.stanford.edu/~backrub/google.html

Cheers

 

[Peter Hansen]

Hmmm... they don't use a "database" in the traditional sense of the term.

http://www-db.stanford.edu/~backrub/google.html

Section "4.3 Crawling the Web" in that discusses the role Python
plays in Google, at least in the form it (Google) had when this
paper was written.

-Peter

 

[pyguy2]

It depends on what you mean by database.

If you want really fast I/O, try pytables.
"PyTables is a hierarchical database package designed to efficiently
manage very large amounts of data."

http://pytables.sourceforge.net/html/WelcomePage.html

some more comments from the webpage:

# High performance I/O: On modern systems, and for large amounts of
data, tables and array objects can be read and written at a speed only
limited by the performance of the underlying I/O subsystem. Moreover,
if your data is compressible, even faster than your I/O maximum
throughput (!).

# Support of files bigger than 2 GB: So that you won't be limited if
you want to deal with very large datasets. In fact, PyTables support
full 64-bit file addressing even on 32-bit platforms (provided that the
underlying filesystem does so too, of course).

# Architecture-independent: PyTables has been carefully coded (as HDF5
itself) with little-endian/big-endian byte orderings issues in mind .
So, you can write a file in a big-endian machine (like a Sparc or MIPS)
and read it in other little-endian (like Intel or Alpha) without
problems.

# Portability: PyTables has been ported to many architectures, namely
Linux, Windows, MacOSX, FreeBSD, Solaris, IRIX and probably works in
many more. Moreover, it runs just fine also in 64 bit plaforms (like
AMD64, Intel64, UltraSparc or MIPS RXX000 processors).

 

[Terry Reedy]

What type database do they use / software ?

On the hardware side, Google's secret is massively parallel cluster
computing, coupled with proprietary software for splitting tasks and
joining results. They have perhaps 200,000 CPUs. A query might be given
to hundreds of them for a fraction of a second.

Terry J. Reedy

 

[km]

Hi all,

Google has specially designed file system 'Goolgle File System' too.

KM
-------------------------------------------------------------------------

 

[yasso]

Fastest Database ? Try Lexst

Actually google doesn't use "database",


Lexst-database cluster actually is not a database, their technology is partitioning the data to numerous servers, then the searching tasks are distributed to all servers. so Lexst claims they can search billions of rows while keep the performance. you can see lexst.com here.