Database in memory

[Jim]

I have an application that will maintain an in-memory database in the
form of a list of lists. Does anyone know of a way to search for and
retreive "records" from such a structure?

Many thanks,
bootkey

 

[Gabriel Genellina]

I have an application that will maintain an in-memory database in the
form of a list of lists. Does anyone know of a way to search for and
retreive "records" from such a structure?

Why not a true database? SQLite can handle an in-memory database.

 

[Marc 'BlackJack' Rintsch]

I have an application that will maintain an in-memory database in the
form of a list of lists. Does anyone know of a way to search for and
retreive "records" from such a structure?

Scan the list of lists with a ``for`` loop. Or build indexes with
dictionaries.

Ciao,
Marc 'BlackJack' Rintsch

 

[Jeremy Sanders]

I have an application that will maintain an in-memory database in the
form of a list of lists. Does anyone know of a way to search for and
retreive "records" from such a structure?

The dictionary is the obvious way to index things:

# items consist of name and age
data = [
['fred', 42],
['jim', 16], ...
]

name_index = {}
for item in data:
name_index[item[0]] = item

name_index['fred']

['fred', 42]

Dictionaries are one of the most useful things in Python. Make sure you know
how to take adavantage of them...

Jeremy

 

[Travis Oliphant]

I have an application that will maintain an in-memory database in the
form of a list of lists. Does anyone know of a way to search for and
retreive "records" from such a structure?

Actually, the new NumPy can work as a very-good fast and efficient
simple in-memory database (or memory-mapped data-base for that matter).

The elements of a NumPy array can be arbitrary records. You would
search using logical combinations of comparision. I think the ability
for NumPy (which now handles arbitrary records) to be used as a
data-base is under-appreciated.

Mind you, it is SQL-less. NumPy only provides the "tables" it does not
provide the fancy logic on-top of the tables. So, perhaps it would be
better to say that NumPy could serve as the foundation for a simple
data-base application.

-Travis

 

[Hendrik van Rooyen]

Dictionaries are one of the most useful things in Python. Make sure you know
how to take adavantage of them...

+1 for QOTW

- Hendrik

 

[Nicko]

I have an application that will maintain an in-memory database in the
form of a list of lists. Does anyone know of a way to search for and
retreive "records" from such a structure?

The answer very much depends on the manner in which you want to do the
look-up. If you only need to do exact-match look-up for items with
unique keys (e.g. find the single record where SSN=1234567890) then
using a dictionary is by far the best solution. It's fast and it's
easy.

If you expect to do exact-match look-up where the keys are not unique
then build a dictionary containing 'set' objects which are the sets of
records which have the given key. This lets you neatly find the
intersection of selections on multiple criteria (e.g. matches =
zipcode_index["94101"] & hometype_index["condo"] ).

If you need to do range matching (e.g. 20000 <= salary < 50000) then
your best bet is to keep a list of the records sorted in the ordering
of the key, do a binary search to find where the lower and upper
bounds lie within the sorted list and then take a slice. If you also
have some index dictionaries containing sets then you can combine
these two methods with something like 'matches =
set(salary_index[lo_sal:hi_sal]) & zipcode_index["81435"] '

Having said all that, if you think that there is any possibility that
you might ever want to expand the functionality of your program to
require either (a) more complex and flexible searching and/or (b)
putting the database somewhere else, then I would strongly suggest
that you use PySQLite. SQLite is an efficient in-memory database with
an SQL engine and the Python interface conforms to the DB-API spec, so
you won't need to change your code (much) if you want to move the
database to some MySQL, Oracle, Sybase or DB2 server at a later date.
Furthermore SQLite is included in Python 2.5 as standard.

 

[Nicko]

If you expect to do exact-match look-up where the keys are not unique
then build a dictionary containing 'set' objects which are the sets of
records which have the given key. This lets you neatly find the
intersection of selections on multiple criteria (e.g. matches =
zipcode_index["94101"] & hometype_index["condo"] ).

Just FYI, if you're going to go this route then the items that you are
indexing have to be hashable, which the built in 'list' type is not.
Tuples are, or you can make some custom class (or your own subtype of
list) which implements the __hash__ method based on some 'primary key'
value from your data. Or you could just go for SQLite...