In-memory Relational Data Storage

C

Curt Sampson

I've been dealing with some slightly more complex groups of data lately.
Say, something like this:

I have a dozen Libraries. Each library carries some volumes of zero
or more Journals. For example, Library L1 might have volumes 1-35 of
Journal J1, and Library L2 might have volumes 25-40 of Journal J1. Each
Volume has one or more Articles in it, and the same Article might be
found in multiple Volumes of the same or different Journals.

So I make classes for all of these and started hooking them up together,
beginning with the obvious tree structure (Libraries reference Journals,
which reference Volumes, which reference Articles.) But then I realize
that I had some lookups I wanted to do that needed some different types
of references, for example, "name all the Journals that have published
Article A37."

So I start adding more ways of doing efficient lookups, start to worry about
various data structures getting out of sync, get some fairly nasty code
to search down through tree structures, and start to become unhappy.

And then it hits me. The source of all my problems is that, stuck in my
'A has a B' reference rut, I've just constructed a hierarchial database,
which is the source of all my pain.

Well, this is a problem we solved forty years ago when Codd came up with
the idea of relational databases. So, has anybody built some sort of
relational storage system for Ruby that will let me describe my data in
some sort of E/R-ish way and get stuff out from any direction I care to use?

cjs
 
R

Robert Klemme

Curt Sampson said:
I've been dealing with some slightly more complex groups of data lately.
Say, something like this:

I have a dozen Libraries. Each library carries some volumes of zero
or more Journals. For example, Library L1 might have volumes 1-35 of
Journal J1, and Library L2 might have volumes 25-40 of Journal J1. Each
Volume has one or more Articles in it, and the same Article might be
found in multiple Volumes of the same or different Journals.

So I make classes for all of these and started hooking them up together,
beginning with the obvious tree structure (Libraries reference Journals,
which reference Volumes, which reference Articles.) But then I realize
that I had some lookups I wanted to do that needed some different types
of references, for example, "name all the Journals that have published
Article A37."

So I start adding more ways of doing efficient lookups, start to worry about
various data structures getting out of sync, get some fairly nasty code
to search down through tree structures, and start to become unhappy.

And then it hits me. The source of all my problems is that, stuck in my
'A has a B' reference rut, I've just constructed a hierarchial database,
which is the source of all my pain.

Well, this is a problem we solved forty years ago when Codd came up with
the idea of relational databases. So, has anybody built some sort of
relational storage system for Ruby that will let me describe my data in
some sort of E/R-ish way and get stuff out from any direction I care to
use?

You probably just need bidrectional relations plus hashes as indexes. You
could define some utility code that establishes a relation between two
classes and a specialized indexing class that tracks attribute changes
automatically.

Something along this rough outline:

require 'set'

class Foo; attr_accessor :bar end

class Index
INDEXES = {}

def initialize(cl, field)
name = "#{cl.name}::#{field}"
@cl = cl
@field = field
@h = Hash.new {|h,k| h[k]=Set.new}

self.class::INDEXES[name] = self

@cl.class_eval "def #{field}=(x) idx = Index::INDEXES['#{name}'];
idx.remove(self, @#{field}); @#{field}=x; idx.add(self, @#{field}) end"
end

def add(obj, val)
@h[val].add(obj)
end

def remove(obj, val)
@h[val].delete(obj)
end

def get_all(val)
@h[val]
end
end

Index.new(Foo, :bar)
f=Foo.new
f.bar = 100
Index::INDEXES["Foo::bar"].get_all(100)

I would not use a relational approach because you loose the expressiveness
of an OO language.

Kind regards

robert
 
S

Shashank Date

Curt Sampson wrote:
Well, this is a problem we solved forty years ago when Codd came up with
the idea of relational databases. So, has anybody built some sort of
relational storage system for Ruby that will let me describe my data in
some sort of E/R-ish way and get stuff out from any direction I care to use?

You may want to take a look at Metakit (http://equi4.com).
Not exactly relational, but definitely "in-memory".

Quoting from the website:

" Metakit is an efficient embedded database library with a small
footprint. It fills the gap between flat-file, relational,
object-oriented, and tree-structured databases, supporting
relational joins, serialization, nested structures, and instant
schema evolution. "

And from an article in Dr.Dobbs (Pg. 65, Dec'04):

"Metakit databases are called 'storages' and they can either be
on disk or completely in-memory".

However, there are no Ruby bindings (yet) :-(
-- shanko
 
E

Eivind Eklund

Well, this is a problem we solved forty years ago when Codd came up with
the idea of relational databases. So, has anybody built some sort of
relational storage system for Ruby that will let me describe my data in
some sort of E/R-ish way and get stuff out from any direction I care to use?

I am have code-in-progress that is destinied to do this - an
implementation of the relational algebra directly in Ruby. At the
moment it is generating SQL and I'm working on implementing full
support for relational difference. When I've got the SQL generation
down, I was planning to use those tests to "lock down" aspects of the
functionality, and then mutate the code to support more table types
and doing operations across table types.

I unfortunately lack any date for when this will be finished; there's
two or three fairly large "chunks" between here and there (difference
implementation, possibly divide implementation, refactoring to support
in-memory tables), and there's a lot of things going on in my life at
the moment.

Eivind.
 
A

Avi Bryant

Eivind Eklund said:
I am have code-in-progress that is destinied to do this - an
implementation of the relational algebra directly in Ruby. At the
moment it is generating SQL and I'm working on implementing full
support for relational difference. When I've got the SQL generation
down, I was planning to use those tests to "lock down" aspects of the
functionality, and then mutate the code to support more table types
and doing operations across table types.

If you read Smalltalk, I have an implementation of something similar
that you might be interested in. It's called Relational Object
Expressions (ROE, meant to sound like rho and row... but what kind of
roe is it? Codd roe, of course...). Unfortunately there's not really
any documentation, but there are test cases, and you can ask me if you
need help understanding what I'm doing.

The in-memory table type is extremely naive and slow, however - it's
used solely for running tests (to compare results with the same tests
run using SQL generation).

http://map1.squeakfoundation.org/sm/package/81c0253b-3baa-4253-bca1-15819cd25366

Avi
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top