In-memory Relational Data Storage

Discussion in 'Ruby' started by Curt Sampson, Nov 18, 2004.

  1. Curt Sampson

    Curt Sampson Guest

    I've been dealing with some slightly more complex groups of data lately.
    Say, something like this:

    I have a dozen Libraries. Each library carries some volumes of zero
    or more Journals. For example, Library L1 might have volumes 1-35 of
    Journal J1, and Library L2 might have volumes 25-40 of Journal J1. Each
    Volume has one or more Articles in it, and the same Article might be
    found in multiple Volumes of the same or different Journals.

    So I make classes for all of these and started hooking them up together,
    beginning with the obvious tree structure (Libraries reference Journals,
    which reference Volumes, which reference Articles.) But then I realize
    that I had some lookups I wanted to do that needed some different types
    of references, for example, "name all the Journals that have published
    Article A37."

    So I start adding more ways of doing efficient lookups, start to worry about
    various data structures getting out of sync, get some fairly nasty code
    to search down through tree structures, and start to become unhappy.

    And then it hits me. The source of all my problems is that, stuck in my
    'A has a B' reference rut, I've just constructed a hierarchial database,
    which is the source of all my pain.

    Well, this is a problem we solved forty years ago when Codd came up with
    the idea of relational databases. So, has anybody built some sort of
    relational storage system for Ruby that will let me describe my data in
    some sort of E/R-ish way and get stuff out from any direction I care to use?

    cjs
    --
    Curt Sampson <> +81 90 7737 2974 http://www.NetBSD.org
    Make up enjoying your city life...produced by BIC CAMERA
    Curt Sampson, Nov 18, 2004
    #1
    1. Advertising

  2. "Curt Sampson" <> schrieb im Newsbeitrag
    news:p...
    >
    > I've been dealing with some slightly more complex groups of data lately.
    > Say, something like this:
    >
    > I have a dozen Libraries. Each library carries some volumes of zero
    > or more Journals. For example, Library L1 might have volumes 1-35 of
    > Journal J1, and Library L2 might have volumes 25-40 of Journal J1. Each
    > Volume has one or more Articles in it, and the same Article might be
    > found in multiple Volumes of the same or different Journals.
    >
    > So I make classes for all of these and started hooking them up together,
    > beginning with the obvious tree structure (Libraries reference Journals,
    > which reference Volumes, which reference Articles.) But then I realize
    > that I had some lookups I wanted to do that needed some different types
    > of references, for example, "name all the Journals that have published
    > Article A37."
    >
    > So I start adding more ways of doing efficient lookups, start to worry

    about
    > various data structures getting out of sync, get some fairly nasty code
    > to search down through tree structures, and start to become unhappy.
    >
    > And then it hits me. The source of all my problems is that, stuck in my
    > 'A has a B' reference rut, I've just constructed a hierarchial database,
    > which is the source of all my pain.
    >
    > Well, this is a problem we solved forty years ago when Codd came up with
    > the idea of relational databases. So, has anybody built some sort of
    > relational storage system for Ruby that will let me describe my data in
    > some sort of E/R-ish way and get stuff out from any direction I care to

    use?

    You probably just need bidrectional relations plus hashes as indexes. You
    could define some utility code that establishes a relation between two
    classes and a specialized indexing class that tracks attribute changes
    automatically.

    Something along this rough outline:

    require 'set'

    class Foo; attr_accessor :bar end

    class Index
    INDEXES = {}

    def initialize(cl, field)
    name = "#{cl.name}::#{field}"
    @cl = cl
    @field = field
    @h = Hash.new {|h,k| h[k]=Set.new}

    self.class::INDEXES[name] = self

    @cl.class_eval "def #{field}=(x) idx = Index::INDEXES['#{name}'];
    idx.remove(self, @#{field}); @#{field}=x; idx.add(self, @#{field}) end"
    end

    def add(obj, val)
    @h[val].add(obj)
    end

    def remove(obj, val)
    @h[val].delete(obj)
    end

    def get_all(val)
    @h[val]
    end
    end

    Index.new(Foo, :bar)
    f=Foo.new
    f.bar = 100
    Index::INDEXES["Foo::bar"].get_all(100)

    I would not use a relational approach because you loose the expressiveness
    of an OO language.

    Kind regards

    robert
    Robert Klemme, Nov 18, 2004
    #2
    1. Advertising

  3. Curt Sampson wrote:
    <snip>

    > Well, this is a problem we solved forty years ago when Codd came up with
    > the idea of relational databases. So, has anybody built some sort of
    > relational storage system for Ruby that will let me describe my data in
    > some sort of E/R-ish way and get stuff out from any direction I care to use?


    You may want to take a look at Metakit (http://equi4.com).
    Not exactly relational, but definitely "in-memory".

    Quoting from the website:

    " Metakit is an efficient embedded database library with a small
    footprint. It fills the gap between flat-file, relational,
    object-oriented, and tree-structured databases, supporting
    relational joins, serialization, nested structures, and instant
    schema evolution. "

    And from an article in Dr.Dobbs (Pg. 65, Dec'04):

    "Metakit databases are called 'storages' and they can either be
    on disk or completely in-memory".

    However, there are no Ruby bindings (yet) :-(
    -- shanko
    Shashank Date, Nov 19, 2004
    #3
  4. On Thu, 18 Nov 2004 14:43:42 +0900, Curt Sampson <> wrote:
    > Well, this is a problem we solved forty years ago when Codd came up with
    > the idea of relational databases. So, has anybody built some sort of
    > relational storage system for Ruby that will let me describe my data in
    > some sort of E/R-ish way and get stuff out from any direction I care to use?


    I am have code-in-progress that is destinied to do this - an
    implementation of the relational algebra directly in Ruby. At the
    moment it is generating SQL and I'm working on implementing full
    support for relational difference. When I've got the SQL generation
    down, I was planning to use those tests to "lock down" aspects of the
    functionality, and then mutate the code to support more table types
    and doing operations across table types.

    I unfortunately lack any date for when this will be finished; there's
    two or three fairly large "chunks" between here and there (difference
    implementation, possibly divide implementation, refactoring to support
    in-memory tables), and there's a lot of things going on in my life at
    the moment.

    Eivind.
    --
    Hazzle free packages for Ruby?
    RPA is available from http://www.rubyarchive.org/
    Eivind Eklund, Nov 19, 2004
    #4
  5. Curt Sampson

    Avi Bryant Guest

    Eivind Eklund <> wrote in message news:<>...

    > I am have code-in-progress that is destinied to do this - an
    > implementation of the relational algebra directly in Ruby. At the
    > moment it is generating SQL and I'm working on implementing full
    > support for relational difference. When I've got the SQL generation
    > down, I was planning to use those tests to "lock down" aspects of the
    > functionality, and then mutate the code to support more table types
    > and doing operations across table types.


    If you read Smalltalk, I have an implementation of something similar
    that you might be interested in. It's called Relational Object
    Expressions (ROE, meant to sound like rho and row... but what kind of
    roe is it? Codd roe, of course...). Unfortunately there's not really
    any documentation, but there are test cases, and you can ask me if you
    need help understanding what I'm doing.

    The in-memory table type is extremely naive and slow, however - it's
    used solely for running tests (to compare results with the same tests
    run using SQL generation).

    http://map1.squeakfoundation.org/sm/package/81c0253b-3baa-4253-bca1-15819cd25366

    Avi
    Avi Bryant, Nov 20, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Carse
    Replies:
    1
    Views:
    3,643
    Carse
    Jan 28, 2005
  2. Joe Estock
    Replies:
    7
    Views:
    353
    Karl Heinz Buchegger
    Nov 18, 2003
  3. Alex Hunsley
    Replies:
    2
    Views:
    457
    dimitar
    Jun 2, 2006
  4. sarathy
    Replies:
    2
    Views:
    649
    sarathy
    Jul 17, 2006
  5. Paul McGuire
    Replies:
    0
    Views:
    177
    Paul McGuire
    Oct 18, 2010
Loading...

Share This Page