querying persistent ruby objects in memory

Discussion in 'Ruby' started by braver, May 26, 2007.

  1. braver

    braver Guest

    I have a data-mining task which loads data as a big XML tree (10+ MB)
    and then reorganizes it. Even loading it with Hpricot takes 10-20
    seconds. I don't want to do it for every manilupation I want to try,
    especially for sequences of transformations.

    Thus I wonder what's a good way to keep the huge object in memory
    between the runs of querying scripts. Can Rails be used for that?
    I'd rather avoid writing a client-server platform, or using it per se,
    unless there's already an existing one. A vague intuition is, it
    should be something like threads -- one thread parses XML and keeps it
    in memory, another starts up later, somehow joins the memory space of
    the first one, queries/transforms it, and ends. Then other queries/
    transformations can all be run. Is there anything like it?

    Cheers,
    Alexy
    braver, May 26, 2007
    #1
    1. Advertising

  2. On 26.05.2007 23:00, braver wrote:
    > I have a data-mining task which loads data as a big XML tree (10+ MB)
    > and then reorganizes it. Even loading it with Hpricot takes 10-20
    > seconds. I don't want to do it for every manilupation I want to try,
    > especially for sequences of transformations.
    >
    > Thus I wonder what's a good way to keep the huge object in memory
    > between the runs of querying scripts. Can Rails be used for that?
    > I'd rather avoid writing a client-server platform, or using it per se,
    > unless there's already an existing one. A vague intuition is, it
    > should be something like threads -- one thread parses XML and keeps it
    > in memory, another starts up later, somehow joins the memory space of
    > the first one, queries/transforms it, and ends. Then other queries/
    > transformations can all be run. Is there anything like it?


    I'd consider using Marshal.

    Kind regards

    robert
    Robert Klemme, May 26, 2007
    #2
    1. Advertising

  3. braver

    braver Guest

    On May 26, 2:24 pm, Robert Klemme <> wrote:
    > I'd consider using Marshal.


    That's just plain serialization, isn't it? I've seen that and
    Madelaine; but my wish is to keep the objects in memory without the
    need to dump/reload it, however fast. (That would be a last resort.)

    The question is, can we keep an object in memory in one thread, and
    explore/change it from another? In the worst case, we can probably
    quickly dump an object into a memory region and reload it back via
    Marshal -- I guess a crude solution is forming here, using shared
    memory or RAM disk -- have to see what's there for macs... But still
    I wonder what folks think in terms of all kinds of RAM persistence in
    ruby solutions.

    Cheers,
    Alexy
    braver, May 26, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Perry St-Germain

    C++ and persistent objects

    Perry St-Germain, Nov 18, 2003, in forum: C++
    Replies:
    2
    Views:
    766
    Jeff F
    Nov 18, 2003
  2. Paul Rubin

    Persistent objects

    Paul Rubin, Dec 12, 2004, in forum: Python
    Replies:
    16
    Views:
    535
    Paul Rubin
    Dec 14, 2004
  3. gk
    Replies:
    7
    Views:
    943
    Tom Anderson
    Oct 12, 2010
  4. Farrel Lifson
    Replies:
    1
    Views:
    68
    Brian Candler
    Nov 9, 2008
  5. Casimir Pohjanraito

    Persistent Objects with Ruby - simple beginning

    Casimir Pohjanraito, Dec 16, 2008, in forum: Ruby
    Replies:
    2
    Views:
    105
    Brian Candler
    Dec 16, 2008
Loading...

Share This Page