querying persistent ruby objects in memory

[braver]

I have a data-mining task which loads data as a big XML tree (10+ MB)
and then reorganizes it. Even loading it with Hpricot takes 10-20
seconds. I don't want to do it for every manilupation I want to try,
especially for sequences of transformations.

Thus I wonder what's a good way to keep the huge object in memory
between the runs of querying scripts. Can Rails be used for that?
I'd rather avoid writing a client-server platform, or using it per se,
unless there's already an existing one. A vague intuition is, it
should be something like threads -- one thread parses XML and keeps it
in memory, another starts up later, somehow joins the memory space of
the first one, queries/transforms it, and ends. Then other queries/
transformations can all be run. Is there anything like it?

Cheers,
Alexy

 

[Robert Klemme]

I have a data-mining task which loads data as a big XML tree (10+ MB)
and then reorganizes it. Even loading it with Hpricot takes 10-20
seconds. I don't want to do it for every manilupation I want to try,
especially for sequences of transformations.

Thus I wonder what's a good way to keep the huge object in memory
between the runs of querying scripts. Can Rails be used for that?
I'd rather avoid writing a client-server platform, or using it per se,
unless there's already an existing one. A vague intuition is, it
should be something like threads -- one thread parses XML and keeps it
in memory, another starts up later, somehow joins the memory space of
the first one, queries/transforms it, and ends. Then other queries/
transformations can all be run. Is there anything like it?

I'd consider using Marshal.

Kind regards

robert

 

[braver]

I'd consider using Marshal.

That's just plain serialization, isn't it? I've seen that and
Madelaine; but my wish is to keep the objects in memory without the
need to dump/reload it, however fast. (That would be a last resort.)

The question is, can we keep an object in memory in one thread, and
explore/change it from another? In the worst case, we can probably
quickly dump an object into a memory region and reload it back via
Marshal -- I guess a crude solution is forming here, using shared
memory or RAM disk -- have to see what's there for macs... But still
I wonder what folks think in terms of all kinds of RAM persistence in
ruby solutions.

Cheers,
Alexy