Loading to Memory

[hillgoogle]

On my unix box I want to load some data into memory using one program
and then retrieve it using another.

Anyone have any examples of how I can load that to my unix machine
memory?

Mike

 

[Michael Vilain]

On my unix box I want to load some data into memory using one program
and then retrieve it using another.

Anyone have any examples of how I can load that to my unix machine
memory?

Mike

In Unix, you have to create a shared region of memory and coordinate
access to it somehow. Oracle does this with their SGA. There may be a
way to do this using a CPAN module but it all depends on how much memory
you're talking about. Oracle requires special OS configuration, at
least on Solaris, to allow the OS to permit an application to allocate
shared memory in the GB range. Don't know how you'd do this in Linux.
How you coordinate access the the shared memory once it's created is
something you'll have to work out. This is a rather advanced
programming topic. You'll have to do a lot of reading and trial coding
to get something you want. Perl may not be the best tool for this.

The modules I got when I searched the CPAN library span are for using
shared memory for persistence between sessions. Not really what you're
looking for since it's being used to implement sessions.

 

[Jürgen Exner]

On my unix box I want to load some data into memory using one program
and then retrieve it using another.

Anyone have any examples of how I can load that to my unix machine
memory?

Just an idea: you may want to search for IPC (interprocess
communication) using shared memory. Maybe this will get you some
pointers.

jue

 

[Dr.Ruud]

On my unix box I want to load some data into memory using one program
and then retrieve it using another.

Anyone have any examples of how I can load that to my unix machine
memory?

What do you need to do exactly? Any good OS cleverly caches the file
system, so maybe you can just use a file?

 

[Tobias Nissen]

On my unix box I want to load some data into memory using one program
and then retrieve it using another.

Anyone have any examples of how I can load that to my unix machine
memory?

You're not very clear about what problem you are trying to solve. In
this generality I'd suggest using a key-value store like memcached¹ or
redis². Both have packages available on all major Linux/Unix
distributions and are pretty easy to set up and use.

¹ http://memcached.org/
http://search.cpan.org/~mdorman/Memcached-Client-2.01/lib/Memcached/Client.pm
² http://redis.io/
http://search.cpan.org/~melo/Redis-1.904/lib/Redis.pm

 

[hillgoogle]

Thanks all. Specifically, I'm looking at getting output from Memory
instaed of a DB because memory is volatile and I it read it faster.
So ... I would insert data from my DB (mysql) and then my app would
read data NOT from the DB, but the memory. Right, when the machine
went down I would re-load to memory, or when I had updates from the
DB.

@Tobias ... I'll look at those links, ty!

@Others ... Anything else?

TY, Mike

 

[Tobias Nissen]

Thanks all. Specifically, I'm looking at getting output from Memory
instaed of a DB because memory is volatile and I it read it faster.
So ... I would insert data from my DB (mysql) and then my app would
read data NOT from the DB, but the memory. Right, when the machine
went down I would re-load to memory, or when I had updates from the
DB.

That's one of the use cases of memcached. However, when using memcached
one tries to take the burden of the DB not only because of the "memory
is faster" argument, but also to avoid doing the same computations over
and over again. Using memcached requires changes to your source code
and in most cases program structure.

If you just want quick access and don't have complex computations to
do, just upgrading your RAM could be enough. Modern DBs are doing
their own caching (in memory) and they are doing it well.

 

[Ted Zlatanov]

On Thu, 25 Aug 2011 15:31:55 -0700 (PDT) (e-mail address removed) wrote:

h> Thanks all. Specifically, I'm looking at getting output from Memory
h> instaed of a DB because memory is volatile and I it read it faster.
h> So ... I would insert data from my DB (mysql) and then my app would
h> read data NOT from the DB, but the memory. Right, when the machine
h> went down I would re-load to memory, or when I had updates from the
h> DB.

h> @Tobias ... I'll look at those links, ty!

h> @Others ... Anything else?

As Tobias suggested memcached is your best bet today IMHO. It's
extremely easy to install, configure, use locally and remotely, and
understand. I've used it in production with 16GB of memory allocated to
it (specialized status monitoring application) and it has performed
flawlessly *without a restart*, no memory leaks or any other issues for
almost a year or continuous usage.

Ted

 

[Dr.Ruud]

On my unix box I want to load some data into memory using one program
and then retrieve it using another.

Anyone have any examples of how I can load that to my unix machine
memory?

Check out CHI, which is in the Task::Kensho list.

http://search.cpan.org/perldoc?CHI
http://search.cpan.org/perldoc?Task::Kensho

 

[Jorgen Grahn]

Thanks all. Specifically, I'm looking at getting output from Memory
instaed of a DB because memory is volatile and I it read it faster.
So ... I would insert data from my DB (mysql) and then my app would
read data NOT from the DB, but the memory. Right, when the machine
went down I would re-load to memory, or when I had updates from the
DB.

All of that is the job of the database software (and it might delegate
that work to the OS, which caches disk accesses in memory).

If you have nothing better to do, and you know that in a few seconds
you will want to access some part of the database, it *might* make
sense to do "dummy reads" from the database, or perhaps a
"cat the_database_file >/dev/null". Or it might not.

/Jorgen

 

[Dr.Ruud]

All of that is the job of the database software (and it might delegate
that work to the OS, which caches disk accesses in memory).

If you have nothing better to do, and you know that in a few seconds
you will want to access some part of the database, it *might* make
sense to do "dummy reads" from the database, or perhaps a
"cat the_database_file>/dev/null". Or it might not.

A hot database is indeed often much faster than a cold one. Further make
sure that you added the right indexes, and are using the proper storage
engine (like InnoDB vs. MyISAM).

The MySQL way for "cat the_database_file>/dev/null" is 'CHECKSUM TABLE
mytable'. If the number of concurrent users is normally less than 20,
consider using SQLite.

If you have popular big queries with big result sets, also look into
tuning the query cache settings. And look into replication, to do all
reading from a slave.