Loading to Memory

Discussion in 'Perl Misc' started by hillgoogle@charter.net, Aug 24, 2011.

  1. Guest

    On my unix box I want to load some data into memory using one program
    and then retrieve it using another.

    Anyone have any examples of how I can load that to my unix machine
    memory?

    Mike
     
    , Aug 24, 2011
    #1
    1. Advertising

  2. In article
    <>,
    wrote:

    > On my unix box I want to load some data into memory using one program
    > and then retrieve it using another.
    >
    > Anyone have any examples of how I can load that to my unix machine
    > memory?
    >
    > Mike


    In Unix, you have to create a shared region of memory and coordinate
    access to it somehow. Oracle does this with their SGA. There may be a
    way to do this using a CPAN module but it all depends on how much memory
    you're talking about. Oracle requires special OS configuration, at
    least on Solaris, to allow the OS to permit an application to allocate
    shared memory in the GB range. Don't know how you'd do this in Linux.
    How you coordinate access the the shared memory once it's created is
    something you'll have to work out. This is a rather advanced
    programming topic. You'll have to do a lot of reading and trial coding
    to get something you want. Perl may not be the best tool for this.

    The modules I got when I searched the CPAN library span are for using
    shared memory for persistence between sessions. Not really what you're
    looking for since it's being used to implement sessions.

    --
    DeeDee, don't press that button! DeeDee! NO! Dee...
    [I filter all Goggle Groups posts, so any reply may be automatically ignored]
     
    Michael Vilain, Aug 25, 2011
    #2
    1. Advertising

  3. wrote:
    >On my unix box I want to load some data into memory using one program
    >and then retrieve it using another.
    >
    >Anyone have any examples of how I can load that to my unix machine
    >memory?


    Just an idea: you may want to search for IPC (interprocess
    communication) using shared memory. Maybe this will get you some
    pointers.

    jue
     
    Jürgen Exner, Aug 25, 2011
    #3
  4. Dr.Ruud Guest

    On 2011-08-25 00:49, wrote:

    > On my unix box I want to load some data into memory using one program
    > and then retrieve it using another.
    >
    > Anyone have any examples of how I can load that to my unix machine
    > memory?


    What do you need to do exactly? Any good OS cleverly caches the file
    system, so maybe you can just use a file?

    --
    Ruud
     
    Dr.Ruud, Aug 25, 2011
    #4
  5. wrote:
    > On my unix box I want to load some data into memory using one program
    > and then retrieve it using another.
    >
    > Anyone have any examples of how I can load that to my unix machine
    > memory?


    You're not very clear about what problem you are trying to solve. In
    this generality I'd suggest using a key-value store like memcached¹ or
    redis². Both have packages available on all major Linux/Unix
    distributions and are pretty easy to set up and use.

    ¹ http://memcached.org/
    http://search.cpan.org/~mdorman/Memcached-Client-2.01/lib/Memcached/Client.pm
    ² http://redis.io/
    http://search.cpan.org/~melo/Redis-1.904/lib/Redis.pm
     
    Tobias Nissen, Aug 25, 2011
    #5
  6. Guest

    Thanks all. Specifically, I'm looking at getting output from Memory
    instaed of a DB because memory is volatile and I it read it faster.
    So ... I would insert data from my DB (mysql) and then my app would
    read data NOT from the DB, but the memory. Right, when the machine
    went down I would re-load to memory, or when I had updates from the
    DB.

    @Tobias ... I'll look at those links, ty!

    @Others ... Anything else?

    TY, Mike
     
    , Aug 25, 2011
    #6
  7. wrote:
    > Thanks all. Specifically, I'm looking at getting output from Memory
    > instaed of a DB because memory is volatile and I it read it faster.
    > So ... I would insert data from my DB (mysql) and then my app would
    > read data NOT from the DB, but the memory. Right, when the machine
    > went down I would re-load to memory, or when I had updates from the
    > DB.


    That's one of the use cases of memcached. However, when using memcached
    one tries to take the burden of the DB not only because of the "memory
    is faster" argument, but also to avoid doing the same computations over
    and over again. Using memcached requires changes to your source code
    and in most cases program structure.

    If you just want quick access and don't have complex computations to
    do, just upgrading your RAM could be enough. Modern DBs are doing
    their own caching (in memory) and they are doing it well.
     
    Tobias Nissen, Aug 26, 2011
    #7
  8. Ted Zlatanov Guest

    On Thu, 25 Aug 2011 15:31:55 -0700 (PDT) wrote:

    h> Thanks all. Specifically, I'm looking at getting output from Memory
    h> instaed of a DB because memory is volatile and I it read it faster.
    h> So ... I would insert data from my DB (mysql) and then my app would
    h> read data NOT from the DB, but the memory. Right, when the machine
    h> went down I would re-load to memory, or when I had updates from the
    h> DB.

    h> @Tobias ... I'll look at those links, ty!

    h> @Others ... Anything else?

    As Tobias suggested memcached is your best bet today IMHO. It's
    extremely easy to install, configure, use locally and remotely, and
    understand. I've used it in production with 16GB of memory allocated to
    it (specialized status monitoring application) and it has performed
    flawlessly *without a restart*, no memory leaks or any other issues for
    almost a year or continuous usage.

    Ted
     
    Ted Zlatanov, Aug 26, 2011
    #8
  9. Dr.Ruud Guest

    Dr.Ruud, Aug 27, 2011
    #9
  10. Jorgen Grahn Guest

    On Thu, 2011-08-25, wrote:
    > Thanks all. Specifically, I'm looking at getting output from Memory
    > instaed of a DB because memory is volatile and I it read it faster.
    > So ... I would insert data from my DB (mysql) and then my app would
    > read data NOT from the DB, but the memory. Right, when the machine
    > went down I would re-load to memory, or when I had updates from the
    > DB.


    All of that is the job of the database software (and it might delegate
    that work to the OS, which caches disk accesses in memory).

    If you have nothing better to do, and you know that in a few seconds
    you will want to access some part of the database, it *might* make
    sense to do "dummy reads" from the database, or perhaps a
    "cat the_database_file >/dev/null". Or it might not.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
     
    Jorgen Grahn, Aug 27, 2011
    #10
  11. Dr.Ruud Guest

    On 2011-08-27 22:05, Jorgen Grahn wrote:
    > On Thu, 2011-08-25, wrote:


    >> Thanks all. Specifically, I'm looking at getting output from Memory
    >> instaed of a DB because memory is volatile and I it read it faster.
    >> So ... I would insert data from my DB (mysql) and then my app would
    >> read data NOT from the DB, but the memory. Right, when the machine
    >> went down I would re-load to memory, or when I had updates from the
    >> DB.

    >
    > All of that is the job of the database software (and it might delegate
    > that work to the OS, which caches disk accesses in memory).
    >
    > If you have nothing better to do, and you know that in a few seconds
    > you will want to access some part of the database, it *might* make
    > sense to do "dummy reads" from the database, or perhaps a
    > "cat the_database_file>/dev/null". Or it might not.


    A hot database is indeed often much faster than a cold one. Further make
    sure that you added the right indexes, and are using the proper storage
    engine (like InnoDB vs. MyISAM).

    The MySQL way for "cat the_database_file>/dev/null" is 'CHECKSUM TABLE
    mytable'. If the number of concurrent users is normally less than 20,
    consider using SQLite.

    If you have popular big queries with big result sets, also look into
    tuning the query cache settings. And look into replication, to do all
    reading from a slave.

    --
    Ruud
     
    Dr.Ruud, Aug 28, 2011
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Robert Zurer
    Replies:
    0
    Views:
    467
    Robert Zurer
    Jun 28, 2005
  2. David Segall
    Replies:
    2
    Views:
    452
    Thomas Kellerer
    Jan 2, 2007
  3. Replies:
    4
    Views:
    325
  4. edfialk
    Replies:
    0
    Views:
    194
    edfialk
    May 10, 2007
  5. ofir
    Replies:
    0
    Views:
    190
Loading...

Share This Page