How to persist and query a hash lookup in memory with Perl ?

Discussion in 'Perl Misc' started by Jack, Nov 29, 2007.

  1. Jack

    Jack Guest

    Hi I have a web application that needs to on an adhoc basis query a
    hash lookup which should be persisted in memory - I have seen this
    module "Persistent::Memory - A Persistent Class implemented using
    Memory (RAM)" but I wanted to see if there was a much simpler approach
    to taking a hash map and making it 'callable' while persisted in
    memory using standard features not requiring a special Perl module
    install..

    Any thoughts would be appreciated.
    Thanks,
    Jack
     
    Jack, Nov 29, 2007
    #1
    1. Advertising

  2. Jack wrote:
    > Hi I have a web application that needs to on an adhoc basis query a
    > hash lookup which should be persisted in memory - I have seen this
    > module "Persistent::Memory - A Persistent Class implemented using
    > Memory (RAM)" but I wanted to see if there was a much simpler approach
    > to taking a hash map and making it 'callable' while persisted in
    > memory using standard features not requiring a special Perl module
    > install..
    >
    > Any thoughts would be appreciated.


    Why not a tied hash? See e.g. "perldoc DB_File".

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
     
    Gunnar Hjalmarsson, Nov 29, 2007
    #2
    1. Advertising

  3. Jack

    Guest

    On Nov 28, 6:37 pm, Gunnar Hjalmarsson <> wrote:
    > Jack wrote:
    > > Hi I have a web application that needs to on an adhoc basis query a
    > > hash lookup which should be persisted in memory - I have seen this
    > > module "Persistent::Memory - A Persistent Class implemented using
    > > Memory (RAM)" but I wanted to see if there was a much simpler approach
    > > to taking a hash map and making it 'callable' while persisted in
    > > memory using standard features not requiring a special Perl module
    > > install..

    >
    > > Any thoughts would be appreciated.

    >
    > Why not a tied hash? See e.g. "perldoc DB_File".
    >
    > --
    > Gunnar Hjalmarsson
    > Email:http://www.gunnar.cc/cgi-bin/contact.pl


    Hi and thank you however that looks too complex - just trying to put a
    hash in memory, why is that so hard to do ? Is there a simpler way ?
     
    , Nov 29, 2007
    #3
  4. wrote in
    news::

    > On Nov 28, 6:37 pm, Gunnar Hjalmarsson <> wrote:
    >> Jack wrote:
    >> > Hi I have a web application that needs to on an adhoc basis query a
    >> > hash lookup which should be persisted in memory - I have seen this
    >> > module "Persistent::Memory - A Persistent Class implemented using
    >> > Memory (RAM)" but I wanted to see if there was a much simpler
    >> > approach to taking a hash map and making it 'callable' while
    >> > persisted in memory using standard features not requiring a special
    >> > Perl module install..

    >>
    >> > Any thoughts would be appreciated.

    >>
    >> Why not a tied hash? See e.g. "perldoc DB_File".
    >>
    >> --
    >> Gunnar Hjalmarsson
    >> Email:http://www.gunnar.cc/cgi-bin/contact.pl


    Don't quote sigs.

    > Hi and thank you however that looks too complex - just trying to put a
    > hash in memory, why is that so hard to do ? Is there a simpler way ?


    It looks like programming might be too hard for you.

    Sinan

    --
    A. Sinan Unur <>
    (remove .invalid and reverse each component for email address)
    clpmisc guidelines: <URL:http://www.augustmail.com/~tadmc/clpmisc.shtml>
     
    A. Sinan Unur, Nov 29, 2007
    #4
  5. Jack

    Ted Zlatanov Guest

    On Wed, 28 Nov 2007 18:22:34 -0800 (PST) Jack <> wrote:

    J> Hi I have a web application that needs to on an adhoc basis query a
    J> hash lookup which should be persisted in memory - I have seen this
    J> module "Persistent::Memory - A Persistent Class implemented using
    J> Memory (RAM)" but I wanted to see if there was a much simpler approach
    J> to taking a hash map and making it 'callable' while persisted in
    J> memory using standard features not requiring a special Perl module
    J> install..

    Are you talking about sharing the data between several instances of your
    web application, so whichever one handles a request can read or update
    data?

    Use a database. There are cases where this is not the right approach,
    but it doesn't sound like it in yours.

    Ted
     
    Ted Zlatanov, Nov 29, 2007
    #5
  6. Jack

    Guest

    wrote:
    > On Nov 28, 6:37 pm, Gunnar Hjalmarsson <> wrote:
    > > Jack wrote:
    > > > Hi I have a web application that needs to on an adhoc basis query a
    > > > hash lookup which should be persisted in memory - I have seen this
    > > > module "Persistent::Memory - A Persistent Class implemented using
    > > > Memory (RAM)" but I wanted to see if there was a much simpler
    > > > approach to taking a hash map and making it 'callable' while
    > > > persisted in memory using standard features not requiring a special
    > > > Perl module install..

    > >
    > > > Any thoughts would be appreciated.

    > >
    > > Why not a tied hash? See e.g. "perldoc DB_File".

    >
    > Hi and thank you however that looks too complex - just trying to put a
    > hash in memory, why is that so hard to do ? Is there a simpler way ?


    If you aren't willing to install modules and you aren't willing to use the
    ones that are (often) standard, then I guess the next option would be to
    hire someone who is willing to do one of those things.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Nov 29, 2007
    #6
  7. Jack

    Jack Guest

    On Nov 29, 9:56 am, Jim Gibson <> wrote:
    > In article
    > <>,
    >
    >
    >
    >
    >
    > <> wrote:
    > > On Nov 28, 6:37 pm, Gunnar Hjalmarsson <> wrote:
    > > > Jack wrote:
    > > > > Hi I have a web application that needs to on an adhoc basis query a
    > > > > hash lookup which should be persisted in memory - I have seen this
    > > > > module "Persistent::Memory - A Persistent Class implemented using
    > > > > Memory (RAM)" but I wanted to see if there was a much simpler approach
    > > > > to taking a hash map and making it 'callable' while persisted in
    > > > > memory using standard features not requiring a special Perl module
    > > > > install..

    >
    > > > > Any thoughts would be appreciated.

    >
    > > > Why not a tied hash? See e.g. "perldoc DB_File".

    >
    > > > --
    > > > Gunnar Hjalmarsson
    > > > Email:http://www.gunnar.cc/cgi-bin/contact.pl

    >
    > > Hi and thank you however that looks too complex - just trying to put a
    > > hash in memory, why is that so hard to do ? Is there a simpler way ?

    >
    > Your question is confusing. All Perl variables, including hashes, are
    > in memory. The trick to achieving "persistence" between execution
    > instances of your web application is to have the data on a permanent
    > data store, such as a disk file. Memory variables disappear once your
    > Perl process terminates. Using a tied hash, per Gunnar's suggestion, is
    > one way to accomplish this. There are others, such as using the
    > Storable module to serialize your hash so you can write it to memory.
    >
    > I have not use Persistent::Memory, but you might want to check out the
    > Persistent::File module instead. Both of those modules seem to be part
    > of the Persistent framework of Perl modules.
    >
    > See
    > <http://search.cpan.org/~dwinters/Persistent-Base-0.52/lib/Persistent.po
    > d>
    >
    > --
    > Jim Gibson
    >
    > Posted Via Usenet.com Premium Usenet Newsgroup Services
    > ----------------------------------------------------------
    > ** SPEED ** RETENTION ** COMPLETION ** ANONYMITY **
    > ----------------------------------------------------------
    > http://www.usenet.com- Hide quoted text -
    >
    > - Show quoted text -


    Thanks I am willing to install Perl Modules, really wondering if there
    is a native ability to do it out of the box in Perl. I got persistent
    memory to work but its not clear how to "pin" a hash lookup into
    memory since .pl files and the persistent memory code example itself
    are only as good as the life of the execution of the script !! Still
    not clear how to hold a has in Memory - even if you were to create a
    windows service out of it with activestate PDK, not clear how to write
    the script so the script doesnt stop execution but sits and waits for
    a socket connection or something.. Do I just have script 1 generate
    a hash with "sleep forever" and then script 2 can call that hash ??
    I dont want to use another database for this. So, not using another
    DB, how does one pin and persist and then query a hash in memory using
    perl or a perl module.

    Thanks for the help all !
    Jack
     
    Jack, Nov 29, 2007
    #7
  8. Jack

    Ben Morrow Guest

    Quoth Jack <>:
    <snip: some sort of persistant shared memory>
    >
    > Thanks I am willing to install Perl Modules, really wondering if there
    > is a native ability to do it out of the box in Perl.


    Well, on systems that support it you can use SysV shared memory, for
    which Perl natively has the raw shm* calls. This doesn't work on
    Windows, though.

    You could try IPC::Shareable for Unix and Win32::MMF::Shareable (which
    has the same interface) for Win32, which provide an easier interface to
    shared memory.

    Another alternative would be mmap, using the Sys::MMap module. mmap is
    substantially better behaved than SysV shm, and portable to Windows.

    > I got persistent memory to work but its not clear how to "pin" a hash
    > lookup into memory since .pl files and the persistent memory code
    > example itself are only as good as the life of the execution of the
    > script !!


    In what sense then is this memory 'persistent'? I think you are rather
    confused about what is and isn't possible under modern operating
    systems: unless you take steps to prevent it, all memory allocated by a
    process is released when that process exits. This is a feature :).

    Ben
     
    Ben Morrow, Nov 29, 2007
    #8
  9. Jack

    J. Gleixner Guest

    Jack wrote:
    [...]
    > So, not using another DB


    If you're already using a Database then just store the data there, why
    would you use another Database to store it?

    >how does one pin and persist and then query a hash in memory using
    > perl or a perl module.


    Whatever 'pin' a hash means.

    Why do you want to do this? Maybe that's the real problem. You
    think that's what you want to do, but it's really not needed.

    You could have a daemon running, that reads/creates the hash and
    then it would have the data in memory. You could then then connect
    to it via a socket, but why the heck do you want to do all of
    that? That's really not Perl 101 and I'm pretty sure you'd
    end up spending a lot of time understanding it and trying to get it
    working, however it'll take 20-seconds to write and will be very
    fast, using a tied hash.

    There are a lot of examples online and you can read the documentation
    that comes with perl.

    perldoc DB_File

    If you want to go the whole daemon and IPC/socket route:

    perldoc perlipc
     
    J. Gleixner, Nov 29, 2007
    #9
  10. Jack

    Jack Guest

    On Nov 29, 3:35 pm, Ben Morrow <> wrote:
    > Quoth Jack <>:
    > <snip: some sort of persistant shared memory>
    >
    >
    >
    > > Thanks I am willing to install Perl Modules, really wondering if there
    > > is a native ability to do it out of the box in Perl.

    >
    > Well, on systems that support it you can use SysV shared memory, for
    > which Perl natively has the raw shm* calls. This doesn't work on
    > Windows, though.
    >
    > You could try IPC::Shareable for Unix and Win32::MMF::Shareable (which
    > has the same interface) for Win32, which provide an easier interface to
    > shared memory.
    >
    > Another alternative would be mmap, using the Sys::MMap module. mmap is
    > substantially better behaved than SysV shm, and portable to Windows.
    >
    > > I got persistent memory to work but its not clear how to "pin" a hash
    > > lookup into memory since .pl files and the persistent memory code
    > > example itself are only as good as the life of the execution of the
    > > script !!

    >
    > In what sense then is this memory 'persistent'? I think you are rather
    > confused about what is and isn't possible under modern operating
    > systems: unless you take steps to prevent it, all memory allocated by a
    > process is released when that process exits. This is a feature :).
    >
    > Ben


    ok.. lets talk with code - I got this to work with DB_File.. my
    fundamental question is, once you create the tied hash like the below,
    and the program is done executing, how do I access the hash and query
    it from another Perl execution program ?? I tried using the API
    interface afterwards as described here http://perldoc.perl.org/DB_File.html
    , see the error at bottom:
    ### RUN PROGRAM 1 #####
    use warnings ;
    use strict ;
    use DB_File ;
    our (%h, $k, $v) ; unlink "fruit" ;
    tie %h, "DB_File", "fruit", O_RDWR|O_CREAT, 0666, $DB_HASH
    or die "Cannot open file 'fruit': $!\n"; # Add a few key/
    value pairs to the file
    $h{"apple"} = "red" ;
    $h{"orange"} = "orange" ;
    $h{"banana"} = "yellow" ;
    $h{"tomato"} = "red" ; # Check for existence of a key
    print "Banana Exists\n\n" if $h{"banana"} ; # Delete a key/
    value pair.
    delete $h{"apple"} ; # print the contents of the file
    while (($k, $v) = each %h)
    { print "$k -> $v\n" }
    untie %h ;

    #####START SEPARATE PROGRAM 2 ###
    use warnings ;
    use strict ;
    use DB_File ;
    $db = tie %h, "DB_File", "testfile1" ;
    foreach (keys %h)
    { print "$_\n" }
    ### PROGRAM 2 ERRORS:
    Global symbol "$db" requires explicit package name at
    test_dbfile1_query.pl .
    Global symbol "%h" requires explicit package name at
    test_dbfile1_query.pl .
    Global symbol "%h" requires explicit package name at
    test_dbfile1_query.pl 5.
    Execution of test_dbfile1_query.pl aborted due to compilation errors.
     
    Jack, Nov 30, 2007
    #10
  11. Jack wrote:
    > once you create the tied hash like the below, and the program is done
    > executing, how do I access the hash and query it from another Perl
    > execution program ??


    Like this, for instance:

    use DB_File;
    tie my %h, 'DB_File', 'fruit', O_RDONLY or die $!;
    print "$_: $h{$_}\n" for keys %h;

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
     
    Gunnar Hjalmarsson, Nov 30, 2007
    #11
  12. Jack

    Jack Guest

    On Nov 30, 2:15 am, Gunnar Hjalmarsson <> wrote:
    > Jackwrote:
    > > once you create the tiedhashlike the below, and the program is done
    > > executing, how do I access thehashand query it from another Perl
    > > execution program ??

    >
    > Like this, for instance:
    >
    > use DB_File;
    > tie my %h, 'DB_File', 'fruit', O_RDONLY or die $!;
    > print "$_: $h{$_}\n" for keys %h;
    >
    > --
    > Gunnar Hjalmarsson
    > Email:http://www.gunnar.cc/cgi-bin/contact.pl


    Thanks that worked but cant believe I need berkeley db just to keep a
    hash in memory after a program has executed. In C# and other
    languages its a simple statement. Can someone who knows
    persistent::memory provide a real working example of this, or other
    approach.
    thanks,
    Jack
     
    Jack, Dec 4, 2007
    #12
  13. Jack

    Ben Morrow Guest

    Quoth Jack <>:
    > On Nov 30, 2:15 am, Gunnar Hjalmarsson <> wrote:
    > > Jackwrote:
    > > > once you create the tiedhashlike the below, and the program is done
    > > > executing, how do I access thehashand query it from another Perl
    > > > execution program ??

    > >
    > > Like this, for instance:
    > >
    > > use DB_File;
    > > tie my %h, 'DB_File', 'fruit', O_RDONLY or die $!;
    > > print "$_: $h{$_}\n" for keys %h;

    >
    > Thanks that worked but cant believe I need berkeley db just to keep a
    > hash in memory after a program has executed. In C# and other
    > languages its a simple statement.


    I don't believe you. Can you provide an example, and explain how it
    works? If you want something to persist after your process exits you
    have to put it somewhere, either in the filesystem or somewhere like a
    SysV shm segment that *ought* to be in the filesystem. This is a feature
    of your OS.

    Ben
     
    Ben Morrow, Dec 4, 2007
    #13
  14. Jack

    Guest

    Jack <> wrote:
    > On Nov 30, 2:15 am, Gunnar Hjalmarsson <> wrote:
    > > Jackwrote:
    > > > once you create the tiedhashlike the below, and the program is done
    > > > executing, how do I access thehashand query it from another Perl
    > > > execution program ??

    > >
    > > Like this, for instance:
    > >
    > > use DB_File;
    > > tie my %h, 'DB_File', 'fruit', O_RDONLY or die $!;
    > > print "$_: $h{$_}\n" for keys %h;
    > >
    > > --
    > > Gunnar Hjalmarsson
    > > Email:http://www.gunnar.cc/cgi-bin/contact.pl

    >
    > Thanks that worked but cant believe I need berkeley db just to keep a
    > hash in memory after a program has executed. In C# and other
    > languages its a simple statement.


    Which statement?

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Dec 4, 2007
    #14
  15. Jack

    Dave Weaver Guest

    On Mon, 3 Dec 2007 21:45:45 -0800 (PST), Jack <> wrote:
    >
    > Thanks that worked but cant believe I need berkeley db just to keep a
    > hash in memory after a program has executed.


    Why? Modern operating systems relase memory used by a process once
    the process finishes (which is a Good Thing). Just leaving something
    hanging around in memory isn't something simple.

    > In C# and other languages its a simple statement.


    Is it? Care to share?

    > Can someone who knows persistent::memory provide a real working example of
    > this, or other approach.


    I think you mean Persistent::Memory (case is important in Perl).
    I also think you misunderstand what Peristent::Memory does - from a quick
    glance it appears to be part of the Persistent framework and a base for other
    classes, not some magic that keeps data in memory between process invocations.
     
    Dave Weaver, Dec 4, 2007
    #15
  16. Jack wrote:
    > On Nov 30, 2:15 am, Gunnar Hjalmarsson <> wrote:
    >> Jackwrote:
    >>> once you create the tiedhashlike the below, and the program is done
    >>> executing, how do I access thehashand query it from another Perl
    >>> execution program ??

    >> Like this, for instance:
    >>
    >> use DB_File;
    >> tie my %h, 'DB_File', 'fruit', O_RDONLY or die $!;
    >> print "$_: $h{$_}\n" for keys %h;

    >
    > Thanks that worked but cant believe I need berkeley db just to keep a
    > hash in memory after a program has executed.


    I didn't say you do. As others have mentioned, there are other options.
    For instance, there are other, simpler dbm files. SDBM_File is available
    in Perl on *nix and Windows.

    C:\home>type test.pl
    use SDBM_File;
    use Fcntl;
    tie my %h, 'SDBM_File', 'fruit', O_RDWR|O_CREAT, 0600 or die $!;
    %h = ( banana => 'yellow', apple => 'green', tomato => 'red' );

    C:\home>type test2.pl
    use SDBM_File;
    use Fcntl;
    tie my %h, 'SDBM_File', 'fruit', O_RDONLY, 0400 or die $!;
    print "$_ -> $h{$_}\n" for keys %h;

    C:\home>test.pl

    C:\home>test2.pl
    banana -> yellow
    apple -> green
    tomato -> red

    C:\home>

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
     
    Gunnar Hjalmarsson, Dec 4, 2007
    #16
  17. * Jack <> wrote:
    >
    > Hi I have a web application that needs to on an adhoc basis query a
    > hash lookup which should be persisted in memory - I have seen this
    > module "Persistent::Memory - A Persistent Class implemented using
    > Memory (RAM)" but I wanted to see if there was a much simpler approach
    > to taking a hash map and making it 'callable' while persisted in
    > memory using standard features not requiring a special Perl module
    > install..


    Why don't you tell us a bit more about your particular use case?

    For all we know, what you're really looking for is something
    like memcached┬╣.

    --
    Lars Haugseth

    "If anyone disagrees with anything I say, I am quite prepared not only to
    retract it, but also to deny under oath that I ever said it." -Tom Lehrer

    ┬╣ http://www.danga.com/memcached/
     
    Lars Haugseth, Dec 4, 2007
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Pieter Claassen

    hash table lookup consistenly fails

    Pieter Claassen, Aug 16, 2004, in forum: C Programming
    Replies:
    2
    Views:
    288
    CBFalconer
    Aug 16, 2004
  2. rp
    Replies:
    1
    Views:
    536
    red floyd
    Nov 10, 2011
  3. Srijayanth Sridhar
    Replies:
    19
    Views:
    625
    David A. Black
    Jul 2, 2008
  4. Paul

    Hash Lookup Problem

    Paul, Oct 25, 2008, in forum: Ruby
    Replies:
    1
    Views:
    131
    Trans
    Oct 26, 2008
  5. Replies:
    4
    Views:
    277
Loading...

Share This Page