How to persist and query a hash lookup in memory with Perl ?

J

Jack

Hi I have a web application that needs to on an adhoc basis query a
hash lookup which should be persisted in memory - I have seen this
module "Persistent::Memory - A Persistent Class implemented using
Memory (RAM)" but I wanted to see if there was a much simpler approach
to taking a hash map and making it 'callable' while persisted in
memory using standard features not requiring a special Perl module
install..

Any thoughts would be appreciated.
Thanks,
Jack
 
G

Gunnar Hjalmarsson

Jack said:
Hi I have a web application that needs to on an adhoc basis query a
hash lookup which should be persisted in memory - I have seen this
module "Persistent::Memory - A Persistent Class implemented using
Memory (RAM)" but I wanted to see if there was a much simpler approach
to taking a hash map and making it 'callable' while persisted in
memory using standard features not requiring a special Perl module
install..

Any thoughts would be appreciated.

Why not a tied hash? See e.g. "perldoc DB_File".
 
J

jackposemsky

Why not a tied hash? See e.g. "perldoc DB_File".

Hi and thank you however that looks too complex - just trying to put a
hash in memory, why is that so hard to do ? Is there a simpler way ?
 
A

A. Sinan Unur

(e-mail address removed) wrote in

Don't quote sigs.
Hi and thank you however that looks too complex - just trying to put a
hash in memory, why is that so hard to do ? Is there a simpler way ?

It looks like programming might be too hard for you.

Sinan
 
T

Ted Zlatanov

J> Hi I have a web application that needs to on an adhoc basis query a
J> hash lookup which should be persisted in memory - I have seen this
J> module "Persistent::Memory - A Persistent Class implemented using
J> Memory (RAM)" but I wanted to see if there was a much simpler approach
J> to taking a hash map and making it 'callable' while persisted in
J> memory using standard features not requiring a special Perl module
J> install..

Are you talking about sharing the data between several instances of your
web application, so whichever one handles a request can read or update
data?

Use a database. There are cases where this is not the right approach,
but it doesn't sound like it in yours.

Ted
 
X

xhoster

Hi and thank you however that looks too complex - just trying to put a
hash in memory, why is that so hard to do ? Is there a simpler way ?

If you aren't willing to install modules and you aren't willing to use the
ones that are (often) standard, then I guess the next option would be to
hire someone who is willing to do one of those things.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
J

Jack

Your question is confusing. All Perl variables, including hashes, are
in memory. The trick to achieving "persistence" between execution
instances of your web application is to have the data on a permanent
data store, such as a disk file. Memory variables disappear once your
Perl process terminates. Using a tied hash, per Gunnar's suggestion, is
one way to accomplish this. There are others, such as using the
Storable module to serialize your hash so you can write it to memory.

I have not use Persistent::Memory, but you might want to check out the
Persistent::File module instead. Both of those modules seem to be part
of the Persistent framework of Perl modules.

See
<http://search.cpan.org/~dwinters/Persistent-Base-0.52/lib/Persistent.po
d>

Thanks I am willing to install Perl Modules, really wondering if there
is a native ability to do it out of the box in Perl. I got persistent
memory to work but its not clear how to "pin" a hash lookup into
memory since .pl files and the persistent memory code example itself
are only as good as the life of the execution of the script !! Still
not clear how to hold a has in Memory - even if you were to create a
windows service out of it with activestate PDK, not clear how to write
the script so the script doesnt stop execution but sits and waits for
a socket connection or something.. Do I just have script 1 generate
a hash with "sleep forever" and then script 2 can call that hash ??
I dont want to use another database for this. So, not using another
DB, how does one pin and persist and then query a hash in memory using
perl or a perl module.

Thanks for the help all !
Jack
 
B

Ben Morrow

Quoth Jack said:
Thanks I am willing to install Perl Modules, really wondering if there
is a native ability to do it out of the box in Perl.

Well, on systems that support it you can use SysV shared memory, for
which Perl natively has the raw shm* calls. This doesn't work on
Windows, though.

You could try IPC::Shareable for Unix and Win32::MMF::Shareable (which
has the same interface) for Win32, which provide an easier interface to
shared memory.

Another alternative would be mmap, using the Sys::MMap module. mmap is
substantially better behaved than SysV shm, and portable to Windows.
I got persistent memory to work but its not clear how to "pin" a hash
lookup into memory since .pl files and the persistent memory code
example itself are only as good as the life of the execution of the
script !!

In what sense then is this memory 'persistent'? I think you are rather
confused about what is and isn't possible under modern operating
systems: unless you take steps to prevent it, all memory allocated by a
process is released when that process exits. This is a feature :).

Ben
 
J

J. Gleixner

Jack wrote:
[...]
So, not using another DB

If you're already using a Database then just store the data there, why
would you use another Database to store it?
how does one pin and persist and then query a hash in memory using
perl or a perl module.

Whatever 'pin' a hash means.

Why do you want to do this? Maybe that's the real problem. You
think that's what you want to do, but it's really not needed.

You could have a daemon running, that reads/creates the hash and
then it would have the data in memory. You could then then connect
to it via a socket, but why the heck do you want to do all of
that? That's really not Perl 101 and I'm pretty sure you'd
end up spending a lot of time understanding it and trying to get it
working, however it'll take 20-seconds to write and will be very
fast, using a tied hash.

There are a lot of examples online and you can read the documentation
that comes with perl.

perldoc DB_File

If you want to go the whole daemon and IPC/socket route:

perldoc perlipc
 
J

Jack

Quoth Jack <[email protected]>:
<snip: some sort of persistant shared memory>




Well, on systems that support it you can use SysV shared memory, for
which Perl natively has the raw shm* calls. This doesn't work on
Windows, though.

You could try IPC::Shareable for Unix and Win32::MMF::Shareable (which
has the same interface) for Win32, which provide an easier interface to
shared memory.

Another alternative would be mmap, using the Sys::MMap module. mmap is
substantially better behaved than SysV shm, and portable to Windows.


In what sense then is this memory 'persistent'? I think you are rather
confused about what is and isn't possible under modern operating
systems: unless you take steps to prevent it, all memory allocated by a
process is released when that process exits. This is a feature :).

Ben

ok.. lets talk with code - I got this to work with DB_File.. my
fundamental question is, once you create the tied hash like the below,
and the program is done executing, how do I access the hash and query
it from another Perl execution program ?? I tried using the API
interface afterwards as described here http://perldoc.perl.org/DB_File.html
, see the error at bottom:
### RUN PROGRAM 1 #####
use warnings ;
use strict ;
use DB_File ;
our (%h, $k, $v) ; unlink "fruit" ;
tie %h, "DB_File", "fruit", O_RDWR|O_CREAT, 0666, $DB_HASH
or die "Cannot open file 'fruit': $!\n"; # Add a few key/
value pairs to the file
$h{"apple"} = "red" ;
$h{"orange"} = "orange" ;
$h{"banana"} = "yellow" ;
$h{"tomato"} = "red" ; # Check for existence of a key
print "Banana Exists\n\n" if $h{"banana"} ; # Delete a key/
value pair.
delete $h{"apple"} ; # print the contents of the file
while (($k, $v) = each %h)
{ print "$k -> $v\n" }
untie %h ;

#####START SEPARATE PROGRAM 2 ###
use warnings ;
use strict ;
use DB_File ;
$db = tie %h, "DB_File", "testfile1" ;
foreach (keys %h)
{ print "$_\n" }
### PROGRAM 2 ERRORS:
Global symbol "$db" requires explicit package name at
test_dbfile1_query.pl .
Global symbol "%h" requires explicit package name at
test_dbfile1_query.pl .
Global symbol "%h" requires explicit package name at
test_dbfile1_query.pl 5.
Execution of test_dbfile1_query.pl aborted due to compilation errors.
 
G

Gunnar Hjalmarsson

Jack said:
once you create the tied hash like the below, and the program is done
executing, how do I access the hash and query it from another Perl
execution program ??

Like this, for instance:

use DB_File;
tie my %h, 'DB_File', 'fruit', O_RDONLY or die $!;
print "$_: $h{$_}\n" for keys %h;
 
J

Jack

Jackwrote:

Like this, for instance:

use DB_File;
tie my %h, 'DB_File', 'fruit', O_RDONLY or die $!;
print "$_: $h{$_}\n" for keys %h;

Thanks that worked but cant believe I need berkeley db just to keep a
hash in memory after a program has executed. In C# and other
languages its a simple statement. Can someone who knows
persistent::memory provide a real working example of this, or other
approach.
thanks,
Jack
 
B

Ben Morrow

Quoth Jack said:
Thanks that worked but cant believe I need berkeley db just to keep a
hash in memory after a program has executed. In C# and other
languages its a simple statement.

I don't believe you. Can you provide an example, and explain how it
works? If you want something to persist after your process exits you
have to put it somewhere, either in the filesystem or somewhere like a
SysV shm segment that *ought* to be in the filesystem. This is a feature
of your OS.

Ben
 
X

xhoster

Jack said:
Thanks that worked but cant believe I need berkeley db just to keep a
hash in memory after a program has executed. In C# and other
languages its a simple statement.

Which statement?

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
D

Dave Weaver

Thanks that worked but cant believe I need berkeley db just to keep a
hash in memory after a program has executed.

Why? Modern operating systems relase memory used by a process once
the process finishes (which is a Good Thing). Just leaving something
hanging around in memory isn't something simple.
In C# and other languages its a simple statement.

Is it? Care to share?
Can someone who knows persistent::memory provide a real working example of
this, or other approach.

I think you mean Persistent::Memory (case is important in Perl).
I also think you misunderstand what Peristent::Memory does - from a quick
glance it appears to be part of the Persistent framework and a base for other
classes, not some magic that keeps data in memory between process invocations.
 
G

Gunnar Hjalmarsson

Jack said:
Thanks that worked but cant believe I need berkeley db just to keep a
hash in memory after a program has executed.

I didn't say you do. As others have mentioned, there are other options.
For instance, there are other, simpler dbm files. SDBM_File is available
in Perl on *nix and Windows.

C:\home>type test.pl
use SDBM_File;
use Fcntl;
tie my %h, 'SDBM_File', 'fruit', O_RDWR|O_CREAT, 0600 or die $!;
%h = ( banana => 'yellow', apple => 'green', tomato => 'red' );

C:\home>type test2.pl
use SDBM_File;
use Fcntl;
tie my %h, 'SDBM_File', 'fruit', O_RDONLY, 0400 or die $!;
print "$_ -> $h{$_}\n" for keys %h;

C:\home>test.pl

C:\home>test2.pl
banana -> yellow
apple -> green
tomato -> red

C:\home>
 
L

Lars Haugseth

* Jack said:
Hi I have a web application that needs to on an adhoc basis query a
hash lookup which should be persisted in memory - I have seen this
module "Persistent::Memory - A Persistent Class implemented using
Memory (RAM)" but I wanted to see if there was a much simpler approach
to taking a hash map and making it 'callable' while persisted in
memory using standard features not requiring a special Perl module
install..

Why don't you tell us a bit more about your particular use case?

For all we know, what you're really looking for is something
like memcached¹.

--
Lars Haugseth

"If anyone disagrees with anything I say, I am quite prepared not only to
retract it, but also to deny under oath that I ever said it." -Tom Lehrer

¹ http://www.danga.com/memcached/
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,769
Messages
2,569,579
Members
45,053
Latest member
BrodieSola

Latest Threads

Top