Accessing an Array Stored in a File

[hamidrezah]

Hi all:

I have a BIG array. I store it in a file using Storable::store.

I want to use it in another Perl program. The BIG array cannot be
loaded into memory since it is just too BIG.

Any trick to use its copy in the file as an object and access it? It
might be very slow but I don't care about speed. I just want to get
over this memory problem.

In otherwords, can Perl access an object stored in a file as if it
were in memory?

NOTE: I know I can load it in memory. I don't want to. I want to
access it while it sits in the file.

Thanks for any hint.

Regards,

Hrh

 

[xhoster]

Hi all:

I have a BIG array. I store it in a file using Storable::store.

Is Storable::store negotiable?

I want to use it in another Perl program. The BIG array cannot be
loaded into memory since it is just too BIG.

But then how did it get stored in Storable in the first place?

Any trick to use its copy in the file as an object and access it? It
might be very slow but I don't care about speed. I just want to get
over this memory problem.

There are many options for tying an array to disk. Many of them are not
very good, at least not for the general case. My favorite would be to
store the array to disk using DBM:eep instead of Storable.

In otherwords, can Perl access an object stored in a file as if it
were in memory?

Yes, there are many ways. I think DBM:eep is a very good one. It has
some gotchas, but less so than other modules I've tried to use to do the
same thing.

If you use DBM:eep, I'd stick with version 0.983 of it, as after that
point the code takes a turn into trying to be a transactional database,
rather than a nifty way to tie Perl arrays and hashes to disk.


Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

 

[Jürgen Exner]

I have a BIG array. I store it in a file using Storable::store.

I want to use it in another Perl program. The BIG array cannot be
loaded into memory since it is just too BIG.
In otherwords, can Perl access an object stored in a file as if it
were in memory?

NOTE: I know I can load it in memory. I don't want to. I want to
access it while it sits in the file.

Honestly I don't know. However one simple workaroud might be use virtual
memory by enlarging your swap space.

A real solution would be to use a database. Or maybe (depending on your
needs) to upgrade to a 64-bit OS such that you can address more than 4GB
of RAM.

jue

 

[C.DeRykus]

Hi all:

I have a BIG array. I store it in a file using Storable::store.

I want to use it in another Perl program. The BIG array cannot be
loaded into memory since it is just too BIG.

Any trick to use its copy in the file as an object and access it? It
might be very slow but I don't care about speed. I just want to get
over this memory problem.

In otherwords, can Perl access an object stored in a file as if it
were in memory?

NOTE: I know I can load it in memory. I don't want to. I want to
access it while it sits in the file.

DB_File or MLDBM (if multi-level)
are other possible options.

 

[cartercc]

I want to use it in another Perl program. The BIG array cannot be
loaded into memory since it is just too BIG.

Any trick to use its copy in the file as an object and access it? It
might be very slow but I don't care about speed. I just want to get
over this memory problem.

In otherwords, can Perl access an object stored in a file as if it
were in memory?

NOTE: I know I can load it in memory. I don't want to. I want to
access it while it sits in the file.

In an array, individual elements are accessed sequentially according
to the subscript. Do the same thing with your file.

my $subscript = 0;
open BIGFILE, "<bigfile.dat";
while(<BIGFILE>)
{
if ($subscript eq $target) { process_element($_); }
elsif (/$match/) { process_element($_); }
$subscript++;
}
close BIGFILE;
}

CC

 

[John Bokma]

open BIGFILE, "<bigfile.dat";

my $filename = 'bigfile.dat';
open my $fh, '<', $filename
or die "Can't open '$filename' for reading: $!";


See perldoc -f open