Accessing an Array Stored in a File

Discussion in 'Perl Misc' started by hamidrezah@yahoo.com, Sep 18, 2008.

  1. Guest

    Hi all:

    I have a BIG array. I store it in a file using Storable::store.

    I want to use it in another Perl program. The BIG array cannot be
    loaded into memory since it is just too BIG.

    Any trick to use its copy in the file as an object and access it? It
    might be very slow but I don't care about speed. I just want to get
    over this memory problem.

    In otherwords, can Perl access an object stored in a file as if it
    were in memory?

    NOTE: I know I can load it in memory. I don't want to. I want to
    access it while it sits in the file.

    Thanks for any hint.

    Regards,

    Hrh
    , Sep 18, 2008
    #1
    1. Advertising

  2. Guest

    wrote:
    > Hi all:
    >
    > I have a BIG array. I store it in a file using Storable::store.


    Is Storable::store negotiable?

    > I want to use it in another Perl program. The BIG array cannot be
    > loaded into memory since it is just too BIG.


    But then how did it get stored in Storable in the first place?

    >
    > Any trick to use its copy in the file as an object and access it? It
    > might be very slow but I don't care about speed. I just want to get
    > over this memory problem.


    There are many options for tying an array to disk. Many of them are not
    very good, at least not for the general case. My favorite would be to
    store the array to disk using DBM::Deep instead of Storable.

    > In otherwords, can Perl access an object stored in a file as if it
    > were in memory?


    Yes, there are many ways. I think DBM::Deep is a very good one. It has
    some gotchas, but less so than other modules I've tried to use to do the
    same thing.

    If you use DBM::Deep, I'd stick with version 0.983 of it, as after that
    point the code takes a turn into trying to be a transactional database,
    rather than a nifty way to tie Perl arrays and hashes to disk.


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
    , Sep 18, 2008
    #2
    1. Advertising

  3. wrote:
    >I have a BIG array. I store it in a file using Storable::store.
    >
    >I want to use it in another Perl program. The BIG array cannot be
    >loaded into memory since it is just too BIG.
    >In otherwords, can Perl access an object stored in a file as if it
    >were in memory?
    >
    >NOTE: I know I can load it in memory. I don't want to. I want to
    >access it while it sits in the file.


    Honestly I don't know. However one simple workaroud might be use virtual
    memory by enlarging your swap space.

    A real solution would be to use a database. Or maybe (depending on your
    needs) to upgrade to a 64-bit OS such that you can address more than 4GB
    of RAM.

    jue
    Jürgen Exner, Sep 18, 2008
    #3
  4. C.DeRykus Guest

    On Sep 18, 6:50 am, wrote:
    > Hi all:
    >
    > I have a BIG array. I store it in a file using Storable::store.
    >
    > I want to use it in another Perl program. The BIG array cannot be
    > loaded into memory since it is just too BIG.
    >
    > Any trick to use its copy in the file as an object and access it? It
    > might be very slow but I don't care about speed. I just want to get
    > over this memory problem.
    >
    > In otherwords, can Perl access an object stored in a file as if it
    > were in memory?
    >
    > NOTE: I know I can load it in memory. I don't want to. I want to
    > access it while it sits in the file.
    >


    DB_File or MLDBM (if multi-level)
    are other possible options.

    --
    Charles DeRykus
    C.DeRykus, Sep 18, 2008
    #4
  5. cartercc Guest

    On Sep 18, 9:50 am, wrote:
    > I want to use it in another Perl program. The BIG array cannot be
    > loaded into memory since it is just too BIG.
    >
    > Any trick to use its copy in the file as an object and access it? It
    > might be very slow but I don't care about speed. I just want to get
    > over this memory problem.
    >
    > In otherwords, can Perl access an object stored in a file as if it
    > were in memory?
    >
    > NOTE: I know I can load it in memory. I don't want to. I want to
    > access it while it sits in the file.


    In an array, individual elements are accessed sequentially according
    to the subscript. Do the same thing with your file.

    my $subscript = 0;
    open BIGFILE, "<bigfile.dat";
    while(<BIGFILE>)
    {
    if ($subscript eq $target) { process_element($_); }
    elsif (/$match/) { process_element($_); }
    $subscript++;
    }
    close BIGFILE;
    }

    CC
    cartercc, Sep 18, 2008
    #5
  6. John Bokma Guest

    John Bokma, Sep 19, 2008
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    2
    Views:
    622
    Thomas Matthews
    Feb 27, 2007
  2. loudking
    Replies:
    9
    Views:
    450
    Keith Thompson
    May 1, 2007
  3. Replies:
    0
    Views:
    299
  4. Peter Mueller
    Replies:
    4
    Views:
    1,111
  5. Mriganka
    Replies:
    1
    Views:
    114
    kaeli
    May 21, 2004
Loading...

Share This Page