How do you reduce memory requirements of a large hash

Discussion in 'Perl Misc' started by timnels@gmail.com, Aug 15, 2005.

  1. Guest

    I've tried using DBM::Deep to store a large hash on disk since I am
    getting "Out of memory" errors. It works, but runs too slow, so I was
    considering using ?DBM_File to attempt the same thing, something like:

    $inputs = {};
    tie(%{$inputs}, 'NDBM_File', 'foo', O_RDWR|O_CREAT, 0666)
    or die "Couldn't tie SDBM file 'foo': $!; aborting";

    and then checking "ps" as it's adding 200,000 records, it seems to be
    chewing up memory as I add those keys whereas DBM::Deep doesn't. Am I
    missing something obvious here?

    Thanks.
     
    , Aug 15, 2005
    #1
    1. Advertising

  2. Guest

    wrote:
    > I've tried using DBM::Deep to store a large hash on disk since I am
    > getting "Out of memory" errors. It works, but runs too slow, so I was
    > considering using ?DBM_File to attempt the same thing, something like:


    If you store the data to disk rather than to memory, it is going to be
    slow. If it is stored to memory as well as disk, then it will not cure your
    "out of memory" errors. There is no general solution to this problem,
    other than to buy more memory. There may be specific solutions, but they
    would depend on the specifics of what you are doing with this large hash.

    >
    > $inputs = {};
    > tie(%{$inputs}, 'NDBM_File', 'foo', O_RDWR|O_CREAT, 0666)
    > or die "Couldn't tie SDBM file 'foo': $!; aborting";
    >
    > and then checking "ps" as it's adding 200,000 records, it seems to be
    > chewing up memory as I add those keys whereas DBM::Deep doesn't. Am I
    > missing something obvious here?


    The docs for NDBM_File suggests that its purpose is data persistency, not
    memory efficiency.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Aug 15, 2005
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?UmljYXJkbyBRLkcu?=

    why Session.Abandon() does not reduce memory use?

    =?Utf-8?B?UmljYXJkbyBRLkcu?=, Jul 29, 2005, in forum: ASP .Net
    Replies:
    3
    Views:
    599
    jasonkester
    Jul 29, 2005
  2. Replies:
    17
    Views:
    3,513
    Andrey Kuznetsov
    Oct 25, 2005
  3. nick
    Replies:
    58
    Views:
    1,928
    Bart van Ingen Schenau
    Mar 16, 2009
  4. rp
    Replies:
    1
    Views:
    533
    red floyd
    Nov 10, 2011
  5. Srijayanth Sridhar
    Replies:
    19
    Views:
    624
    David A. Black
    Jul 2, 2008
Loading...

Share This Page