Small memory problem

Discussion in 'Perl Misc' started by David Morel, Aug 23, 2003.

  1. David Morel

    David Morel Guest

    Hi all,

    I am having a small problem with memory management.
    Here is my code:

    my $num = 500000;
    my @paths;
    # memory usage is now low
    sleep 5;

    for (my $i = 0; $i <= $num; $i++) {
    my $path = [$i, $i];
    push @paths, $path;
    }
    # memory usage is now high
    sleep 5;

    undef @paths;
    # now I want to return to low memory usage,
    # but the above undef statement doesn't get the job done!
    sleep 5;

    Notice that in the second block of code I create an array with 500000
    pointers. Then, in the third block of code, I delete all of the
    pointers with undef (I think). Then why doesn't the garbage collector
    kick in? I mean the total memory usage after the second block is about
    75MB, and then it only drops to about 73MB... (as opposed to about
    1MB) ?

    Thanks!
     
    David Morel, Aug 23, 2003
    #1
    1. Advertising

  2. David Morel <> wrote:
    DM> Hi all,

    DM> I am having a small problem with memory management.
    DM> Here is my code:

    DM> my $num = 500000;
    DM> my @paths;
    DM> # memory usage is now low
    DM> sleep 5;

    DM> for (my $i = 0; $i <= $num; $i++) {
    DM> my $path = [$i, $i];
    DM> push @paths, $path;
    DM> }
    DM> # memory usage is now high
    DM> sleep 5;

    DM> undef @paths;
    DM> # now I want to return to low memory usage,
    DM> # but the above undef statement doesn't get the job done!
    DM> sleep 5;

    DM> Notice that in the second block of code I create an array with 500000
    DM> pointers. Then, in the third block of code, I delete all of the
    DM> pointers with undef (I think). Then why doesn't the garbage collector
    DM> kick in? I mean the total memory usage after the second block is about
    DM> 75MB, and then it only drops to about 73MB... (as opposed to about
    DM> 1MB) ?

    Did you check 'perldoc -q free'?

    Found in /usr/share/perl/5.8.0/pod/perlfaq3.pod
    How can I free an array or hash so my program shrinks?

    You usually can't. On most operating systems, memory allocated to a
    program can never be returned to the system. That's why long-running
    programs sometimes re-exec themselves. Some operating systems (notably,
    systems that use mmap(2) for allocating large chunks of memory) can
    reclaim memory that is no longer used, but on such systems, perl must
    be configured and compiled to use the OS's malloc, not perl's.

    . . . .

    The pages of memory perl allocated *are* freed within the context
    of the process. What you're seeing, however, is that the memory
    allocater hasn't shrunk the heap -- this is normal behavior. Pages
    allocated from the heap are "returned" to the operating system's
    free pool of memory when the process exits.

    Regards,

    Nicholas

    --
    "Why shouldn't I top-post?" http://www.aglami.com/tpfaq.html
    "Meanings are another story." http://www.ifas.org/wa/glossolalia.html
     
    Nicholas Dronen, Aug 23, 2003
    #2
    1. Advertising

  3. Tore Aursand wrote:
    >
    > On Sat, 23 Aug 2003 13:26:17 -0700, David Morel wrote:
    > > # now I want to return to low memory usage,
    > > # but the above undef statement doesn't get the job done!

    >
    > That's right, and that's the way it is; When a program exits, it doesn't
    > necessarily free up its memory and return it to the OS. Check the perldoc
    > for information about this.
    >
    > A comment on your code, however. Seems a bit C'ish to me, as you could
    > have written the same thing like this;
    >
    > my @paths = ();
    > sleep 5;
    > for ( 1 .. 500_000 ) {
    > push( @paths, [$_, $_] );
    > }
    > sleep 5;


    Surely you mean:

    my @paths;
    sleep 5;
    @paths = map [$_, $_], 1 .. 500_000;
    sleep 5;

    :)


    --
    $a=24;split//,240513;s/\B/ => /for@@=qw(ac ab bc ba cb ca
    );{push(@b,$a),($a-=6)^=1 for 2..$a/6x--$|;print "$@[$a%6
    ]\n";((6<=($a-=6))?$a+=$_[$a%6]-$a%6:($a=pop @b))&&redo;}
     
    Benjamin Goldberg, Aug 25, 2003
    #3
  4. [A complimentary Cc of this posting was sent to
    Benjamin Goldberg
    <>], who wrote in article <>:
    > > my @paths = ();
    > > sleep 5;
    > > for ( 1 .. 500_000 ) {
    > > push( @paths, [$_, $_] );
    > > }
    > > sleep 5;

    >
    > Surely you mean:
    >
    > my @paths;
    > sleep 5;
    > @paths = map [$_, $_], 1 .. 500_000;
    > sleep 5;


    Are you sure that 1 .. 500_000 is lazy in this context too?

    Thanks,
    Ilya
     
    Ilya Zakharevich, Aug 26, 2003
    #4
  5. David Morel

    Anno Siegel Guest

    Ilya Zakharevich <> wrote in comp.lang.perl.misc:
    > [A complimentary Cc of this posting was sent to
    > Benjamin Goldberg
    > <>], who wrote in article <>:
    > > > my @paths = ();
    > > > sleep 5;
    > > > for ( 1 .. 500_000 ) {
    > > > push( @paths, [$_, $_] );
    > > > }
    > > > sleep 5;

    > >
    > > Surely you mean:
    > >
    > > my @paths;
    > > sleep 5;
    > > @paths = map [$_, $_], 1 .. 500_000;
    > > sleep 5;

    >
    > Are you sure that 1 .. 500_000 is lazy in this context too?


    Does it matter? I'd expect the list of arrayrefs to occupy about
    ten times of what the list of integers takes, and rough measurements
    bear that out. That's not too much of an impact.

    Anno
     
    Anno Siegel, Aug 26, 2003
    #5
  6. David Morel

    Anno Siegel Guest

    Ilya Zakharevich <> wrote in comp.lang.perl.misc:
    > [A complimentary Cc of this posting was sent to
    > Anno Siegel
    > <-berlin.de>], who wrote in article
    > <bifm3c$rc7$-Berlin.DE>:
    > > > > @paths = map [$_, $_], 1 .. 500_000;

    >
    > > > Are you sure that 1 .. 500_000 is lazy in this context too?

    >
    > > Does it matter? I'd expect the list of arrayrefs to occupy about
    > > ten times of what the list of integers takes, and rough measurements
    > > bear that out. That's not too much of an impact.

    >
    > I think it matters. 1..500K is a *constant* array. It is created at
    > *compile* time, and is never free()ed. @paths may be bigger, but it
    > has a chance to come and go (e.g., when the subroutine exits).


    I still wouldn't worry about it much if the overall memory consumption
    is much larger.

    > However, if you use this construct in 50 subroutines, the *minimal*
    > footprint of your program is 50 large arrays - even before the program
    > starts.


    This would indeed be an unfortunate state of affairs -- one I didn't
    think of.

    use constant A_MILLION_TIMES => 0 .. 999_999;

    map { ... } A_MILLION_TIMES;


    ....would actually save space in this case. But that's no longer serious.

    Anno
     
    Anno Siegel, Aug 27, 2003
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jimmy Zhang
    Replies:
    1
    Views:
    317
    Enrique
    May 19, 2004
  2. gbrun
    Replies:
    1
    Views:
    464
    Andrey Kuznetsov
    Feb 19, 2006
  3. Jimmy Zhang
    Replies:
    1
    Views:
    395
    Enrique
    May 19, 2004
  4. Replies:
    2
    Views:
    8,792
  5. Ola Natvig
    Replies:
    1
    Views:
    297
    Brian Quinlan
    Mar 8, 2005
Loading...

Share This Page