Memory leak in loop when not using "my" - why?

Discussion in 'Perl Misc' started by Sinisa Susnjar, Jul 19, 2005.

  1. Hello Group,

    could a Perl Guru please explain to me what is going on here (so I can
    become one too ;-) )?!?

    Progy1.pl:
    #!/usr/bin/perl
    while ($line = <>) {
    chomp($line);
    ($key, $value) = split(':', $line);
    $myhash{$key} = $value;
    }

    Progy1.pl shows a small piece of code from a larger program that is
    supposed to do mass-data manipulation (>50 mio lines of data).
    I was asked to check for memory leaks... it was leaking tons (>4GB) of
    memory and I discovered that with rewriting it like Progy2.pl below, the
    memory leaks would go away:

    Progy2.pl:
    #!/usr/bin/perl
    use strict;
    use warnings;
    my %myhash;
    while (my $line = <>) {
    chomp($line);
    my ($key, $value) = split(':', $line);
    $myhash{$key} = $value;
    }

    Yeah, sure - I could lean back and say problem solved, but I would
    really like to understand why Progy1 is leaking and Progy2 not...

    It seems to me that perl is allocating memory in Progy1.pl for the
    variables within the while {} for every single pass through the loop and
    never giving them back or reuse or garbage collect them (is the GC too
    slow?), while in Progy2.pl - with the variables declared with "my" -
    everything works as expected, i.e. no memory leaks, overall memory
    consumption aroung 70MB which is ok for the type of files being processed.

    btw, I am using perl 5.6.1 on Solaris 2.8...

    I read the FAQs (I hope thorougly enough...)

    Hope, somebody can help...

    Regards,
    Sinisa Susnjar
    Sinisa Susnjar, Jul 19, 2005
    #1
    1. Advertising

  2. Sinisa Susnjar

    Guest

    Sinisa Susnjar wrote:
    > Hello Group,
    >
    > could a Perl Guru please explain to me what is going on here (so I can
    > become one too ;-) )?!?
    >
    > Progy1.pl:
    > #!/usr/bin/perl
    > while ($line = <>) {
    > chomp($line);
    > ($key, $value) = split(':', $line);
    > $myhash{$key} = $value;
    > }


    maybe your just filling %myhash endlessly without deleting elements you
    no longer need.

    >
    > Progy1.pl shows a small piece of code from a larger program that is
    > supposed to do mass-data manipulation (>50 mio lines of data).
    > I was asked to check for memory leaks... it was leaking tons (>4GB) of
    > memory and I discovered that with rewriting it like Progy2.pl below, the
    > memory leaks would go away:
    >
    > Progy2.pl:
    > #!/usr/bin/perl
    > use strict;
    > use warnings;
    > my %myhash;


    see. Your starting fresh every time with an empty hash.

    > while (my $line = <>) {
    > chomp($line);
    > my ($key, $value) = split(':', $line);
    > $myhash{$key} = $value;
    > }
    >
    > Yeah, sure - I could lean back and say problem solved, but I would
    > really like to understand why Progy1 is leaking and Progy2 not...
    >
    > It seems to me that perl is allocating memory in Progy1.pl for the
    > variables within the while {} for every single pass through the loop and
    > never giving them back or reuse or garbage collect them (is the GC too
    > slow?), while in Progy2.pl - with the variables declared with "my" -
    > everything works as expected, i.e. no memory leaks, overall memory
    > consumption aroung 70MB which is ok for the type of files being processed.
    >
    > btw, I am using perl 5.6.1 on Solaris 2.8...
    >
    > I read the FAQs (I hope thorougly enough...)
    >
    > Hope, somebody can help...
    >
    > Regards,
    > Sinisa Susnjar
    , Jul 19, 2005
    #2
    1. Advertising

  3. Sinisa Susnjar

    Guest

    Sinisa Susnjar <> wrote:
    > Hello Group,
    >
    > could a Perl Guru please explain to me what is going on here (so I can
    > become one too ;-) )?!?
    >
    > Progy1.pl:
    > #!/usr/bin/perl
    > while ($line = <>) {
    > chomp($line);
    > ($key, $value) = split(':', $line);
    > $myhash{$key} = $value;
    > }
    >
    > Progy1.pl shows a small piece of code from a larger program that is
    > supposed to do mass-data manipulation (>50 mio lines of data).
    > I was asked to check for memory leaks... it was leaking tons (>4GB) of
    > memory


    What was leaking memory? The above code, or the larger program you
    abstracted it from?

    > and I discovered that with rewriting it like Progy2.pl below, the
    > memory leaks would go away:
    >
    > Progy2.pl:
    > #!/usr/bin/perl
    > use strict;
    > use warnings;
    > my %myhash;
    > while (my $line = <>) {
    > chomp($line);
    > my ($key, $value) = split(':', $line);
    > $myhash{$key} = $value;
    > }


    I find it very hard to believe that, of the code you showed, one leaks
    and the other doesn't.

    >
    > Yeah, sure - I could lean back and say problem solved, but I would
    > really like to understand why Progy1 is leaking and Progy2 not...
    >
    > It seems to me that perl is allocating memory in Progy1.pl for the
    > variables within the while {} for every single pass through the loop and
    > never giving them back or reuse or garbage collect them (is the GC too
    > slow?), while in Progy2.pl - with the variables declared with "my" -
    > everything works as expected, i.e. no memory leaks, overall memory
    > consumption aroung 70MB which is ok for the type of files being
    > processed.


    How could you fit a hash with >50 million key-value pairs into 70MB of
    memory?

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Jul 19, 2005
    #3
  4. wrote:
    > Sinisa Susnjar <> wrote:
    >
    >>Hello Group,
    >>
    >>could a Perl Guru please explain to me what is going on here (so I can
    >>become one too ;-) )?!?
    >>
    >>Progy1.pl:
    >>#!/usr/bin/perl
    >>while ($line = <>) {
    >> chomp($line);
    >> ($key, $value) = split(':', $line);
    >> $myhash{$key} = $value;
    >>}
    >>
    >>Progy1.pl shows a small piece of code from a larger program that is
    >>supposed to do mass-data manipulation (>50 mio lines of data).
    >>I was asked to check for memory leaks... it was leaking tons (>4GB) of
    >>memory

    >
    >
    > What was leaking memory? The above code, or the larger program you
    > abstracted it from?
    >

    The above code. The larger program was filled with multiple
    code-snippets like the above.
    >
    >>and I discovered that with rewriting it like Progy2.pl below, the
    >>memory leaks would go away:
    >>
    >>Progy2.pl:
    >>#!/usr/bin/perl
    >>use strict;
    >>use warnings;
    >>my %myhash;
    >>while (my $line = <>) {
    >> chomp($line);
    >> my ($key, $value) = split(':', $line);
    >> $myhash{$key} = $value;
    >>}

    >
    >
    > I find it very hard to believe that, of the code you showed, one leaks
    > and the other doesn't.


    I could not believe it myself! I tried to reproduce it @home under x86
    Linux with perl 5.8.3 and I couldn't! I will try downloading the newest
    perl version today at my jobs place and see if that makes a difference...

    >
    >
    >>Yeah, sure - I could lean back and say problem solved, but I would
    >>really like to understand why Progy1 is leaking and Progy2 not...
    >>
    >>It seems to me that perl is allocating memory in Progy1.pl for the
    >>variables within the while {} for every single pass through the loop and
    >>never giving them back or reuse or garbage collect them (is the GC too
    >>slow?), while in Progy2.pl - with the variables declared with "my" -
    >>everything works as expected, i.e. no memory leaks, overall memory
    >>consumption aroung 70MB which is ok for the type of files being
    >>processed.

    >
    >
    > How could you fit a hash with >50 million key-value pairs into 70MB of
    > memory?


    Ok ok, you cought me... it was more like 128MB - the 70MB was from 30
    mio records... (plus: I do not think the key/value pairs are all unique)

    >
    > Xho
    >
    Sinisa Susnjar, Jul 19, 2005
    #4
  5. Sinisa Susnjar

    Joe Smith Guest

    wrote:
    >>
    >>Progy1.pl:
    >>#!/usr/bin/perl
    >>while ($line = <>) {
    >> chomp($line);
    >> ($key, $value) = split(':', $line);
    >> $myhash{$key} = $value;
    >>}

    >
    > maybe your just filling %myhash endlessly without deleting elements you
    > no longer need.


    That is a possibility, but does not explain why Progy2 acts
    different from Progy1.

    >>discovered that with rewriting it like Progy2.pl below, the
    >>memory leaks would go away:
    >>
    >>Progy2.pl:
    >>#!/usr/bin/perl
    >>use strict;
    >>use warnings;
    >>my %myhash;

    >
    > see. Your starting fresh every time with an empty hash.


    You're not making sense.
    Progy1 starts with an empty hash every time the script is run.
    Progy2 starts with an empty hash every time the script is run.

    -Joe
    Joe Smith, Jul 19, 2005
    #5
  6. Sinisa Susnjar

    Guest

    Joe Smith wrote:
    > wrote:
    > >>
    > >>Progy1.pl:
    > >>#!/usr/bin/perl
    > >>while ($line = <>) {
    > >> chomp($line);
    > >> ($key, $value) = split(':', $line);
    > >> $myhash{$key} = $value;
    > >>}

    > >
    > > maybe your just filling %myhash endlessly without deleting elements you
    > > no longer need.

    >
    > That is a possibility, but does not explain why Progy2 acts
    > different from Progy1.
    >
    > >>discovered that with rewriting it like Progy2.pl below, the
    > >>memory leaks would go away:
    > >>
    > >>Progy2.pl:
    > >>#!/usr/bin/perl
    > >>use strict;
    > >>use warnings;
    > >>my %myhash;

    > >
    > > see. Your starting fresh every time with an empty hash.

    >
    > You're not making sense.
    > Progy1 starts with an empty hash every time the script is run.
    > Progy2 starts with an empty hash every time the script is run.


    Sorry for not clarifying. In mod_perl, global variables retain their
    values from one execution of a script to another. This means you can
    run Progy1 over and over and keep filling up the hash with new values.
    In Progy2, the hash is lexically scoped and is therefore is garbage
    collected when it goes out of scope. A new hash is created each time
    the script is run.

    -wana

    >
    > -Joe
    , Jul 19, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mr. SweatyFinger

    why why why why why

    Mr. SweatyFinger, Nov 28, 2006, in forum: ASP .Net
    Replies:
    4
    Views:
    879
    Mark Rae
    Dec 21, 2006
  2. Mr. SweatyFinger
    Replies:
    2
    Views:
    1,842
    Smokey Grindel
    Dec 2, 2006
  3. s.subbarayan

    Dynamic memory allocation and memory leak...

    s.subbarayan, Mar 18, 2005, in forum: C Programming
    Replies:
    10
    Views:
    699
    Eric Sosman
    Mar 22, 2005
  4. Mike Solomon

    memory leak in loop

    Mike Solomon, Apr 15, 2005, in forum: Perl Misc
    Replies:
    4
    Views:
    81
    Mike Solomon
    Apr 18, 2005
  5. Isaac Won
    Replies:
    9
    Views:
    372
    Ulrich Eckhardt
    Mar 4, 2013
Loading...

Share This Page