Memory leak in loop when not using "my" - why?

[Sinisa Susnjar]

Hello Group,

could a Perl Guru please explain to me what is going on here (so I can
become one too ;-) )?!?

Progy1.pl:
#!/usr/bin/perl
while ($line = <>) {
chomp($line);
($key, $value) = split(':', $line);
$myhash{$key} = $value;
}

Progy1.pl shows a small piece of code from a larger program that is
supposed to do mass-data manipulation (>50 mio lines of data).
I was asked to check for memory leaks... it was leaking tons (>4GB) of
memory and I discovered that with rewriting it like Progy2.pl below, the
memory leaks would go away:

Progy2.pl:
#!/usr/bin/perl
use strict;
use warnings;
my %myhash;
while (my $line = <>) {
chomp($line);
my ($key, $value) = split(':', $line);
$myhash{$key} = $value;
}

Yeah, sure - I could lean back and say problem solved, but I would
really like to understand why Progy1 is leaking and Progy2 not...

It seems to me that perl is allocating memory in Progy1.pl for the
variables within the while {} for every single pass through the loop and
never giving them back or reuse or garbage collect them (is the GC too
slow?), while in Progy2.pl - with the variables declared with "my" -
everything works as expected, i.e. no memory leaks, overall memory
consumption aroung 70MB which is ok for the type of files being processed.

btw, I am using perl 5.6.1 on Solaris 2.8...

I read the FAQs (I hope thorougly enough...)

Hope, somebody can help...

Regards,
Sinisa Susnjar

 

[ioneabu]

Hello Group,

could a Perl Guru please explain to me what is going on here (so I can
become one too ;-) )?!?

Progy1.pl:
#!/usr/bin/perl
while ($line = <>) {
chomp($line);
($key, $value) = split(':', $line);
$myhash{$key} = $value;
}

maybe your just filling %myhash endlessly without deleting elements you
no longer need.

Progy1.pl shows a small piece of code from a larger program that is
supposed to do mass-data manipulation (>50 mio lines of data).
I was asked to check for memory leaks... it was leaking tons (>4GB) of
memory and I discovered that with rewriting it like Progy2.pl below, the
memory leaks would go away:

Progy2.pl:
#!/usr/bin/perl
use strict;
use warnings;
my %myhash;

see. Your starting fresh every time with an empty hash.

 

[xhoster]

Hello Group,

could a Perl Guru please explain to me what is going on here (so I can
become one too ;-) )?!?

Progy1.pl:
#!/usr/bin/perl
while ($line = <>) {
chomp($line);
($key, $value) = split(':', $line);
$myhash{$key} = $value;
}

Progy1.pl shows a small piece of code from a larger program that is
supposed to do mass-data manipulation (>50 mio lines of data).
I was asked to check for memory leaks... it was leaking tons (>4GB) of
memory

What was leaking memory? The above code, or the larger program you
abstracted it from?

and I discovered that with rewriting it like Progy2.pl below, the
memory leaks would go away:

Progy2.pl:
#!/usr/bin/perl
use strict;
use warnings;
my %myhash;
while (my $line = <>) {
chomp($line);
my ($key, $value) = split(':', $line);
$myhash{$key} = $value;
}

I find it very hard to believe that, of the code you showed, one leaks
and the other doesn't.

Yeah, sure - I could lean back and say problem solved, but I would
really like to understand why Progy1 is leaking and Progy2 not...

It seems to me that perl is allocating memory in Progy1.pl for the
variables within the while {} for every single pass through the loop and
never giving them back or reuse or garbage collect them (is the GC too
slow?), while in Progy2.pl - with the variables declared with "my" -
everything works as expected, i.e. no memory leaks, overall memory
consumption aroung 70MB which is ok for the type of files being
processed.

How could you fit a hash with >50 million key-value pairs into 70MB of
memory?

Xho

 

[Sinisa Susnjar]

What was leaking memory? The above code, or the larger program you
abstracted it from?

The above code. The larger program was filled with multiple
code-snippets like the above.

I find it very hard to believe that, of the code you showed, one leaks
and the other doesn't.

I could not believe it myself! I tried to reproduce it @home under x86
Linux with perl 5.8.3 and I couldn't! I will try downloading the newest
perl version today at my jobs place and see if that makes a difference...

How could you fit a hash with >50 million key-value pairs into 70MB of
memory?

Ok ok, you cought me... it was more like 128MB - the 70MB was from 30
mio records... (plus: I do not think the key/value pairs are all unique)

 

[Joe Smith]

maybe your just filling %myhash endlessly without deleting elements you
no longer need.

That is a possibility, but does not explain why Progy2 acts
different from Progy1.

see. Your starting fresh every time with an empty hash.

You're not making sense.
Progy1 starts with an empty hash every time the script is run.
Progy2 starts with an empty hash every time the script is run.

-Joe

 

[ioneabu]

That is a possibility, but does not explain why Progy2 acts
different from Progy1.


You're not making sense.
Progy1 starts with an empty hash every time the script is run.
Progy2 starts with an empty hash every time the script is run.

Sorry for not clarifying. In mod_perl, global variables retain their
values from one execution of a script to another. This means you can
run Progy1 over and over and keep filling up the hash with new values.
In Progy2, the hash is lexically scoped and is therefore is garbage
collected when it goes out of scope. A new hash is created each time
the script is run.

-wana