perl eating up memory slowly until program stops - any ideas ?

[Jasper]

Hello, I am clearing the contents of the arrays with @array = (); for
all arrays, and I print out the $#array to get the size while the
program is running for all arrays...and the arrays are not growing..so
I am stumped as to what could be causing the fact that memory needs
continue to increase...I am using GMP::BigFloat and doing large
precision subtractions..

Any tips are appreciated as to how I can manage/stop this..
Thank you,

Jasper

 

[Matt Garrish]

However, what none will tell you is although lexical scoping
releases memory for reuse, you will trade some efficiency for
better memory management.

Use of lexical scoping will decrease efficiency by percentages
which are easy to measure.

When memory is an issue, lexical scoping is useful, otherwise
a programmer should question if lexical scoping is beneficial
within a context of efficiency.


Purl Gurl
--
#!perl

print "Content-type: text/plain\n\n";

use Benchmark;

print "Run One:\n\n";
&Time;

print "\n\nRun Two:\n\n";
&Time;

print "\n\nRun Three:\n\n";
&Time;

sub Time
{
timethese (100000,
{
'Purl_Gurl' =>
'@Array1 = qw (Now is the time for all good men to come to the aid of their country);
@Array2 = qw (Behind every successful man walks a woman pushing him along);
$var1 = "@Array1";
@Array1 = ();
$var2 = "@Array2";
@Array2 = ();',

'Purl_Gurl2' =>
'{
my @Array3 = qw (Now is the time for all good men to come to the aid of their country);
my $var3 = "@Array3";
@Array3= ();
}
{
my @Array4 = qw (Behind every successful man walks a woman pushing him along);
my $var4 = "@Array4";
@Array4= ();
}',
} );
}

If you're trying to prove that well-conceived and structured code might
sometimes run marginally slower than spaghetti code, you're not exactly onto
something new (or do you just randomly wrap code in blocks for fun). The
fact remains, however, that spaghetti code is usually a sign of bad code,
and optimizing it will more often than not reduce the runtime anyway. But I
suppose there's no point in preaching to someone who has never had a job as
a programmer. Rulez are for losers! Rock on PG!

Matt

 

[Matt Garrish]

Matt Garrish continues to ignorantly troll:


I read no spaghetti code in my benchmark example. You are
practicing poorly veiled deceit as a premise for trolling.

My example exemplifies lexical scoping has an efficiency penalty.

Actually, it doesn't. I ran it a few times for a laugh and each time it
alternated which was faster. Moreover, if you did an accurate test and also
placed your globals inside a block (since a block does not enforce scoping,
using strictures and my variables does), your global example runs
considerably slower. This proves nothing though, except that you aren't very
good at benchmarking.

Not bothering to scope your variables (and that means *limiting* the scope)
might be fine for short scripts, but is terrible programming practice in
general. Programming is rarely about squeezing every second out of your
code, and certainly not at the expense of churning out spaghetti. And if you
can't look at your simple example and see that the global example is well on
the path to spaghetti-dom, then in the words of someone here, that's your
problem, not mine.

Matt

 

[Jasper]

However, what none will tell you is although lexical scoping
releases memory for reuse, you will trade some efficiency for
better memory management.

Use of lexical scoping will decrease efficiency by percentages
which are easy to measure.

When memory is an issue, lexical scoping is useful, otherwise
a programmer should question if lexical scoping is beneficial
within a context of efficiency.


Purl Gurl
--
#!perl

print "Content-type: text/plain\n\n";

use Benchmark;

print "Run One:\n\n";
&Time;

print "\n\nRun Two:\n\n";
&Time;

print "\n\nRun Three:\n\n";
&Time;

sub Time
{
timethese (100000,
{
'Purl_Gurl' =>
'@Array1 = qw (Now is the time for all good men to come to the aid of their country);
@Array2 = qw (Behind every successful man walks a woman pushing him along);
$var1 = "@Array1";
@Array1 = ();
$var2 = "@Array2";
@Array2 = ();',

'Purl_Gurl2' =>
'{
my @Array3 = qw (Now is the time for all good men to come to the aid of their country);
my $var3 = "@Array3";
@Array3= ();
}
{
my @Array4 = qw (Behind every successful man walks a woman pushing him along);
my $var4 = "@Array4";
@Array4= ();
}',
} );
}



PRINTED RESULTS:
________________

Run One:

Benchmark: timing 100000 iterations of Purl_Gurl, Purl_Gurl2...
Purl_Gurl: 6 wallclock secs ( 5.43 usr + 0.00 sys = 5.43 CPU) @ 18416.21/s
Purl_Gurl2: 6 wallclock secs ( 5.66 usr + 0.00 sys = 5.66 CPU) @ 17667.84/s


Run Two:

Benchmark: timing 100000 iterations of Purl_Gurl, Purl_Gurl2...
Purl_Gurl: 6 wallclock secs ( 5.49 usr + 0.00 sys = 5.49 CPU) @ 18214.94/s
Purl_Gurl2: 6 wallclock secs ( 5.77 usr + 0.00 sys = 5.77 CPU) @ 17331.02/s


Run Three:

Benchmark: timing 100000 iterations of Purl_Gurl, Purl_Gurl2...
Purl_Gurl: 5 wallclock secs ( 5.44 usr + 0.00 sys = 5.44 CPU) @ 18382.35/s
Purl_Gurl2: 4 wallclock secs ( 5.76 usr + 0.00 sys = 5.76 CPU) @ 17361.11/s

Thanks...
Great, but I have a subroutine which populates my arrays and then
OTHER subroutines analyze the data which are in the arrays...I cant
think of a way to make the arrays local....the whole point of
subroutines is modularity..how do folks conquer this with their
programs ? I cant copy large sections of code redundantly everywhere
! I dont know C so I am stuck...Can I use some kind of Perl "malloc"
syntax ? any ideas ?

Here is an example of what I am doing:

I have an array with no special declaration in Sub X:
for ($xx=$lastElem;$xx>=($lastElem-($additive-1));$xx--)
{
push @bottomArray1, $dataSet_init[$xx];
}

And then using it in Sub Y:
$subtractVal = $bottomArray1[0]- $bottomArray1[1];

What can I do to get around this ?

 

[Eugene Mikheyev]

Thanks...

Great, but I have a subroutine which populates my arrays and then
OTHER subroutines analyze the data which are in the arrays...I cant
think of a way to make the arrays local....the whole point of
subroutines is modularity..how do folks conquer this with their
programs ? I cant copy large sections of code redundantly everywhere
! I dont know C so I am stuck...Can I use some kind of Perl "malloc"
syntax ? any ideas ?

Here is an example of what I am doing:

I have an array with no special declaration in Sub X:
for ($xx=$lastElem;$xx>=($lastElem-($additive-1));$xx--)
{
push @bottomArray1, $dataSet_init[$xx];
}

And then using it in Sub Y:
$subtractVal = $bottomArray1[0]- $bottomArray1[1];

What can I do to get around this ?

You should have shown a place where your @bottomArray is declared, and how
your subs are called.

 

[Tassilo v. Parseval]

Also sprach Purl Gurl:

However, what none will tell you is although lexical scoping
releases memory for reuse, you will trade some efficiency for
better memory management.

Use of lexical scoping will decrease efficiency by percentages
which are easy to measure.

When memory is an issue, lexical scoping is useful, otherwise
a programmer should question if lexical scoping is beneficial
within a context of efficiency.

[ benchmark snipped ]

You only measures the time it takes to create and populate those
variables. Not a very realistic scenario considering that programs
hardly ever only create them. They also want to access them.

When it comes to that, lexicals usually perform better than globals.
There are a few exceptions, notably $_ and @_, which get special
treatment by the perl core through the global symbol PL_defgv.

No such shortcut exists for ordinary globals, though. Perl first has to
get the corresponding glob from the symbol-table after which it must get
at the correct slot. Lexicals however just live on the pad as SVs.
All perl has to do is C-cast the variable to its correct type.

Tassilo

 

[Joe Smith]

Great, but I have a subroutine which populates my arrays and then
OTHER subroutines analyze the data which are in the arrays...I cant
think of a way to make the arrays local....the whole point of
subroutines is modularity..how do folks conquer this with their
programs ? I cant copy large sections of code redundantly everywhere
! I dont know C so I am stuck...Can I use some kind of Perl "malloc"
syntax ? any ideas ?

{
my @bottomArray1;
X(\@bottomArray1);
Y(\@bottomArray1);
}
sub X {
$reference_to_array = shift;
push @{$reference_to_array},...
}
sub Y {
$array_ref = shift;
$subtractVal = $array_ref->[0] - $array_ref->[1];
...
}

If you use function prototypes, the \ can be eliminated from \@bottomArray1.
-Joe

 

[Tassilo v. Parseval]

Also sprach Purl Gurl:

(snipped)

[ benchmark snipped ]

You only measures the time it takes to create and populate those
variables. Not a very realistic scenario considering that programs
hardly ever only create them. They also want to access them.

Actually, my benchmark does a bit more to closer reflect
a simple script. Arrays are created, variables are set,
arrays are set to null.

You don't show how long it takes to access single elements, for
instance.

Difference in timing is contributable to creating private
blocks and is contributed to reuse of memory allotments.

Without private blocks, unique "malloc" are created for
each array (and variables) during compilation.

With private blocks, blocks are created and a "single" malloc
is created flagged for reuse. My use of "single" is not to
imply only one malloc is created in all cases.

For my example extra processing is invoked by perl core to
keep track of blocks and malloc reuse.

I don't understand anything of the above. What is a '"single" malloc'
which is 'flagged for reuse'? Furthermore, what are those 'private
blocks'?

Inherently, there are lots of exceptions, oddities and other
externalities effecting efficiency. For my benchmark, difference
in efficiency is so small it is basically insignificant.

For large programs, efficiency difference is significant.

Yes, but not in the sense that you would expect:

#! /usr/bin/perl -w

use Benchmark qw(cmpthese);

cmpthese( -2, {
pkg => sub {
@array = (1 .. 10000);
for $e (@array) {
$e++;
}
},
lex => sub {
my @ary = (1 .. 10000);
for my $e (@ary) {
$e++;
}
},
});
__END__
Rate pkg lex
pkg 89.6/s -- -41%
lex 151/s 68% --

Is that what you meant with 'significant'? ;-)

However, for large programs reuse of memory may prove to
be more efficient than not using lexical scoping. I have
not tested large programs with benchmark. Still, this
noted loss in efficiency, logically would be expected
to amplify with larger scripts.

'reuse of memory may prove to be more efficient than not using lexical
scoping'? Another one I don't understand.

Tassilo

 

[Tassilo v. Parseval]

Also sprach Purl Gurl:

Tassilo v. Parseval wrote:
Purl Gurl wrote:
Purl Gurl wrote:
Jasper wrote:

(snipped)

[ benchmark snipped ]
Actually, my benchmark does a bit more to closer reflect
a simple script. Arrays are created, variables are set,
arrays are set to null.

You don't show how long it takes to access single elements, for
instance.

That is unrelated. The point of my test is to exemplify
identical code takes longer to run when lexical scoping
is employed.

I don't understand anything of the above. What is a '"single" malloc'
which is 'flagged for reuse'? Furthermore, what are those 'private
blocks'?

My hunch is you already know and are testing my knowledge.

For lexical scoping, a single chunk of memory, reserved by
perl core, can be reused many times. That is a single "malloc"
serving memory needs multiple times.

Simple example, you can push an array into that "malloc" then
later destroy it via lexical scoping, followed by another array
being pushed into that same "malloc" memory chunk.

Another here used a very good expression of calling those
memory chunks an "arena" which is more understandable;
blocks of reserved memory for use.

Ah, that. Without having looked too closely, global (package) variables
use the same memory allocation strategy. That means that memory attached
to dynamic variables will be reused as well, provided you tell perl that
it is allowed to reuse it, which for instance happens through a
statement like this:

@array = (1 .. 10000);
...
@array = (); # the memory of the 10000 scalars reusable now

"Inherently, there are lots of exceptions, oddities and other...."

As I was trying to tell you, lexical variables being quicker than
dynamic ones is not an exception, it's the rule.

In general, use of global variables lends to a faster
running script. Lexical scoping, slower. Use of "my"
declarations within private blocks increases activity
by perl core, hence, slower script completion.

No, it doesn't. The Perl interpreter runs through fewer lines of C code
when using lexical values. The whole procedure of unwrapping the scalar
(or array or hash) from a glob isn't done. With a lexical, perl knows at
compile time which of the various C structures to use. When they are put
into the pad (which is a special AV that holds the lexicals for a given
scope), they are simply casted to an SV. This is exactly one line of C
code:

#define PAD_SETSV(po,sv) PL_curpad[po] = (sv)

And retrieving a, say, array from this pad is just a matter of doing

AV *ary = (AV*)PL_curpad[po];

Whereas you want to tell me that retrieving a dynamic variable is less
than subscripting a C array plus a typecast? After all, you said that
lexicals increase the activity in the Perl core.

As pointed out previously, it is very possible better
use of memory may actually lend to better efficiency.
I am thinking in terms of high RAM usage creating a
greater degree of paging / swap file usage, which
is slower because of hard drive access.

It doesn't have anything to do with that. Lexicals are faster because
perl has to carry out less work. I don't understand how one could
possibly argue about that.

Of course, you are free to continue using globals and pretend that
they'll make your programs perform better. Even if they slow them down.

Tassilo

 

[Jasper]

Then you are in a position to use lexical scoping.

Combine your subroutines into one. Use of if / else logic
is more efficient than invoking subroutines.

Combine into 1 subroutine ? Are you joking ? The sub that populates
the arrays is called constantly in several areas, collapsing into 1
subroutine would mean I would need to paste the same in multiple
places within that routine...that makes no sense..please clarify why
this would work..this program is very iterative and more than 1 sub
makes sense, but doing this has compromised my memory..

Thanks
Jasper

I cant think of a way to make the arrays local.

Not local, but rather lexically scoped.

Perhaps you need to think more?

the whole point of subroutines is modularity.

Modules are modularity. Subroutines are used simply
to write one task which is invoked multiple times.

I cant copy large sections of code redundantly everywhere

Copy and paste works well for this. However, you should
debate if you want to repeat code or want to write a
single piece of code which is more efficient.

for ($xx=$lastElem;$xx>=($lastElem-($additive-1));$xx--)
{
push @bottomArray1, $dataSet_init[$xx];
}

$subtractVal = $bottomArray1[0]- $bottomArray1[1];

Not enough information for comment nor do I read
any use of lexical scoping.


Purl Gurl

 

[Tassilo v. Parseval]

Also sprach Purl Gurl:

Tassilo v. Parseval wrote:

I made no mention of dynamic variables.

You used them in your benchmark. Every variable not declared with 'my'
is a dynamic variable. They are called that way because they are subject
to 'local' and they can temporarily be given new values.

Perl benchmark contradicts your statement.

My benchmark backed up my statement. You have yet to come up with a
benchmark that models a real-world scenario and that would contradict my
statement. With which I mean none of those meaningless benchmarks that
you always come up with.

Tassilo

 

[MichiganBob]

Programming is rarely about squeezing every second out of your
code, and certainly not at the expense of churning out spaghetti. And if you
can't look at your simple example and see that the global example is well on
the path to spaghetti-dom, then in the words of someone here, that's your
problem, not mine.

Stop it right now, you hear? Call PG's code spaghetti gives
spaghetti a bad name and I just won't stand for that...

 

[Bob Walton]

Jasper wrote:
....


{
my @bottomArray1;
X(\@bottomArray1);
Y(\@bottomArray1);
}
sub X {
$reference_to_array = shift;
my^


push @{$reference_to_array},...
}
sub Y {
$array_ref = shift;
my^


$subtractVal = $array_ref->[0] - $array_ref->[1];
my^


....


-Joe

 

[Matt Garrish]

Those are global variables.

Take the time to figure out what a dynamic variable is before posting:

local() always affects global variables, also called package variables or
dynamic variables.

(perlfaq7 -- What's the difference between dynamic and lexical (static)
scoping? Between local() and my()?)

Retreating to personal insults does not
lend well to your credibility.

Strange how that doesn't hold true for yourself, though...

Matt

 

[Matt Garrish]

For those interested in learning Perl,


$purlgurl = "rocks and rolls";

That is a global variable.

And what package would that be in? To everyone but you, it's the same as
$main:urlgurl. A global variable is always a package variable. Likewise,
global variable are localized to the package they're in, which also make
them dynamic. Starting to see the bigger picture yet?

Now please stop flaunting your ignorance.

Matt

 

[Matt Garrish]

Others are now beginning to see the big picture,
of which I have been aware for years.

Here is a shovel. Clean up this mule manure
mess you make of this newsgroup, daily;
some are becoming annoyed by this stench
you and your friends bring to this group.

Didn't I tell you to get a dictionary a couple of years back and work on
expanding your vocabulary? This pathetic song-and-dance routine of yours
bores me and everyone else here, so please drop it already.

Matt

 

[Dale Henderson]

I know this is a waste of time but perhaps it will help protect
the innocent.




PG> For those interested in learning Perl,


PG> $purlgurl = "rocks and rolls";

PG> That is a global variable.

Yes this is the global variable $main:urlgurl (assuming main is
the current package);

PG> if ($purlgurl eq "rocks and rolls")
PG> { local ($purlgurl) = "rocks my socks off"; }

PG> That is a dynamic variable.

No. That is a global variable thats dynamically scoped.

From perlfaq7:

In summary, local() doesn't make what you think of as pri-
vate, local variables. It gives a global variable a tem-
porary value. my() is what you're looking for if you want
private variables.


PG> package PurlGurl;
PG> $Purlgurl = "born to be wild";

PG> That is a package variable.

That is the global variable $PurlGurl:urlGurl.

PG> There are many here, perhaps a majority here, who present a
PG> facade of knowing Perl. In reality, their knowledge of Perl is
PG> less than adequate, but they are quite talented at passing off
PG> BS upon others, and upon themselves.

Well I know of at least one.

PG> Years back, Elaine wrote I know more about Perl than ninety
PG> percent of the participants here. You, plural, continue to
PG> prove her correct.

Could you post the Message-ID or a google link. I'd be interested
in reading this.

By the way what does Abigail think of your perl skills?


PG> Purl Gurl --
PG> http://www.purlgurl.net/~callgirl/rockmusi.cgi?bornwild

Encouraging hacking again I see.

 

[David H. Adler]

You have forgotten you have me killfiled.

You keep saying that. I do not think it means what you think it means.