How to release memory ?

[zdbrg]

Hello,

Is there a way to release perl unused memory ?

eg: under linux, the following code would not release memory (unlike a similar
program written in
C)

perl -e '$m = "m" x 50_000_000; $m = ""; undef $m; sleep 1000'

 

[Lukas Mai]

Hello,

Is there a way to release perl unused memory ?

perldoc -q shrink

HTH, Lukas

 

[zdbrg]

Lukas Mai a dit le Mon, 20 Mar 2006 13:36:52 +0100:

perldoc -q shrink

Thanks for your helpfull answer. (-q memory was useless)

So according to what Is said, lexicals ("my" variables) cannot release their
memory because "It is reserved in case the variables come back into scope"
while globals can.


While I do not understand very well the reasons for this optimization,
This does not seem to be always true :

eg: those 2 exemples show similar memory use under linux :

# global variable example
perl -e 'our $m = "m" x 50_000_000; $m = ""; undef $m; sleep 1000'
# same with lexicals
perl -e 'sub m { my $m = "m" x 50_000_000; undef $m; } &m; sleep 1000'

Also, my perl is compiled with system malloc :
$ perl -V:usemymalloc
usemymalloc='n'
So it should release memory (for globals at least) ...

 

[zentara]

Hello,

Is there a way to release perl unused memory ?

eg: under linux, the following code would not release memory (unlike a similar
program written in
C)

perl -e '$m = "m" x 50_000_000; $m = ""; undef $m; sleep 1000'

Read these links.
http://www.perlmonks.org?node_id=336883

Here is an example of releasing memory in Perl. Most of the time,
you will not be able to get such a simple program, so the best you
can do is to "reuse" memory.
This example is an exception to how things normally happen.
Perl will generally not release back to the system, until it exits. It
will free the memory internally for it's own reuse, so you need to
devise strategies for reusing namespaces, when using objects.

If you read those links above, you will find out about the "ref count".
Perl only frees the memory, when the ref count for the data structure
assigned to it, goes to zero. In simple programs, like below, it is easy
to do, but in complex programs, where variables and namespaces get
intertwined, it is usually impossible to get the ref count to go to
zero.

In Tk programming, it gets very tangled, and you can only reuse objects
to prevent memory from climbing. Gtk2 is quite a bit better, and
undef'ing an object usually cleans up after itself and frees the memory
for reuse. But in a complex script, which use objects, you will probably
never see memory freed back to the OS, until the script exits.

The rule in the following is that the variable must be undefined
within it's scope.

#!/usr/bin/perl -w
use strict;
$| = 1;

{
my $string;
for ( 1 .. 100000 ) {
$string .= ( 'x' x 1000 );
}
print "check memory then press enter to release";
<>;

# try running this with the following undef commented out
undef $string;
print "undefined but still in scope of sub, hit enter\n";

<>;

# if the variable only goes out of scope.
# you *need* to undef it!
}

print "ok,out of scope, check mem, press enter to exit";
<>;
__END__

 

[zdbrg]

zentara a dit le Mon, 20 Mar 2006 13:42:26 GMT:

The rule in the following is that the variable must be undefined
within it's scope.

#!/usr/bin/perl -w
use strict;
$| = 1;

{
my $string;
for ( 1 .. 100000 ) {
$string .= ( 'x' x 1000 );
}
print "check memory then press enter to release";
<>;

# try running this with the following undef commented out
undef $string;
print "undefined but still in scope of sub, hit enter\n";

<>;

# if the variable only goes out of scope.
# you *need* to undef it!
}

print "ok,out of scope, check mem, press enter to exit";
<>;
__END__

Thanks for that nice example.

I commented out the "undef" as suggested and memory is not released in that
case, even when it goes out of scope, when memory cannot be reused anymore !
Isn't that a bug ? Shouldn't the refcount decrease to 0 when leaving the
variable's scope
?


Also, When you change the "$string" creation code to :
$string = 'x' x 100_000_000;

Most memory is not released.

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
zdbrg

So according to what Is said, lexicals ("my" variables) cannot release their
memory because "It is reserved in case the variables come back into scope"
while globals can.

Not lexical variables, by lexical variables containing strings.

# global variable example
perl -e 'our $m = "m" x 50_000_000; $m = ""; undef $m; sleep 1000'
# same with lexicals
perl -e 'sub m { my $m = "m" x 50_000_000; undef $m; } &m; sleep 1000'

Sorry, you have very little understanding of what happens there (as
most people do). Do not call m(). You will still get high memory usage.
(Hint: constant folding at compile time.)

Also, my perl is compiled with system malloc :
$ perl -V:usemymalloc
usemymalloc='n'
So it should release memory (for globals at least) ...

I do not think usemymalloc='n' is usable for anything but toy
examples. Good rule is: do not expect your memory going back to
system in any real-life situation.

Hope this helps,
Ilya

 

[Peter J. Holzer]

I do not think usemymalloc='n' is usable for anything but toy
examples.

The perl-porters obviously disagree with you. usemymalloc='n' is the
default on Linux. It is also the way perl is built on at least two major
Linux distributions (Redhat and Debian).

hp

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Peter J. Holzer

The perl-porters obviously disagree with you.

Well, this is not a compliment to perl-porters. ;-)

Hope this helps,
Ilya

 

[zdbrg]

Ilya Zakharevich a dit le Mon, 20 Mar 2006 21:28:26 +0000 (UTC):

Good rule is: do not expect your memory going back to
system in any real-life situation.

Thanks for your answer.

So, how do perl developers handle quite commons tasks where one program needs
to use/allocate really big variables periodicaly which then should be released
to let others tasks run
?

Starting a new process for those jobs which are known to use a lot of memory is
a solution, is there others
?

 

[QoS]

Ilya Zakharevich a dit le Mon, 20 Mar 2006 21:28:26 +0000 (UTC):

Thanks for your answer.

So, how do perl developers handle quite commons tasks where one program needs
to use/allocate really big variables periodicaly which then should be released
to let others tasks run
?

Starting a new process for those jobs which are known to use a lot of memory is
a solution, is there others
?

Perhaps you could use a file; search it by separator (newline, comma, etc..)
value or binary (seek method). Then just work on a bit of the data at
a time, possibly saving output to another file. data-in->process->data-out

side-note: I belive Perl6 has some features which allow for manual
garbage collection.

gl

 

[Dr.Ruud]

zdbrg schreef:

So, how do perl developers handle quite commons tasks where one
program needs to use/allocate really big variables periodicaly which
then should be released to let others tasks run

Do you have an example of such a task? Is Perl the right language (is
perl the right implementation) for that task?

 

[zentara]

Ilya Zakharevich a dit le Mon, 20 Mar 2006 21:28:26 +0000 (UTC):

Thanks for your answer.

So, how do perl developers handle quite commons tasks where one program needs
to use/allocate really big variables periodicaly which then should be released
to let others tasks run
?

Reuse the space yourself, commonly by using a $global to hold the data.
When you are done with the first $data run, set $data = '', and do the
next task, using $data again. The memory from the first use of $data
will be freed internally by Perl, and will be reused, but it "probably"
won't be returned to the system.

So it is common in Perl programs, to see the system memory usage rise to
a peak, of whatever the greatest data size was. Like a peak-meter.

Usually in complex programs, objects are being used. There are various
tricks you learn to reuse the same object over-and-over, and just undef
the data portion of the object. You cannot reliably use the
"object-create-destroy" cycle, without gainig memory. Each object has
it's own technique for clearing out old data.

I'm not a Perl internals type of person, but the developers are trying
to improve Perl's memory conservation. For instance in Perl/Tk, it is
absolutely essential to reuse any object you create. While Perl/Gtk2
has made significant strides in cleaning itself up automatically, so you
can often use the create-destroy method of object handling.

Perl6 has made promises of improvements across the board.

Starting a new process for those jobs which are known to use a lot of memory is
a solution, is there others
?

It is common to use disk-based databases to handle sharing of huge data.
That way you can fork to do whatever you need, and when the forked
process is done, it leaves it's results in the db, and totally returns
the memory to the system.

Another way to share forked data, is thru shared memory segments, but
they are tricky.

Threads will act like objects. If you continually create threads, then
destroy them when done, memory will just climb. You need to reuse
threads, just feeding them different data for each run. Threads are
great for sharing data in realtime, but they will suck system memory if
not carefully handled. Once again, the memory will rise to peak value.

So there is not going to be 1 general purpose answer to your question.
Each program needs to make it's own tradeoffs, concerning size,
performance, ease of sharing data, etc.

But for programs which use 100's of megs of data, you are best off
forking them, and storing results in a common disk-file-db. The Storable
module is often used for this.

Good luck.

 

[xhoster]

Ilya Zakharevich a dit le Mon, 20 Mar 2006 21:28:26 +0000 (UTC):

Thanks for your answer.

So, how do perl developers handle quite commons tasks where one program
needs to use/allocate really big variables periodicaly which then should
be released to let others tasks run
?

Mostly I just let the VM system do it's job. Are you sure that won't work
for you?

Xho

 

[zdbrg]

(e-mail address removed) a dit le 21 Mar 2006 14:50:07 GMT:

Mostly I just let the VM system do it's job. Are you sure that won't work
for you?

Xho

Yes it will, at the cost of swapping hundred megs of memory to disk.

As zentara pointed out, using some kind of persistant database (which was not
wanted at the first time), will surely drop down memory use, adding a bit of
complexity, and probably slowlyness. Forking or using a crontab (on *nix
systems) for that is surely another good solution.



Thanks all for your interesting answers.

 

[xhoster]

(e-mail address removed) a dit le 21 Mar 2006 14:50:07 GMT:


Yes it will, at the cost of swapping hundred megs of memory to disk.

A hundred meg here, an hundred meg there, pretty soon you may be talking
about real IO .

Swapping lots of memory to disk is often a small price to pay for the
benefit of not having to micromanage your program's memory use. And when
that price is too high, then your optimization choices are so intimately
tied in with the nitty-gritty details of your code that it is hard to
discuss them meaningfully in generalities. At various times, I've:

Opened "+<" then used seek, tell, read, etc.
Used Acme::Abuse to isolate certain parts to separate process.
Used various tie modules to tie the data to disk.
Used mmap through XS code (although doing it again I'd probably look into
Sys::Mmap first) Made my code readily restartable, so I could exit after a
particularly memory heavy usage and then pick up where it left off in a
fresh interpreter.

But the vast majority of the time I don't do any of these rather painful
and time consuming (my time, that is) things, I just let the VM system swap
out hundreds of megabytes.

As zentara pointed out, using some kind of persistant database (which was
not wanted at the first time), will surely drop down memory use, adding a
bit of complexity, and probably slowlyness.

This is often a good idea, especially if your large data structure is a
hash. But if it is a scalar, like your example seems to suggest, it is less
clear how useful this will be.

Forking or using a crontab
(on *nix systems) for that is surely another good solution.

I don't see how crontab would help here, but forking certainly could if you
have a lot of mostly independent tasks where some are very big and some
are not. But if in that situation, I'd tend to use forking anyway, even
if memory was not a concern.

Xho

 

[Tad McClellan]

A hundred meg here, an hundred meg there, pretty soon you may be talking
about real IO .

A veiled reference to one of my most favorite quotes:

http://www.quotationspage.com/quote/170.html


But maybe the attribution is not accurate:

http://www.dirksencenter.org/print_emd_billionhere.htm

 

[robic0]

Ilya Zakharevich a dit le Mon, 20 Mar 2006 21:28:26 +0000 (UTC):

Thanks for your answer.

So, how do perl developers handle quite commons tasks where one program needs
to use/allocate really big variables periodicaly which then should be released
to let others tasks run
?

Starting a new process for those jobs which are known to use a lot of memory is
a solution, is there others
?

Consider the price for loose form and structure provided by Pseudo languages
such as Perl, Java, ur's here... to be paid! Actually you should program in
'C/C++' with a good os api. In that environment you can do your own memory
management, including low level routines that do block allocation as well
as defragmentation, without leaving the program..

Perl is a garbage collector that provides no usefull control of the os
primatives.
IMO Perl hasn't got long to live. If you put all ur eggs in this basket
you will be in trouble careerwise as well...

robic0

 

[robic0]

A hundred meg here, an hundred meg there, pretty soon you may be talking
about real IO .

Swapping lots of memory to disk is often a small price to pay for the
benefit of not having to micromanage your program's memory use. And when
that price is too high, then your optimization choices are so intimately
tied in with the nitty-gritty details of your code that it is hard to
discuss them meaningfully in generalities. At various times, I've:

So ahh, when do you have *ANY* programitic control of when and how much
gets swapped to *disk* ?????? Huh...?

Opened "+<" then used seek, tell, read, etc.
Used Acme::Abuse to isolate certain parts to separate process.
Used various tie modules to tie the data to disk.
Used mmap through XS code (although doing it again I'd probably look into
Sys::Mmap first) Made my code readily restartable, so I could exit after a
particularly memory heavy usage and then pick up where it left off in a
fresh interpreter.

But the vast majority of the time I don't do any of these rather painful
and time consuming (my time, that is) things, I just let the VM system swap
out hundreds of megabytes.


This is often a good idea, especially if your large data structure is a
hash. But if it is a scalar, like your example seems to suggest, it is less
clear how useful this will be.


I don't see how crontab would help here, but forking certainly could if you
have a lot of mostly independent tasks where some are very big and some
are not. But if in that situation, I'd tend to use forking anyway, even
if memory was not a concern.

Xho

Your such a blow hard. Excuse me but you don't know your ass from your
elbow. You never did, you never will..

robic0

 

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
zdbrg

So, how do perl developers handle quite commons tasks where one
program needs to use/allocate really big variables periodicaly which
then should be released to let others tasks run ?

If this is A FEW of REALLY BIG allocations, then indeed, an
implementation of malloc() which has a possibility of releasing some
regions of memory back to system may indeed have a chance to ACTUALLY
USE this possibility. So disabling `usemymalloc' in such situations
will be beneficial.

However, in my experience, typical memory hog Perl applications would
hog memory by a lot of relatively small chunks. Then `usemymalloc'
will have a lot of advantages over a typical "system" malloc(): it
should be quickier, use less memory, and, most important of all, it
gives feedback to Perl; in addition to additional speedups, this
feedback simplifies memory debugging a lot.

With feedback, the developer has a real chance to improve the
algorithm. Think about it this way: it is the algorithm, not Perl,
which hogs the memory. 1/3 ;-)

Hope this helps,
Ilya

 

[robic0]

[A complimentary Cc of this posting was sent to
zdbrg

So, how do perl developers handle quite commons tasks where one
program needs to use/allocate really big variables periodicaly which
then should be released to let others tasks run ?

If this is A FEW of REALLY BIG allocations, then indeed, an
implementation of malloc() which has a possibility of releasing some
regions of memory back to system may indeed have a chance to ACTUALLY
USE this possibility. So disabling `usemymalloc' in such situations
will be beneficial.

How in the **** do you know this ???

However, in my experience, typical memory hog Perl applications would
hog memory by a lot of relatively small chunks. Then `usemymalloc'
will have a lot of advantages over a typical "system" malloc(): it
should be quickier, use less memory, and, most important of all, it
gives feedback to Perl; in addition to additional speedups, this
feedback simplifies memory debugging a lot.

With feedback, the developer has a real chance to improve the
algorithm. Think about it this way: it is the algorithm, not Perl,
which hogs the memory. 1/3 ;-)

Hope this helps,
Ilya

Unfortunately, you don't know any of which you talk about. You actually
think you can somehow control the usage of memory in Perl.
Its just astonishing !!

Please tell me how you can. This is a miracle!!!

robic0