ithreads & memory

[Micha³ Lesiak (bler)]

Hello,

A simple script:

#!/usr/bin/perl

use threads;
use Time::HiRes "usleep";

sub th {
usleep(10000);
}

while(1) {
$th_n = threads->new(\&th);
$th_n->detach;
usleep(100000);
}

and that is, every 0,1s a new thread is created, and it runs for 0,01s. So,
when a new $th_n starts, the previous one is dead and gone - but the memory
is not released. The script goes on until it eats up all the memory avaible
and segfaults.

Now, this is just to show the problem, my real application creates a couple
of 10s threads once in a while - and it runs for months, which in due
course ends just like the above, out of memory and segfault. I know there
are some problems with ithreads, but in my case this means it's completely
unusable, so I think I'm doing something wrong. Can you help me?

perl 5.8.7 (5.8.2, 5.8.5 also been tried), kernel 2.6.11.6 (some other
versions tested too), threads 1.05.

 

[xhoster]

Hello,

A simple script:

#!/usr/bin/perl

use threads;
use Time::HiRes "usleep";

sub th {
usleep(10000);
}

while(1) {
$th_n = threads->new(\&th);
$th_n->detach;
usleep(100000);
}

You should check the actual time slept by usleep. It is possible
that the parent is waking up early and hence spawning new threads
more often than you think.

Xho

 

[Micha³ Lesiak (bler)]

You should check the actual time slept by usleep. It is possible
that the parent is waking up early and hence spawning new threads
more often than you think.

It's all the same when I use "sleep 1" and "sleep 4", respectively. It just
take longer. As I said, it may take weeks but it does happen, once the
memory for a thread is allocated it's released only when the program quits.

 

[xhoster]

It's all the same when I use "sleep 1" and "sleep 4", respectively.

I'm not sure that that precludes this possibility.

It
just take longer. As I said, it may take weeks but it does happen, once
the memory for a thread is allocated it's released only when the program
quits.

I can't produce the problem on 5.8.0 with threads 0.99
(or maybe I just am not waiting long enough).

Are the threads hanging around holding memory, or is the memory building
up in the parent process? (e.g. does:
ps -efl | fgrep <progname>
show an ever increasing number of processes, or just one or two processes
with increasing memory use?

Xho

 

[Micha³ Lesiak (bler)]

I'm not sure that that precludes this possibility.

Okay, consider this:

use threads;
use Time::HiRes "usleep";

sub th {
usleep(10000);
}

for(my $inc; $inc < 100; $inc++) {
$th_n = threads->new(\&th);
$th_n->detach;
usleep(100000);
}

while (1) {
sleep 1;
}

So now it creates 100 threads, the while loop is there to pretend that the
script is doing some other things. After the first while execution, the
last thread should be finished. On my system, as before, memory is still
taken (about 43MB).

Are the threads hanging around holding memory, or is the memory building
up in the parent process? (e.g. does:
ps -efl | fgrep <progname>
show an ever increasing number of processes, or just one or two processes
with increasing memory use?

There are only two processes present.

 

[A. Sinan Unur]

Okay, consider this:

use threads;
use
Time::HiRes "usleep";

Oh, what's going on? I see, you have a ton of embedded hard tabs or
something. Please don't do that. It makes it hard on your readers.

Why are don't you have

use strict;
use warnings;

in your script?

for(my $inc; $inc < 100; $inc++) {
$th_n = threads->new(\&th);
$th_n->detach;
usleep(100000);
}

Is there a reason you are using the same $th_n for all the threads over
and over again?

Consider the code below:

#!/usr/bin/perl

use strict;
use warnings;

use threads;
use Time::HiRes "usleep";

sub th { usleep 10000; print "$_[0]: Done\n" }

my $th_n;
for (1 .. 100) {
$th_n = threads->new(\&th, $_);
$th_n->detach;
usleep 100000;
}

sleep 1 while 1;

__END__

When I ran this code on my system (Windows XP SP2, Perl 5.8.7), the
memory footprint of the Perl process grew to 46,696K and stayed there.
This looks like a standard leaker to me (although I could not describe
to you the exact steps in which resources are being leaked), but it is,
in principle, similar to the following C code:

#include <stdlib.h>

void leak(void) {
char *p;
while( 1 ) {
p = malloc(1);
}
}

Now, with one minor change, as in the following:

#!/usr/bin/perl

use strict;
use warnings;

use threads;
use Time::HiRes "usleep";

sub th { usleep 10000; print "$_[0]: Done\n" }

for (1 .. 100) {
my $th_n = threads->new(\&th, $_);
$th_n->detach;
usleep 100000;
}

sleep 1 while 1;

__END__

The maximum memory footprint stays at 3564K throughout the life of the
script.

The maxim: Always declare variables in the smallest applicable scope.

Sinan

 

[xhoster]

Is there a reason you are using the same $th_n for all the threads over
and over again?

Consider the code below:

#!/usr/bin/perl

use strict;
use warnings;

use threads;
use Time::HiRes "usleep";

sub th { usleep 10000; print "$_[0]: Done\n" }

my $th_n;
for (1 .. 100) {
$th_n = threads->new(\&th, $_);
$th_n->detach;
usleep 100000;
}

sleep 1 while 1;

__END__

When I ran this code on my system (Windows XP SP2, Perl 5.8.7), the
memory footprint of the Perl process grew to 46,696K and stayed there.
This looks like a standard leaker to me (although I could not describe
to you the exact steps in which resources are being leaked), but it is,
in principle, similar to the following C code:

#include <stdlib.h>

void leak(void) {
char *p;
while( 1 ) {
p = malloc(1);
}
}

It is not similar in principle to that, because in principle Perl has a
garbage collector and C doesn't. I don't see how this behavior can be
anything other than a bug.

Xho

 

[A. Sinan Unur]

It is not similar in principle to that, because in principle Perl
has a garbage collector and C doesn't.

OK, that was a bad diagnosis and bad analogy.

Sinan

 

[xhoster]

OK, that was a bad diagnosis and bad analogy.

Nah, I think it was a good diagnosis and a bad analogy.

On linux, too, the in-the-loop "my" makes it much more memory efficient.
I just can't figure out why.

Xho

 

[Big and Blue]

On linux, too, the in-the-loop "my" makes it much more memory efficient.
I just can't figure out why.

Adding:

$th_n = undef;

after the detach also stops it gathering memory. This implies that what
your original code did was to continually add threads to $th_n. It did
detach each one after using it, but the documentation for detach doesn't
mention anything about destroying it, or its resources. It implies that
what happens is that the thread is just detached from its parent - a bit
like a fork() - but still exists, albeit unusefully in the example.

Specfically removing $th_n, or implicilty removing it by declaring it
with my in the loop, does destroy all threads.

Presumably it is possible to detach threads and let them continue to run?

 

[xhoster]

Adding:

$th_n = undef;

after the detach also stops it gathering memory. This implies that what
your original code did was to continually add threads to $th_n.

Yes and no. Any given instance of the variable $th_n can only hold one
thread at a time, but there are many instances of $th_n.

Each thread, when it is spawned, inherits a copy of the parents $th_n. For
all spawned threads other than the first, this variable happens to hold a
ref to the previously spawned thread. Apparently, a detached thread
doesn't clean itself up until both of two things happen: it finishes, and
it's own refcount goes to zero. (presumably, a running thread holds a
reference to itself, so its refcount can't go to zero until it finishes).
Without the undef (or "my" inside the loop) the refcount never goes to zero
because there is always some other thread holding a reference to this one.

I would think that once a thread is done, it would decrement the ref counts
of all things it references. But apparently it doesn't do this upon
finishing, but rather upon destruction. I guess this isn't a bug, as I
originally thought, but simply unfortunate.


....

Specfically removing $th_n, or implicilty removing it by declaring it
with my in the loop, does destroy all threads.

It allows the refcount to go to zero, so that it can be destroyed naturally
when its time comes.

Presumably it is possible to detach threads and let them continue to
run?

Indeed, that is the whole point of detach.

Xho