ithreads & memory

Discussion in 'Perl Misc' started by Micha³ Lesiak (bler), Jul 18, 2005.

  1. Hello,

    A simple script:

    #!/usr/bin/perl

    use threads;
    use Time::HiRes "usleep";

    sub th {
    usleep(10000);
    }

    while(1) {
    $th_n = threads->new(\&th);
    $th_n->detach;
    usleep(100000);
    }

    and that is, every 0,1s a new thread is created, and it runs for 0,01s. So,
    when a new $th_n starts, the previous one is dead and gone - but the memory
    is not released. The script goes on until it eats up all the memory avaible
    and segfaults.

    Now, this is just to show the problem, my real application creates a couple
    of 10s threads once in a while - and it runs for months, which in due
    course ends just like the above, out of memory and segfault. I know there
    are some problems with ithreads, but in my case this means it's completely
    unusable, so I think I'm doing something wrong. Can you help me?

    perl 5.8.7 (5.8.2, 5.8.5 also been tried), kernel 2.6.11.6 (some other
    versions tested too), threads 1.05.


    --
    M.
     
    Micha³ Lesiak (bler), Jul 18, 2005
    #1
    1. Advertising

  2. Micha³ Lesiak (bler)

    Guest

    "Micha³ Lesiak (bler)" <> wrote:
    > Hello,
    >
    > A simple script:
    >
    > #!/usr/bin/perl
    >
    > use threads;
    > use Time::HiRes "usleep";
    >
    > sub th {
    > usleep(10000);
    > }
    >
    > while(1) {
    > $th_n = threads->new(\&th);
    > $th_n->detach;
    > usleep(100000);
    > }


    You should check the actual time slept by usleep. It is possible
    that the parent is waking up early and hence spawning new threads
    more often than you think.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 18, 2005
    #2
    1. Advertising

  3. wrote in news:20050718183821.191$:

    > You should check the actual time slept by usleep. It is possible
    > that the parent is waking up early and hence spawning new threads
    > more often than you think.


    It's all the same when I use "sleep 1" and "sleep 4", respectively. It just
    take longer. As I said, it may take weeks but it does happen, once the
    memory for a thread is allocated it's released only when the program quits.


    --
    M.
     
    Micha³ Lesiak (bler), Jul 19, 2005
    #3
  4. Micha³ Lesiak (bler)

    Guest

    "Micha³ Lesiak (bler)" <> wrote:
    > wrote in news:20050718183821.191$:
    >
    > > You should check the actual time slept by usleep. It is possible
    > > that the parent is waking up early and hence spawning new threads
    > > more often than you think.

    >
    > It's all the same when I use "sleep 1" and "sleep 4", respectively.


    I'm not sure that that precludes this possibility.

    > It
    > just take longer. As I said, it may take weeks but it does happen, once
    > the memory for a thread is allocated it's released only when the program
    > quits.


    I can't produce the problem on 5.8.0 with threads 0.99
    (or maybe I just am not waiting long enough).

    Are the threads hanging around holding memory, or is the memory building
    up in the parent process? (e.g. does:
    ps -efl | fgrep <progname>
    show an ever increasing number of processes, or just one or two processes
    with increasing memory use?

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 19, 2005
    #4
  5. wrote in news:20050719002327.291$:

    >> It's all the same when I use "sleep 1" and "sleep 4", respectively.

    >
    > I'm not sure that that precludes this possibility.


    Okay, consider this:

    use threads;
    use Time::HiRes "usleep";

    sub th {
    usleep(10000);
    }

    for(my $inc; $inc < 100; $inc++) {
    $th_n = threads->new(\&th);
    $th_n->detach;
    usleep(100000);
    }

    while (1) {
    sleep 1;
    }

    So now it creates 100 threads, the while loop is there to pretend that the
    script is doing some other things. After the first while execution, the
    last thread should be finished. On my system, as before, memory is still
    taken (about 43MB).

    > Are the threads hanging around holding memory, or is the memory building
    > up in the parent process? (e.g. does:
    > ps -efl | fgrep <progname>
    > show an ever increasing number of processes, or just one or two processes
    > with increasing memory use?


    There are only two processes present.

    --
    M.
     
    Micha³ Lesiak (bler), Jul 19, 2005
    #5
  6. "Micha³ Lesiak (bler)" <> wrote in
    news:Xns9698D8820EDA9michalusenetblaaireu@193.110.122.97:

    > wrote in news:20050719002327.291$:
    >
    >>> It's all the same when I use "sleep 1" and "sleep 4", respectively.

    >>
    >> I'm not sure that that precludes this possibility.

    >
    > Okay, consider this:
    >
    > use threads;
    > use
    > Time::HiRes "usleep";
    >


    Oh, what's going on? I see, you have a ton of embedded hard tabs or
    something. Please don't do that. It makes it hard on your readers.

    Why are don't you have

    use strict;
    use warnings;

    in your script?

    > for(my $inc; $inc < 100; $inc++) {
    > $th_n = threads->new(\&th);
    > $th_n->detach;
    > usleep(100000);
    > }


    Is there a reason you are using the same $th_n for all the threads over
    and over again?

    Consider the code below:

    #!/usr/bin/perl

    use strict;
    use warnings;

    use threads;
    use Time::HiRes "usleep";

    sub th { usleep 10000; print "$_[0]: Done\n" }

    my $th_n;
    for (1 .. 100) {
    $th_n = threads->new(\&th, $_);
    $th_n->detach;
    usleep 100000;
    }

    sleep 1 while 1;

    __END__

    When I ran this code on my system (Windows XP SP2, Perl 5.8.7), the
    memory footprint of the Perl process grew to 46,696K and stayed there.
    This looks like a standard leaker to me (although I could not describe
    to you the exact steps in which resources are being leaked), but it is,
    in principle, similar to the following C code:

    #include <stdlib.h>

    void leak(void) {
    char *p;
    while( 1 ) {
    p = malloc(1);
    }
    }

    Now, with one minor change, as in the following:

    #!/usr/bin/perl

    use strict;
    use warnings;

    use threads;
    use Time::HiRes "usleep";

    sub th { usleep 10000; print "$_[0]: Done\n" }

    for (1 .. 100) {
    my $th_n = threads->new(\&th, $_);
    $th_n->detach;
    usleep 100000;
    }

    sleep 1 while 1;

    __END__

    The maximum memory footprint stays at 3564K throughout the life of the
    script.

    The maxim: Always declare variables in the smallest applicable scope.

    Sinan

    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
     
    A. Sinan Unur, Jul 19, 2005
    #6
  7. Micha³ Lesiak (bler)

    Guest

    "A. Sinan Unur" <> wrote:
    >
    > Is there a reason you are using the same $th_n for all the threads over
    > and over again?
    >
    > Consider the code below:
    >
    > #!/usr/bin/perl
    >
    > use strict;
    > use warnings;
    >
    > use threads;
    > use Time::HiRes "usleep";
    >
    > sub th { usleep 10000; print "$_[0]: Done\n" }
    >
    > my $th_n;
    > for (1 .. 100) {
    > $th_n = threads->new(\&th, $_);
    > $th_n->detach;
    > usleep 100000;
    > }
    >
    > sleep 1 while 1;
    >
    > __END__
    >
    > When I ran this code on my system (Windows XP SP2, Perl 5.8.7), the
    > memory footprint of the Perl process grew to 46,696K and stayed there.
    > This looks like a standard leaker to me (although I could not describe
    > to you the exact steps in which resources are being leaked), but it is,
    > in principle, similar to the following C code:
    >
    > #include <stdlib.h>
    >
    > void leak(void) {
    > char *p;
    > while( 1 ) {
    > p = malloc(1);
    > }
    > }


    It is not similar in principle to that, because in principle Perl has a
    garbage collector and C doesn't. I don't see how this behavior can be
    anything other than a bug.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 20, 2005
    #7
  8. wrote in news:20050719190948.661$:

    > "A. Sinan Unur" <> wrote:
    >> there. This looks like a standard leaker to me (although I could not
    >> describe to you the exact steps in which resources are being leaked),
    >> but it is, in principle, similar to the following C code:
    >>
    >> #include <stdlib.h>
    >>
    >> void leak(void) {
    >> char *p;
    >> while( 1 ) {
    >> p = malloc(1);
    >> }
    >> }

    >
    > It is not similar in principle to that, because in principle Perl
    > has a garbage collector and C doesn't.


    OK, that was a bad diagnosis and bad analogy.

    Sinan

    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
     
    A. Sinan Unur, Jul 20, 2005
    #8
  9. Micha³ Lesiak (bler)

    Guest

    "A. Sinan Unur" <> wrote:
    > wrote in news:20050719190948.661$:
    >
    > > "A. Sinan Unur" <> wrote:
    > >> there. This looks like a standard leaker to me (although I could not
    > >> describe to you the exact steps in which resources are being leaked),
    > >> but it is, in principle, similar to the following C code:
    > >>
    > >> #include <stdlib.h>
    > >>
    > >> void leak(void) {
    > >> char *p;
    > >> while( 1 ) {
    > >> p = malloc(1);
    > >> }
    > >> }

    > >
    > > It is not similar in principle to that, because in principle Perl
    > > has a garbage collector and C doesn't.

    >
    > OK, that was a bad diagnosis and bad analogy.


    Nah, I think it was a good diagnosis and a bad analogy. :)

    On linux, too, the in-the-loop "my" makes it much more memory efficient.
    I just can't figure out why.

    Xho



    >
    > Sinan


    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 20, 2005
    #9
  10. Micha³ Lesiak (bler)

    Big and Blue Guest

    wrote:
    >
    > On linux, too, the in-the-loop "my" makes it much more memory efficient.
    > I just can't figure out why.


    Adding:

    $th_n = undef;

    after the detach also stops it gathering memory. This implies that what
    your original code did was to continually add threads to $th_n. It did
    detach each one after using it, but the documentation for detach doesn't
    mention anything about destroying it, or its resources. It implies that
    what happens is that the thread is just detached from its parent - a bit
    like a fork() - but still exists, albeit unusefully in the example.

    Specfically removing $th_n, or implicilty removing it by declaring it
    with my in the loop, does destroy all threads.

    Presumably it is possible to detach threads and let them continue to run?



    --
    Just because I've written it doesn't mean that
    either you or I have to believe it.
     
    Big and Blue, Jul 20, 2005
    #10
  11. Micha³ Lesiak (bler)

    Guest

    Big and Blue <> wrote:
    > wrote:
    > >
    > > On linux, too, the in-the-loop "my" makes it much more memory
    > > efficient. I just can't figure out why.

    >
    > Adding:
    >
    > $th_n = undef;
    >
    > after the detach also stops it gathering memory. This implies that what
    > your original code did was to continually add threads to $th_n.


    Yes and no. Any given instance of the variable $th_n can only hold one
    thread at a time, but there are many instances of $th_n.

    Each thread, when it is spawned, inherits a copy of the parents $th_n. For
    all spawned threads other than the first, this variable happens to hold a
    ref to the previously spawned thread. Apparently, a detached thread
    doesn't clean itself up until both of two things happen: it finishes, and
    it's own refcount goes to zero. (presumably, a running thread holds a
    reference to itself, so its refcount can't go to zero until it finishes).
    Without the undef (or "my" inside the loop) the refcount never goes to zero
    because there is always some other thread holding a reference to this one.

    I would think that once a thread is done, it would decrement the ref counts
    of all things it references. But apparently it doesn't do this upon
    finishing, but rather upon destruction. I guess this isn't a bug, as I
    originally thought, but simply unfortunate.


    ....
    >
    > Specfically removing $th_n, or implicilty removing it by declaring it
    > with my in the loop, does destroy all threads.


    It allows the refcount to go to zero, so that it can be destroyed naturally
    when its time comes.

    > Presumably it is possible to detach threads and let them continue to
    > run?


    Indeed, that is the whole point of detach.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 21, 2005
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Taewoon Kwon

    Qustion about ithreads join

    Taewoon Kwon, Sep 10, 2004, in forum: Perl
    Replies:
    1
    Views:
    440
    Joe Smith
    Sep 15, 2004
  2. David Morel

    Is the ithreads implementation safe?

    David Morel, Sep 13, 2003, in forum: Perl Misc
    Replies:
    0
    Views:
    110
    David Morel
    Sep 13, 2003
  3. Walter Roberson

    Knight's tour in perl ithreads

    Walter Roberson, Feb 2, 2004, in forum: Perl Misc
    Replies:
    0
    Views:
    140
    Walter Roberson
    Feb 2, 2004
  4. Walter Roberson

    ithreads at runtime?

    Walter Roberson, Feb 3, 2004, in forum: Perl Misc
    Replies:
    3
    Views:
    104
    Walter Roberson
    Feb 4, 2004
  5. Thomas Jahns

    ithreads + signals on modern Unices

    Thomas Jahns, Feb 7, 2005, in forum: Perl Misc
    Replies:
    4
    Views:
    128
Loading...

Share This Page