Threads preventing garbage collection?

Discussion in 'Ruby' started by Brian Candler, Sep 17, 2008.

  1. I wondered if someone can explain the following behaviour to me.

    For some reason, in the following code the instances of object Foo are
    prevented from being garbage collected.

    def make_thread
    Thread.new { sleep 2 }
    end

    class Foo
    attr_accessor :bar
    end

    10.times { |i|
    f = Foo.new
    f.bar = make_thread
    }

    GC.start
    sleep 1
    GC.start

    ObjectSpace.each_object(Foo) { |o| p o }

    But if I modify the code as follows, then the Foo objects are
    garbage-collected just fine:

    --- tst.rb 2008-09-17 10:00:27.000000000 +0100
    +++ tst2.rb 2008-09-17 10:03:04.000000000 +0100
    @@ -6,9 +6,10 @@
    attr_accessor :bar
    end

    + threads = (0..9).collect { make_thread }
    10.times { |i|
    f = Foo.new
    - f.bar = make_thread
    + f.bar = threads
    }

    GC.start

    It's as if the thread created by Thread.new keeps a reference to the Foo
    object which existed at the time, even though inside make_thread it is
    out of scope, so it shouldn't even know about it.

    By tweaking the 'sleep' values, it also seems that when the thread
    terminates it then does permit the Foo instance to be garbage-collected.

    Any ideas as to what's going on? I get the same results on these two
    versions of Ruby:

    ruby 1.8.4 (2005-12-24) [i486-linux]
    ruby 1.8.6 (2008-03-03 patchlevel 114) [i686-linux]

    Thanks,

    Brian.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Sep 17, 2008
    #1
    1. Advertising

  2. 2008/9/17 Brian Candler <>:
    > I wondered if someone can explain the following behaviour to me.


    There is no guarantee that all collectible objects are indeed
    collected during a GC run. GC is not really deterministic - at least
    from the user's point of view.

    > For some reason, in the following code the instances of object Foo are
    > prevented from being garbage collected.


    No, they are not collected. This is something different!

    Kind regards

    robert

    --
    use.inject do |as, often| as.you_can - without end
    Robert Klemme, Sep 17, 2008
    #2
    1. Advertising

  3. Robert Klemme wrote:
    >> For some reason, in the following code the instances of object Foo are
    >> prevented from being garbage collected.

    >
    > No, they are not collected. This is something different!


    OK, please consider my question rephrased as "why are these objects not
    collected?"

    Let me increase the number of objects:

    def make_thread
    Thread.new { sleep 10000 }
    end

    class Foo
    attr_accessor :bar
    end

    10000.times { |i|
    f = Foo.new
    f.bar = make_thread
    }

    GC.start
    sleep 1
    GC.start

    count = 0
    ObjectSpace.each_object(Foo) { |o| count += 1 }
    puts "#{count} objects"

    On my machine, this program uses about 150MB of RAM, and all 10,000
    objects are remaining at the end of the run.

    Have I done something here which is "wrong"? Are the known pitfalls of
    the garbage collector documented anywhere, or ways to write programs so
    as to avoid them? Or can it simply not be relied upon, ever?

    Regards,

    Brian.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Sep 17, 2008
    #3
  4. 2008/9/17 Brian Candler <>:
    > Robert Klemme wrote:
    >>> For some reason, in the following code the instances of object Foo are
    >>> prevented from being garbage collected.

    >>
    >> No, they are not collected. This is something different!

    >
    > OK, please consider my question rephrased as "why are these objects not
    > collected?"
    >
    > Let me increase the number of objects:
    >
    > def make_thread
    > Thread.new { sleep 10000 }
    > end
    >
    > class Foo
    > attr_accessor :bar
    > end
    >
    > 10000.times { |i|
    > f = Foo.new
    > f.bar = make_thread
    > }
    >
    > GC.start
    > sleep 1
    > GC.start
    >
    > count = 0
    > ObjectSpace.each_object(Foo) { |o| count += 1 }
    > puts "#{count} objects"
    >
    > On my machine, this program uses about 150MB of RAM, and all 10,000
    > objects are remaining at the end of the run.
    >
    > Have I done something here which is "wrong"? Are the known pitfalls of
    > the garbage collector documented anywhere, or ways to write programs so
    > as to avoid them?


    You probably just did not allocate enough new objects to make the GC
    collect stuff. AFAIK there is also a minimal process size under which
    no GC occurs.

    > Or can it simply not be relied upon, ever?


    Yes, it can. Try this

    def make_thread
    Thread.new { sleep 2 }
    end

    class Module
    def count
    c = 0
    ObjectSpace.each_object(self) { c += 1 }
    c
    end
    end

    Foo = Struct.new :bar

    1000.times {
    Foo.new.bar = make_thread
    }

    # use memory
    Thread.new { loop { Array.new(1_000_000); sleep 0.1 } }

    loop do
    puts Foo.count
    GC.start
    sleep 1
    end

    Cheers

    robert



    --
    use.inject do |as, often| as.you_can - without end
    Robert Klemme, Sep 17, 2008
    #4
  5. Your version only works because the threads are dying after 2 seconds.
    Change

    def make_thread
    Thread.new { sleep 2 }
    end

    to

    def make_thread
    Thread.new { sleep 10000 }
    end

    and run it again.

    The big arrays are happily garbage-collected; the Foos are not.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Sep 17, 2008
    #5
  6. Brian Candler

    ara.t.howard Guest

    On Sep 17, 2008, at 8:49 AM, Brian Candler wrote:

    > Your version only works because the threads are dying after 2 seconds.
    > Change
    >
    > def make_thread
    > Thread.new { sleep 2 }
    > end
    >
    > to
    >
    > def make_thread
    > Thread.new { sleep 10000 }
    > end
    >
    > and run it again.
    >
    > The big arrays are happily garbage-collected; the Foos are not.
    > --
    > Posted via http://www.ruby-forum.com/.
    >



    but that it expected? you have a Foo which referes to a Thread which
    has not died that is itself referred to by the global Thread.list

    cfp:~ > ruby -e' 41.times{ Thread.new{ sleep } }; p Thread.list.size '
    42


    so if they when the threads die, the Foos are freed


    cfp:~ > cat a.rb
    require 'yaml'

    def make_thread
    Thread.new { sleep 2 }
    end

    class Foo
    attr_accessor :bar
    end


    n = Integer(ARGV.first || 1024)

    3.times do
    info = {}

    n.times { |i|
    f = Foo.new
    f.bar = make_thread
    }

    info.update 'before' => ObjectSpace.each_object(Foo){}

    sleep 2.42
    GC.start

    info.update 'after' => ObjectSpace.each_object(Foo){}

    y info
    end


    cfp:~ > ruby a.rb
    ---
    after: 2
    before: 1024
    ---
    after: 2
    before: 1026
    ---
    after: 2
    before: 1026




    a @ http://codeforpeople.com/
    --
    we can deny everything, except that we have the possibility of being
    better. simply reflect on that.
    h.h. the 14th dalai lama
    ara.t.howard, Sep 17, 2008
    #6
  7. On Wed, Sep 17, 2008 at 10:07 AM, Brian Candler <> wrote:
    > I wondered if someone can explain the following behaviour to me.
    >
    > For some reason, in the following code the instances of object Foo are
    > prevented from being garbage collected.
    >
    > def make_thread
    > Thread.new { sleep 2 }
    > end
    >
    > class Foo
    > attr_accessor :bar
    > end
    >
    > 10.times { |i|
    > f = Foo.new
    > f.bar = make_thread
    > }
    >
    > GC.start
    > sleep 1
    > GC.start
    >
    > ObjectSpace.each_object(Foo) { |o| p o }
    >
    > But if I modify the code as follows, then the Foo objects are
    > garbage-collected just fine:
    >
    > --- tst.rb 2008-09-17 10:00:27.000000000 +0100
    > +++ tst2.rb 2008-09-17 10:03:04.000000000 +0100
    > @@ -6,9 +6,10 @@
    > attr_accessor :bar
    > end
    >
    > + threads = (0..9).collect { make_thread }
    > 10.times { |i|
    > f = Foo.new
    > - f.bar = make_thread
    > + f.bar = threads
    > }
    >
    > GC.start
    >
    > It's as if the thread created by Thread.new keeps a reference to the Foo
    > object which existed at the time, even though inside make_thread it is
    > out of scope, so it shouldn't even know about it.
    >
    > By tweaking the 'sleep' values, it also seems that when the thread
    > terminates it then does permit the Foo instance to be garbage-collected.
    >
    > Any ideas as to what's going on? I get the same results on these two
    > versions of Ruby:
    >
    > ruby 1.8.4 (2005-12-24) [i486-linux]
    > ruby 1.8.6 (2008-03-03 patchlevel 114) [i686-linux]
    >
    > Thanks,
    >
    > Brian.
    > --


    The threads are not garbage collected until they terminate and so the
    Foo instances are not being GC'd. You're not sleeping long enough at
    the end of your script. Try using something like:

    GC.start
    sleep 20
    GC.start

    and see how many instances are still hanging around at the end of your script.

    Regards,
    Sean
    Sean O'Halpin, Sep 17, 2008
    #7
  8. 2008/9/17 ara.t.howard <>:
    >
    > On Sep 17, 2008, at 8:49 AM, Brian Candler wrote:
    >
    >> Your version only works because the threads are dying after 2 seconds.
    >> Change
    >>
    >> def make_thread
    >> Thread.new { sleep 2 }
    >> end
    >>
    >> to
    >>
    >> def make_thread
    >> Thread.new { sleep 10000 }
    >> end
    >>
    >> and run it again.


    Good point!

    > but that it expected? you have a Foo which referes to a Thread which has
    > not died that is itself referred to by the global Thread.list


    Yeah, but there is no reference back to the Foos so they could be collected.

    If you change the code to

    def make_thread
    Thread.new { loop { sleep 2 } }
    end

    ...

    1000.times {
    Foo.new.bar = "x" * rand(100) # make_thread
    }

    Then Foo instances are quickly removed although 1 is still referenced.
    The only explanation I have ATM is that the thread stack might be
    copied somehow and thus keep the ref alive. Interesting enough ruby19
    behaves much differently: the count goes down to 0 instead of 1. But
    with the old version (i.e. Foo.new.bar = make_thread) I get thread
    creation errors. With some changing the number actually goes down to
    0:

    17:43:31 Temp$ cat gc.rb

    def make_thread
    Thread.new { loop { sleep 2 } }
    # Thread.new { loop { Array.new(1_000_000); sleep 0.1 } }
    end

    class Module
    def count
    c = 0
    ObjectSpace.each_object(self) { c += 1 }
    c
    end
    end

    Foo = Struct.new :bar

    10.times {
    # Foo.new.bar = "x" * rand(100)
    Foo.new.bar = make_thread
    }

    puts "Threads created"

    # use memory
    Thread.new { loop { Array.new(1_000_000); sleep 0.1 } }

    loop do
    puts Foo.count
    GC.start
    sleep 1
    end

    17:43:37 Temp$ ruby19 --version
    ruby 1.9.0 (2008-03-01 revision 15664) [i386-cygwin]
    17:43:48 Temp$ ruby19 gc.rb
    Threads created
    10
    0
    0
    0
    0
    0
    0
    0
    0
    0
    17:44:01 Temp$

    Kind regards

    robert

    --
    use.inject do |as, often| as.you_can - without end
    Robert Klemme, Sep 17, 2008
    #8
  9. Ara Howard wrote:
    > but that it expected? you have a Foo which referes to a Thread which
    > has not died that is itself referred to by the global Thread.list
    >
    > cfp:~ > ruby -e' 41.times{ Thread.new{ sleep } }; p Thread.list.size '
    > 42


    Foo ------------>
    thread
    thread list ---->

    I don't think this should prevent garbage collection of Foo, if nothing
    is holding a reference to Foo
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Sep 17, 2008
    #9
  10. Brian Candler

    ara.t.howard Guest

    On Sep 17, 2008, at 10:21 AM, Brian Candler wrote:

    > Ara Howard wrote:
    >> but that it expected? you have a Foo which referes to a Thread which
    >> has not died that is itself referred to by the global Thread.list
    >>
    >> cfp:~ > ruby -e' 41.times{ Thread.new{ sleep } }; p
    >> Thread.list.size '
    >> 42

    >
    > Foo ------------>
    > thread
    > thread list ---->
    >
    > I don't think this should prevent garbage collection of Foo, if
    > nothing
    > is holding a reference to Foo
    > --
    > Posted via http://www.ruby-forum.com/.
    >


    but threads *do* hold their context


    void
    rb_thread_schedule(void)
    {
    thread_debug("rb_thread_schedule\n");
    if (!rb_thread_alone()) {
    rb_thread_t *th = GET_THREAD();

    thread_debug("rb_thread_schedule/switch start\n");


    rb_gc_save_machine_context
    (th); // <-- global state
    native_mutex_unlock(&th->vm->global_interpreter_lock);
    {
    native_thread_yield();
    }
    native_mutex_lock(&th->vm->global_interpreter_lock);

    rb_thread_set_current(th);
    thread_debug("rb_thread_schedule/switch done\n");

    RUBY_VM_CHECK_INTS();
    }
    }












    a @ http://codeforpeople.com/
    --
    we can deny everything, except that we have the possibility of being
    better. simply reflect on that.
    h.h. the 14th dalai lama
    ara.t.howard, Sep 17, 2008
    #10
  11. > but threads *do* hold their context

    Could you define "context" in this, erm, context? :)

    I thought that 'def' started a new scope/context. If a thread is started
    from within def, how could it know about objects which are only
    referenced from outside that scope?

    It's as if the thread is holding on to bits of stack or bindings which
    really belong to another thread.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Sep 17, 2008
    #11
  12. Brian Candler

    ara.t.howard Guest

    On Sep 17, 2008, at 3:24 PM, Brian Candler wrote:

    > Could you define "context" in this, erm, context? :)
    >
    > I thought that 'def' started a new scope/context. If a thread is
    > started
    > from within def, how could it know about objects which are only
    > referenced from outside that scope?


    nope - we'll have to wait for matz! ;-)


    >
    >
    > It's as if the thread is holding on to bits of stack or bindings which
    > really belong to another thread.




    yeah - it certainly looks *exactly* that way - push stack and all...


    a @ http://codeforpeople.com/
    --
    we can deny everything, except that we have the possibility of being
    better. simply reflect on that.
    h.h. the 14th dalai lama
    ara.t.howard, Sep 18, 2008
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kevin Jackson
    Replies:
    1
    Views:
    412
    Alvin Bruney [MVP]
    May 28, 2004
  2. Andrew Mallinson

    Threads and Garbage Collection

    Andrew Mallinson, Jan 30, 2004, in forum: Java
    Replies:
    2
    Views:
    349
    Andrew Mallinson
    Jan 30, 2004
  3. Shirish Rai

    Garbage Collection and Threads

    Shirish Rai, Jun 9, 2004, in forum: Java
    Replies:
    3
    Views:
    345
    Chris Smith
    Jun 9, 2004
  4. Replies:
    8
    Views:
    9,633
    DrChaos
    Nov 17, 2005
  5. Øyvind Isaksen
    Replies:
    1
    Views:
    964
    Øyvind Isaksen
    May 18, 2007
Loading...

Share This Page