Manual Memory Management and Automatic Garbage Collection

Discussion in 'Ruby' started by Tridib Bandopadhyay, Dec 6, 2010.

  1. Hello all

    I am trying to do a research on Manual Memory Management and Automatic
    Garbage Collection on Ruby.So i need to play with Garbage Collector code
    of Ruby.I am using Ruby 1.8.6

    I am also trying to figure out in Ruby when the Garbage Collector starts
    working.But didn't got any clue.

    I read somewhere in wiki about some systems that uses this design of
    Memory Allocation.

    Can any one help me in some of my queries:-

    1. When the Garbage Collector starts working can I see it.(For e.g-It
    will print like Garbage Collector starting.)Somebody referred me use
    GC_NOTIFY but there is no such function in the gc.c file.

    2. What are the main parts in Ruby Garbage Collector I need to know to
    accomplish this research.

    After I get my first step done I hope I will have many questions to
    share with you all.

    Regards...Eagerly waiting for a reply at the earliest

    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 6, 2010
    #1
    1. Advertising

  2. Tridib Bandopadhyay

    Jeremy Bopp Guest

    On 12/6/2010 8:41 AM, Tridib Bandopadhyay wrote:
    > Hello all
    >
    > I am trying to do a research on Manual Memory Management and Automatic
    > Garbage Collection on Ruby.So i need to play with Garbage Collector code
    > of Ruby.I am using Ruby 1.8.6
    >
    > I am also trying to figure out in Ruby when the Garbage Collector starts
    > working.But didn't got any clue.
    >
    > I read somewhere in wiki about some systems that uses this design of
    > Memory Allocation.
    >
    > Can any one help me in some of my queries:-
    >
    > 1. When the Garbage Collector starts working can I see it.(For e.g-It
    > will print like Garbage Collector starting.)Somebody referred me use
    > GC_NOTIFY but there is no such function in the gc.c file.
    >
    > 2. What are the main parts in Ruby Garbage Collector I need to know to
    > accomplish this research.
    >
    > After I get my first step done I hope I will have many questions to
    > share with you all.
    >
    > Regards...Eagerly waiting for a reply at the earliest


    Have you tried sprinkling some printf statements throughout gc.c
    yourself yet? If the diagnostic instrumentation for gc.c is hard to
    find or does not exist, adding your own can at least help you make some
    progress.

    To start, you could add a printf with the function name at the beginning
    of each function in gc.c. Then run your build of ruby with your test
    scripts to trigger garbage collection and see what gets printed. That
    should give you a lead on where to add more printf calls to gc.c in
    order to gather more detailed information.

    -Jeremy
     
    Jeremy Bopp, Dec 6, 2010
    #2
    1. Advertising

  3. On Mon, Dec 6, 2010 at 3:41 PM, Tridib Bandopadhyay <> wrote:

    > I am trying to do a research on Manual Memory Management and Automatic
    > Garbage Collection on Ruby.So i need to play with Garbage Collector code
    > of Ruby.I am using Ruby 1.8.6
    >
    > I am also trying to figure out in Ruby when the Garbage Collector starts
    > working.But didn't got any clue.


    There were some threads here recently about GC. What specific
    questions were left unanswered?

    > I read somewhere in wiki about some systems that uses this design of
    > Memory Allocation.
    >
    > Can any one help me in some of my queries:-
    >
    > 1. When the Garbage Collector starts working can I see it.(For e.g-It
    > will print like Garbage Collector starting.)Somebody referred me use
    > GC_NOTIFY but there is no such function in the gc.c file.


    You know CPP (C preprocessor) defines and macros, do you? Please look
    for "GC_NOTIFY".

    > 2. What are the main parts in Ruby Garbage Collector I need to know to
    > accomplish this research.


    I haven't looked closely but I would start at the end of the file and
    walk my way through reading from rb_gc_start(). You should hit all
    main structures pretty soon.

    > After I get my first step done I hope I will have many questions to
    > share with you all.


    OK, let's hear them.

    > Regards...Eagerly waiting for a reply at the earliest


    Please don't push it. It won't increase your chances of getting answers anyway.
    http://www.catb.org/~esr/faqs/smart-questions.html#urgent

    Cheers

    robert

    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
     
    Robert Klemme, Dec 6, 2010
    #3
  4. Tridib Bandopadhyay

    Ammar Ali Guest

    On Mon, Dec 6, 2010 at 4:41 PM, Tridib Bandopadhyay <> wrote:
    > Hello all
    >
    > 1. When the Garbage Collector starts working can I see it.(For e.g-It
    > will print like Garbage Collector starting.)Somebody referred me use
    > GC_NOTIFY but there is no such function in the gc.c file.


    The GC_NOTIFY flag is not available in ruby 1.8, it's in 1.9 only. If
    you have to use ruby 1.8, you can do it yourself by adding printf
    calls at the begining an end of the rb_gc_start function in gc.c.
    Under 1.9 the function where GC start is called gc_collect, or at
    least that's where GC_NOTIFY is placed.

    > 2. What are the main parts in Ruby Garbage Collector I need to know to
    > accomplish this research.


    I think this was suggested before, it will answer many of your questions.

    http://timetobleed.com/garbage-collection-slides-from-la-ruby-conference/

    Regards,
    Ammar
     
    Ammar Ali, Dec 6, 2010
    #4
  5. Jeremy Bopp wrote in post #966541:

    > Have you tried sprinkling some printf statements throughout gc.c
    > yourself yet? If the diagnostic instrumentation for gc.c is hard to
    > find or does not exist, adding your own can at least help you make some
    > progress.
    >
    > To start, you could add a printf with the function name at the beginning
    > of each function in gc.c. Then run your build of ruby with your test
    > scripts to trigger garbage collection and see what gets printed. That
    > should give you a lead on where to add more printf calls to gc.c in
    > order to gather more detailed information.
    >
    > -Jeremy



    Yes I have tried that long back.. for example in this code fragment--

    VALUE
    rb_newobj()
    {
    VALUE obj;
    printf("newobj"); /*My own line*/
    if (!freelist) garbage_collect();

    obj = (VALUE)freelist;
    freelist = freelist->as.free.next;
    MEMZERO((void*)obj, RVALUE, 1);
    #ifdef GC_DEBUG
    RANY(obj)->file = ruby_sourcefile;
    RANY(obj)->line = ruby_sourceline;
    #endif
    return obj;
    }

    But when I am making this change and trying to compile my ruby. It is
    bombing out with a message Segmentation failure and it is printing my
    print statement numerous times before bombing out.I also tried in my
    other functions.

    Also can you tell me when we compile the ruby with 'make' command how
    does it changes the gc.c file in the executable file.

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 7, 2010
    #5
  6. Ammar Ali wrote in post #966575:

    > The GC_NOTIFY flag is not available in ruby 1.8, it's in 1.9 only. If
    > you have to use ruby 1.8, you can do it yourself by adding printf
    > calls at the begining an end of the rb_gc_start function in gc.c.
    > Under 1.9 the function where GC start is called gc_collect, or at
    > least that's where GC_NOTIFY is placed.
    >


    > Regards,
    > Ammar



    I tried in there also.Here is the code.--


    rb_gc_start()
    {
    rb_gc();
    return Qnil;
    printf("Hello!!")
    }

    It is compiling but is not showing the hello message when i am running
    the code.Here is my ruby code--

    print "Please enter number 1 : ";
    val1 = gets;
    print "Please enter number 2 : ";
    val2 = gets;
    print "Answer : " , (val1.to_i + val2.to_i), "\n";
    ct = 1;
    while(ct<100)
    a=[];
    na = 0;
    while(na<1000)
    a[na]=na;
    na = na+1;
    end;
    print ct;
    ct=ct+1;
    end;

    How to see that hello message and how to know when Garbage Collector is
    doing its work.

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 7, 2010
    #6
  7. On One function void rb_gc() I added one printf statement of my own.

    And when I type the command -

    /ruby -e 'GC.start()'

    Then it is showing me my print statement.But I want to see my print
    statement when i am running one of my ruby codes.

    Regards

    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 7, 2010
    #7
  8. Tridib Bandopadhyay

    Ammar Ali Guest

    On Tue, Dec 7, 2010 at 4:27 AM, Tridib Bandopadhyay <> wr=
    ote:
    >
    > I tried in there also.Here is the code.--
    >
    >
    > rb_gc_start()
    > {
    > =C2=A0 =C2=A0rb_gc();
    > =C2=A0 =C2=A0return Qnil;
    > =C2=A0 =C2=A0printf("Hello!!")
    > }


    It will never work. You're printing after return!

    Regards,
    Ammar
     
    Ammar Ali, Dec 7, 2010
    #8
  9. Ammar Ali wrote in post #966734:
    > On Tue, Dec 7, 2010 at 4:27 AM, Tridib Bandopadhyay <=
    >
    > wrote:
    >>
    >> I tried in there also.Here is the code.--
    >>
    >>
    >> rb_gc_start()
    >> {
    >> =C2=A0 =C2=A0rb_gc();
    >> =C2=A0 =C2=A0return Qnil;
    >> =C2=A0 =C2=A0printf("Hello!!")
    >> }

    >
    > It will never work. You're printing after return!
    >
    > Regards,
    > Ammar


    I shifted the printf statement above the return statement.Here is the =

    output I am getting--

    /ruby doug.rb
    Please enter number 1 : 3
    Please enter number 2 : 4
    Answer : 7
    1234567891011121314151617181920212223242526272829303132333435363738394041=
    4243444546474849505152535455565758596061626364656667686970717273747576777=
    8798081828384858687888990919293949596979899-bash-3.00$

    Regards
    Tridib

    -- =

    Posted via http://www.ruby-forum.com/.=
     
    Tridib Bandopadhyay, Dec 7, 2010
    #9
  10. Tridib Bandopadhyay

    Walton Hoops Guest

    On 12/06/2010 08:48 PM, Tridib Bandopadhyay wrote:
    > I shifted the printf statement above the return statement.Here is the
    > output I am getting--
    >
    > ./ruby doug.rb
    > Please enter number 1 : 3
    > Please enter number 2 : 4
    > Answer : 7
    > 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899-bash-3.00$

    Most likely Garbage Collection is not being run for your (rather short)
    script. Does your message print when you run GC.start()? If so your
    message is working fine. Try writing something that creates a log of
    garbage to be collected. I think something like:

    i=1
    while (true)
    puts "message #{i}"
    i+=1
    end

    should do the trick. Of course, that particular program will never exit.
     
    Walton Hoops, Dec 7, 2010
    #10
  11. Walton Hoops wrote in post #966746:
    123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899-bash-3.00$
    > Most likely Garbage Collection is not being run for your (rather short)
    > script. Does your message print when you run GC.start()? If so your
    > message is working fine. Try writing something that creates a log of
    > garbage to be collected. I think something like:
    >
    > i=1
    > while (true)
    > puts "message #{i}"
    > i+=1
    > end
    >
    > should do the trick. Of course, that particular program will never
    > exit.


    Yes When I can see my message when I run my GC.start().So When does the
    garbage collector comes into play?.When we call them or isn't it
    automatic after each object declaration it will always come for
    cleaning.

    But can't I see the message when I am running a program of my own.

    Secondly how does the make command do interpret the gc.c file into
    *.ext?Is there any mechanism?.

    Regards

    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 7, 2010
    #11
  12. Tridib Bandopadhyay

    Jeremy Bopp Guest

    On 12/06/2010 08:19 PM, Tridib Bandopadhyay wrote:
    > Jeremy Bopp wrote in post #966541:
    >
    >> Have you tried sprinkling some printf statements throughout gc.c
    >> yourself yet? If the diagnostic instrumentation for gc.c is hard to
    >> find or does not exist, adding your own can at least help you make some
    >> progress.
    >>
    >> To start, you could add a printf with the function name at the beginning
    >> of each function in gc.c. Then run your build of ruby with your test
    >> scripts to trigger garbage collection and see what gets printed. That
    >> should give you a lead on where to add more printf calls to gc.c in
    >> order to gather more detailed information.
    >>
    >> -Jeremy

    >
    >
    > Yes I have tried that long back.. for example in this code fragment--
    >
    > VALUE
    > rb_newobj()
    > {
    > VALUE obj;
    > printf("newobj"); /*My own line*/
    > if (!freelist) garbage_collect();
    >
    > obj = (VALUE)freelist;
    > freelist = freelist->as.free.next;
    > MEMZERO((void*)obj, RVALUE, 1);
    > #ifdef GC_DEBUG
    > RANY(obj)->file = ruby_sourcefile;
    > RANY(obj)->line = ruby_sourceline;
    > #endif
    > return obj;
    > }
    >
    > But when I am making this change and trying to compile my ruby. It is
    > bombing out with a message Segmentation failure and it is printing my
    > print statement numerous times before bombing out.I also tried in my
    > other functions.


    Ruby creates plenty of objects during its normal operation, so this
    function will be called frequently. It makes sense then that you'll see
    your output many times. What doesn't make sense is the segmentation fault.

    I would bet that you had more changes than just that single line in gc.c
    when you built your modified version of Ruby. If that's the case, it's
    likely that one of your other changes is the culprit. I suggest that
    you try again from a completely fresh set of sources and add only that
    single line to see if you can avoid the problem.

    > Also can you tell me when we compile the ruby with 'make' command how
    > does it changes the gc.c file in the executable file.


    I'm not sure I understand what you're asking unless you're really asking
    for the details of how the compiler works. Generally, the compiler is
    used to compile each source file into a corresponding object (*.o) file.
    Once all object files are created, they are then linked into the ruby
    executable. In this case, gc.c would be compiled into gc.o which would
    ultimately be linked into ruby.

    The make program is usually smart enough to only rebuild the .o files
    whose corresponding .c files were updated since the last build run, so
    if you only change gc.c, only gc.c should be recompiled. Then the ruby
    program should be recreated by linking the new gc.o file in along with
    all the other .o files.

    None of that really matters for you though. All you need to do is make
    your changes to gc.c and then run make. You should see some output
    during the make process that indicates that gc.c is being compiled and
    that ultimately a new ruby file is created. Try not to get sidetracked
    on the details.

    -Jeremy
     
    Jeremy Bopp, Dec 7, 2010
    #12
  13. Tridib Bandopadhyay

    Jeremy Bopp Guest

    On 12/06/2010 10:54 PM, Tridib Bandopadhyay wrote:
    > Yes When I can see my message when I run my GC.start().So When does the
    > garbage collector comes into play?.When we call them or isn't it
    > automatic after each object declaration it will always come for
    > cleaning.


    Garbage collection is expensive, so the garbage collector's operation is
    usually postponed until there is a large enough unit of work to warrant
    its cost of operation. The nice thing about this is that the GC doesn't
    have to run at all for sufficiently simple scripts. This will make such
    scripts much faster than if you run the GC as a result of every object's
    allocation.

    > But can't I see the message when I am running a program of my own.


    If you want to see the GC run on its own, you need to cause enough
    objects and memory to be allocated to trigger it. I can't personally
    say what that trigger is, but it's probably a safe bet that allocating
    some large number of objects is one such trigger. Walton's example does
    that by creating a new string in each iteration of the while loop.

    > Secondly how does the make command do interpret the gc.c file into
    > *.ext?Is there any mechanism?.


    I covered this to some extent in another response in this thread.
    Obviously, there is a mechanism or you wouldn't get a ruby program to
    run with your scripts in the first place. ;-) The C compiler is used
    under the covers, and the make program automates the process.

    Why do you ask? Do you suspect that your changes are not making it into
    your ruby program for some reason?

    From the sound of things, you don't really have much of a grasp on the
    theory of garbage collection. You might find your task much easier if
    you go read up more on the general topic and perhaps some discussions of
    particular implementations. You should find plenty of that information
    if you search around for Java garbage collection for instance. Without
    more of a background in the theory, it's going to take you much more
    time to understand Ruby's implementation.

    -Jeremy
     
    Jeremy Bopp, Dec 7, 2010
    #13
  14. Thank You all for your responses...I am getting a clear view of the Ruby
    garbage collector and the compiler...

    Regards

    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 7, 2010
    #14
  15. I tried this code fragment..

    i=1
    while (true)
    puts "message #{i}"
    i+=1
    end

    But this one is ending with a message Infinity but I can't see my own
    print statement when the Garbage Collector is coming...

    Regards

    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 9, 2010
    #15
  16. I can see the garbage collector coming when I type.

    /ruby -e 'GC.start()'

    But I want to know while a code is running the printf statement written
    by me should appear.

    Is there any other code to make gc show my message.

    This code is running for infinite time..but not showing the message.


    i=1
    while (true)
    puts "message #{i}"
    i+=1
    end

    What are the other ways?

    I need to convince myself that when there is memory allocation the
    Garbage collector comes for cleaning with a printf statement stating its
    work.

    Regards
    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 10, 2010
    #16
  17. Tridib Bandopadhyay

    Jeremy Bopp Guest

    On 12/10/2010 09:37 AM, Tridib Bandopadhyay wrote:
    > I can see the garbage collector coming when I type.
    >
    > ./ruby -e 'GC.start()'
    >
    > But I want to know while a code is running the printf statement written
    > by me should appear.
    >
    > Is there any other code to make gc show my message.
    >
    > This code is running for infinite time..but not showing the message.
    >
    >
    > i=1
    > while (true)
    > puts "message #{i}"
    > i+=1
    > end
    >
    > What are the other ways?
    >
    > I need to convince myself that when there is memory allocation the
    > Garbage collector comes for cleaning with a printf statement stating its
    > work.


    Maybe there are optimizations for String objects that circumvent parts
    of regular garbage collection. Have you tried looking at the memory
    consumption of your ruby process as you run the while loop? Is memory
    usage growing over time, or does it plateau at some point?

    Try using your ruby executable to run some non-trivial projects
    available online. Once you find one that triggers garbage collection
    the way you want it, you could try to boil that down to something more
    concise.

    Another suggestion would be to pick up a template library such as erb
    and use that to process some large template files with random data
    repeatedly. That should hopefully cause some memory growth with
    something more than simple strings.

    -Jeremy
     
    Jeremy Bopp, Dec 10, 2010
    #17
  18. Jeremy Bopp wrote in post #967674:
    > On 12/10/2010 09:37 AM, Tridib Bandopadhyay wrote:
    >>
    >> Garbage collector comes for cleaning with a printf statement stating its
    >> work.

    >
    > Maybe there are optimizations for String objects that circumvent parts
    > of regular garbage collection. Have you tried looking at the memory
    > consumption of your ruby process as you run the while loop? Is memory
    > usage growing over time, or does it plateau at some point?
    >


    The plateau is at the same point.Even if I ran the code for at-least
    half and Hour.



    >
    > Another suggestion would be to pick up a template library such as erb
    > and use that to process some large template files with random data
    > repeatedly. That should hopefully cause some memory growth with
    > something more than simple strings.
    >


    Sorry..I didn't understand this part..

    Regards

    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, Dec 10, 2010
    #18
  19. Tridib Bandopadhyay

    Jeremy Bopp Guest

    On 12/10/2010 10:58 AM, Tridib Bandopadhyay wrote:
    > Jeremy Bopp wrote in post #967674:
    >> Another suggestion would be to pick up a template library such as erb
    >> and use that to process some large template files with random data
    >> repeatedly. That should hopefully cause some memory growth with
    >> something more than simple strings.
    >>

    >
    > Sorry..I didn't understand this part..


    Use the ERB class:

    http://rdoc.info/docs/ruby-stdlib/1.8.7/ERB

    Create a handful of files to be used as templates of the form required
    for that class. Ensure that the templates have data you can replace.
    Process the templates with generated data in a loop.

    We're basically making what should hopefully be a non-trivial form of
    the while loop you've been using that prints a string with a number
    embedded. Hopefully, this will allocate a sufficient number of objects
    in a way that will eventually trigger garbage collection. If it doesn't
    work, get creative and try to figure out some other way to lots of
    non-trivial objects.

    -Jeremy
     
    Jeremy Bopp, Dec 10, 2010
    #19
  20. I have some questions::--

    1. Is it possible to Manually free memory(without calling GC.start) by
    extending it with C? As i have tried it and its of no use for my work.

    2. Can I add a new file and try to manually free memory(disabling GC)?

    Regards

    Tridib

    --
    Posted via http://www.ruby-forum.com/.
     
    Tridib Bandopadhyay, May 17, 2011
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Richard Jones
    Replies:
    0
    Views:
    367
    Richard Jones
    Apr 29, 2004
  2. Richard Jones
    Replies:
    0
    Views:
    364
    Richard Jones
    Apr 29, 2004
  3. Richard Jones
    Replies:
    0
    Views:
    251
    Richard Jones
    Apr 29, 2004
  4. Øyvind Isaksen
    Replies:
    1
    Views:
    982
    Øyvind Isaksen
    May 18, 2007
  5. Replies:
    14
    Views:
    614
    Bo Persson
    Jun 18, 2008
Loading...

Share This Page