Memory profiling?

Discussion in 'Ruby' started by Scott Ellsworth, Jul 19, 2005.

  1. Hi, all.

    I note that there is a fine package for time profiling, but I did not
    see one for memory profiling.

    Did I miss one?

    NB - if you have not looked in the Java profiler space lately, it might
    be worth a glance. Virtually every profiler package out there can tell
    you how many of each object are live, how many are dead, the line that
    produced each allocation, and the amount of VM taken by each. Thus, you
    can easily find:
    The lines that allocated the most memory
    The lines that churned (allocated and freed) the most memory
    The lines that allocated the most objects
    The lines that churned the most objects
    The objects holding on to the most data

    and so on.

    This kind of stuff usually makes it very easy to discover why an app is
    taking up so much RAM, which is a very good first step when deciding to
    improve both speed and footprint.

    Scott

    --
    Scott Ellsworth

    Java and database consulting for the life sciences
     
    Scott Ellsworth, Jul 19, 2005
    #1
    1. Advertising

  2. Hi,

    In message "Re: Memory profiling?"
    on Tue, 19 Jul 2005 09:45:55 +0900, Scott Ellsworth <> writes:

    |I note that there is a fine package for time profiling, but I did not
    |see one for memory profiling.
    |
    |Did I miss one?

    I'm afraid I haven't seen one. It would be a nice idea to a memory
    profiler. If someone is willing to create one and requires my help (I
    bet he needs to tweak the source), feel free contact me.

    matz.
     
    Yukihiro Matsumoto, Jul 19, 2005
    #2
    1. Advertising

  3. >NB - if you have not looked in the Java profiler space lately, it might
    >be worth a glance. Virtually every profiler package out there can tell
    >you how many of each object are live, how many are dead, the line that
    >produced each allocation, and the amount of VM taken by each. Thus, you
    >can easily find:
    > The lines that allocated the most memory
    > The lines that churned (allocated and freed) the most memory
    > The lines that allocated the most objects
    > The lines that churned the most objects
    > The objects holding on to the most data
    >
    >and so on.
    >
    >This kind of stuff usually makes it very easy to discover why an app is
    >taking up so much RAM, which is a very good first step when deciding to
    >improve both speed and footprint.


    Ruby Memory Validator is in pre-beta at Software Verification. Currently
    it has been released to one person profiling Rails. It can do all of the
    above (except the last as Ruby is really impervious to queries like that
    from C).

    To implement RMV has involved some very intricate binary patches to
    Ruby. We know what needs to be done, but Ruby's construction really
    mitigates against doing this easily.

    I think it should be implemented as a C-only callback so that we can
    avoid creating Ruby objects as part of the callback. Additionally we'd
    need to be able to provide a means to get a Ruby callstack from inside
    the callback.

    The callback would be called for the following events:
    o Object creation. This would provide:
    VALUE - the object value
    char * - the object type
    Optional, but nice if possible - the size of the object
    o Object destruction. When cleaned up using obj_free() or
    rb_gc_force_recycle().
    o Garbage collection start.
    o Garbage collection end.

    All of the above events are useful and should be supported. Java
    supports them and as a result has many memory profilers. We have
    implemented the above for Ruby, although getting the object type from
    rb_newobj() is impossible so we've had to resort to a rather
    un-satisfactory delayed approach to type detection.

    Other very useful things that the C callable API should provide (again
    without Ruby creating more VALUEs to be put through the GC): These
    actions are required to implement the last item on the list provided by
    Scott.
    o Ability to determine all VALUEs referred to be a specific VALUE.
    o Ability to determine all GC roots.
    o Heap dump, listing all GC roots and identifying each VALUE and its
    referred values.
    The above 3 options allow the creation of very useful heap dumps showing
    the reference graph of all objects. This allows "leaks" to be detected.

    You will note I am very keen on a C callable API. The very action of
    using a Ruby callable API will create and destroy many objects and will
    invoke the Ruby gc many times (it operates very frequently!) just
    servicing on call to the Ruby API. This will distort the reports quite
    badly.

    When we have the help available we will make RMV available as a beta.
    Until then if you are interested please email betatest AT softwareverify
    D0T com if you wish to test the current implementation.

    RMV runs on Windows NT 4.0/W2K/XP or better.

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
     
    Stephen Kellett, Jul 19, 2005
    #3
  4. > RMV runs on Windows NT 4.0/W2K/XP or better.

    why this platform requirement?

    Cameron
     
    Cameron McBride, Jul 19, 2005
    #4
  5. In message <>, Cameron McBride
    <> writes
    >> RMV runs on Windows NT 4.0/W2K/XP or better.

    >
    >why this platform requirement?


    Because we don't have the time or manpower to provide a Linux port at
    this time. It is impossible to write something like RMV without using
    C++/assembler and patching the Ruby binary at runtime to get access to
    the object creation and destruction methods. You cannot do this with
    Ruby/Python/Perl/Java/etc. Even if you could they would be too slow for
    this type of thing.

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
     
    Stephen Kellett, Jul 19, 2005
    #5
  6. Hello Stephen,

    SK> In message <>, Cameron McBride
    SK> <> writes
    >>> RMV runs on Windows NT 4.0/W2K/XP or better.

    >>
    >>why this platform requirement?


    SK> Because we don't have the time or manpower to provide a Linux port at
    SK> this time. It is impossible to write something like RMV without using
    SK> C++/assembler and patching the Ruby binary at runtime to get access to
    SK> the object creation and destruction methods. You cannot do this with
    SK> Ruby/Python/Perl/Java/etc. Even if you could they would be too slow for
    SK> this type of thing.

    See you had to do the same as i did for my debugger.
    Whats about a standard "ruby_d.exe" which adds more hooks into the
    virtual engine and still works with all extensions ?


    --
    Best regards, emailto: scholz at scriptolutions dot com
    Lothar Scholz http://www.ruby-ide.com
    CTO Scriptolutions Ruby, PHP, Python IDE 's
     
    Lothar Scholz, Jul 19, 2005
    #6
  7. > >> RMV runs on Windows NT 4.0/W2K/XP or better.
    > >
    > >why this platform requirement?

    >=20
    > Because we don't have the time or manpower to provide a Linux port at
    > this time. It is impossible to write something like RMV without using
    > C++/assembler and patching the Ruby binary at runtime to get access to
    > the object creation and destruction methods. You cannot do this with
    > Ruby/Python/Perl/Java/etc. Even if you could they would be too slow for
    > this type of thing.


    Makes sense. Honestly, I was just curious. I suspected there might
    be underlying nastiness when you mentioned binary patching, but being
    ignorant of many important details it was a guess and I wanted to
    excluded the possibility of some other constraint (silly GUI or
    something).

    Thanks.

    > Whats about a standard "ruby_d.exe" which adds more hooks into the
    > virtual engine and still works with all extensions ?


    Again, I'm ignorant - but this does sound pretty cool.

    In any case, I'd really love the ability to memory profile ruby code.=20
    I've found it difficult to do with the ad hoc methods I've utilized.

    Cameron
     
    Cameron McBride, Jul 19, 2005
    #7
  8. In message <>, Lothar Scholz
    <> writes
    >SK> this time. It is impossible to write something like RMV without using
    >SK> C++/assembler and patching the Ruby binary at runtime to get access to
    >SK> the object creation and destruction methods.
    >
    >See you had to do the same as i did for my debugger.
    >Whats about a standard "ruby_d.exe" which adds more hooks into the
    >virtual engine and still works with all extensions ?


    I disagree. The standard Ruby already provides various hooks, the nature
    of which is changing slightly for V2.0. It just needs some extra hooks
    in the release version. The overhead of the check

    if (hookPtr != NULL)
    {
    // call the hook
    }

    is negligible for the case when the hook is absent (i.e. normal usage),
    especially when you compare the overhead of that test (how many cycles?
    not many) compared to the cost of doing just about anything in the Ruby
    runtime or in C).

    I had assumed you had been able to write your debugger using the
    standard Ruby API calls.

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
     
    Stephen Kellett, Jul 19, 2005
    #8
  9. In message <>, Cameron McBride
    <> writes
    >I suspected there might
    >be underlying nastiness when you mentioned binary patching, but being
    >ignorant of many important details it was a guess and I wanted to
    >excluded the possibility of some other constraint (silly GUI or
    >something).


    Well, my experience of fixing Memory problems for people extends back to
    the beta of Purify on Solaris in 90/91. That was text only back then.
    Things have moved on. You may regard the GUI interfaces on such data as
    silly, but from my perspective (and those of our customers) they are
    essential, as the requests for "can we have such and such a display"
    indicate.

    Memory problems tend to benefit from advanced GUIs and
    search/interrogate features even more than profiling/coverage/flow
    tracing. Deadlock/thread analysis also seems to benefit more.

    >In any case, I'd really love the ability to memory profile ruby code.
    >I've found it difficult to do with the ad hoc methods I've utilized.


    If you haven't got hooks into the VM the info you have is thoroughly
    misleading (regardless of OS).

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
     
    Stephen Kellett, Jul 19, 2005
    #9
  10. Scott Ellsworth

    John Carter Guest

    On Tue, 19 Jul 2005, Scott Ellsworth wrote:

    > Hi, all.
    >
    > I note that there is a fine package for time profiling, but I did not
    > see one for memory profiling.
    >
    > Did I miss one?


    There is...
    http://rubyforge.org/snippet/detail.php?type=snippet&id=70

    > NB - if you have not looked in the Java profiler space lately, it might
    > be worth a glance. Virtually every profiler package out there can tell
    > you how many of each object are live, how many are dead, the line that
    > produced each allocation, and the amount of VM taken by each. Thus, you
    > can easily find:
    > The lines that allocated the most memory
    > The lines that churned (allocated and freed) the most memory
    > The lines that allocated the most objects
    > The lines that churned the most objects
    > The objects holding on to the most data


    My memory profiler is a bit orthogonal to that. It counts objects still
    active by class and the size of them. ie. It doesn't tell you which
    line it occurred on.

    The flip side is you have to work backwards to work out the guilty line.

    Usually (but not always) that's fairly easy. If BigHairyObjects are
    gobbling your memory, just grep for "BigHairyObject.new"

    It's not a very good profiler for counting churn.

    On the other hand, it is really very very easy to use.


    John Carter Phone : (64)(3) 358 6639
    Tait Electronics Fax : (64)(3) 359 4632
    PO Box 1645 Christchurch Email :
    New Zealand

    Carter's Clarification of Murphy's Law.

    "Things only ever go right so that they may go more spectacularly wrong later."

    From this principle, all of life and physics may be deduced.
     
    John Carter, Jul 20, 2005
    #10
  11. In article <>,
    Yukihiro Matsumoto <> wrote:

    > Hi,
    >
    > In message "Re: Memory profiling?"
    > on Tue, 19 Jul 2005 09:45:55 +0900, Scott Ellsworth <>
    > writes:
    >
    > |I note that there is a fine package for time profiling, but I did not
    > |see one for memory profiling.
    > |
    > |Did I miss one?
    >
    > I'm afraid I haven't seen one. It would be a nice idea to a memory
    > profiler. If someone is willing to create one and requires my help (I
    > bet he needs to tweak the source), feel free contact me.


    I suspect a good implementation would require source tweaking, because
    trying to second guess memory allocations and the like based on the
    executable image gets really, really hard. Low impact memory profiling
    requires knowing what the GC system is doing, and having forensics for
    it. For an example, check out
    <http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/jvmti.html>
    for the rather baroque, but reasonably efficient, system current JVMs
    use for debugging tools.

    We likely do not need quite this level of instrumentation, but we might
    want to hoover the spec for ideas. If nothing else, we are not running
    a bytecode-only environment like Java, so we have both different needs
    and different tools. For example, native C code, as best as I can tell,
    is very cheap in Ruby, often very expensive in Java.

    (I say we. I am still a Ruby tyro - hopefully, someone with both time
    and more experience than I have will find this interesting enough to do.)

    Scott

    --
    Scott Ellsworth

    Java and database consulting for the life sciences
     
    Scott Ellsworth, Jul 20, 2005
    #11
  12. In message <>, Scott
    Ellsworth <> writes
    >(I say we. I am still a Ruby tyro - hopefully, someone with both time
    >and more experience than I have will find this interesting enough to do.)


    I've offered to do it myself but have yet to have any confirmation by
    Matz that he will help me deal with the few bits I either don't
    understand or wish to have changed (I have asked him).

    I'm not going to do this work if I don't know that Matz will accept it
    (or the principle of it) into Ruby. No point doing work that will be
    thrown away.

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
     
    Stephen Kellett, Jul 21, 2005
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. dayjah
    Replies:
    3
    Views:
    12,766
  2. Rune Froysa

    Memory profiling

    Rune Froysa, Sep 24, 2003, in forum: Python
    Replies:
    1
    Views:
    389
    Michael Hudson
    Sep 25, 2003
  3. Eddie

    profiling memory usage

    Eddie, Oct 6, 2006, in forum: Python
    Replies:
    3
    Views:
    355
    NOSHII
    Oct 6, 2006
  4. Ram  Prasad

    OT: Memory profiling on Linux

    Ram Prasad, May 1, 2007, in forum: C Programming
    Replies:
    5
    Views:
    403
    Giorgos Keramidas
    May 2, 2007
  5. Replies:
    9
    Views:
    335
    EventHelix.com
    May 20, 2007
Loading...

Share This Page