memory measurements

Discussion in 'Ruby' started by Eric Mahurin, Sep 23, 2005.

  1. Eric Mahurin

    Eric Mahurin Guest

    In ruby, is there a portable way to measure memory usage? Any
    of these would be useful:

    - peak
    - average
    - current
    - any of the above for just the memory that GC maintains

    It would be nice if Benchmark did this if there was a portable
    way. Many times there is a tradeoff between runtime and memory
    usage and it would be good to see this easily.



    =09
    __________________________________=20
    Yahoo! Mail - PC Magazine Editors' Choice 2005=20
    http://mail.yahoo.com
    Eric Mahurin, Sep 23, 2005
    #1
    1. Advertising

  2. In message <>,
    Eric Mahurin <> writes
    >In ruby, is there a portable way to measure memory usage? Any
    >of these would be useful:


    Not portable, but if you are using Windows you may want to take a look
    at Ruby Memory Validator. No website description at the present time but
    you can apply for the beta at http://www.softwareverify.com

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
    Stephen Kellett, Sep 23, 2005
    #2
    1. Advertising

  3. Eric Mahurin

    Eric Mahurin Guest

    I'm a linux guy. Here's what I do to get the memory of the
    current ruby process:

    IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display

    Of course this is non-portable too.

    I'd imagine to get a non-portable solution, you'd need
    something in GC/ObjectSpace to figure out how much its objects
    are taking up.

    --- Stephen Kellett <> wrote:

    > In message
    > <>,=20
    > Eric Mahurin <> writes
    > >In ruby, is there a portable way to measure memory usage?=20

    > Any
    > >of these would be useful:

    >=20
    > Not portable, but if you are using Windows you may want to
    > take a look=20
    > at Ruby Memory Validator. No website description at the
    > present time but=20
    > you can apply for the beta at http://www.softwareverify.com
    >=20
    > Stephen
    > --=20
    > Stephen Kellett
    > Object Media Limited =20
    > http://www.objmedia.demon.co.uk/software.html
    > Computer Consultancy, Software Development
    > Windows C++, Java, Assembler, Performance Analysis,
    > Troubleshooting
    >=20
    >=20



    __________________________________________________
    Do You Yahoo!?
    Tired of spam? Yahoo! Mail has the best spam protection around=20
    http://mail.yahoo.com=20
    Eric Mahurin, Sep 23, 2005
    #3
  4. In message <>,
    Eric Mahurin <> writes
    >I'd imagine to get a non-portable solution, you'd need
    >something in GC/ObjectSpace to figure out how much its objects
    >are taking up.


    That is what Ruby Memory Validator does. Plus provide lots of view and
    metrics to examine the reference graphs and so on.

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
    Stephen Kellett, Sep 23, 2005
    #4
  5. Eric Mahurin

    Eric Mahurin Guest

    --- Stephen Kellett <> wrote:

    > In message
    > <>,=20
    > Eric Mahurin <> writes
    > >I'd imagine to get a non-portable solution, you'd need
    > >something in GC/ObjectSpace to figure out how much its

    > objects
    > >are taking up.

    >=20
    > That is what Ruby Memory Validator does. Plus provide lots of
    > view and=20
    > metrics to examine the reference graphs and so on.


    I don't care about graphs and such. Just a method (or several)
    in GC/ObjectSpace to get some memory info. If the above did
    what I'm thinking about (walk through each
    object/stack/symbol-table/etc and add up the memory used), it
    would be portable, but you said it is windows only. Also, I
    couldn't find any download link.

    What would be ideal would be if the C call get_rusage worked on
    linux (it returns 0 for various memory fields) and the
    equivalent was available on various platforms. Of course you
    could also make a method that did whatever was appropriate for
    each platform too.




    =09
    __________________________________=20
    Yahoo! Mail - PC Magazine Editors' Choice 2005=20
    http://mail.yahoo.com
    Eric Mahurin, Sep 23, 2005
    #5
  6. On 9/23/05, Eric Mahurin <> wrote:
    > --- Stephen Kellett <> wrote:

    ...snipped...
    > I don't care about graphs and such. Just a method (or several)
    > in GC/ObjectSpace to get some memory info. If the above did
    > what I'm thinking about (walk through each
    > object/stack/symbol-table/etc and add up the memory used), it
    > would be portable, but you said it is windows only. Also, I
    > couldn't find any download link.


    You might find http://rubyforge.org/snippet/detail.php?type=3Dsnippet&id=3D=
    70
    does some of what you want.

    Chris
    Chris McGrath, Sep 23, 2005
    #6
  7. In message <>,
    Eric Mahurin <> writes
    >I don't care about graphs and such. Just a method (or several)
    >in GC/ObjectSpace to get some memory info. If the above did
    >what I'm thinking about (walk through each
    >object/stack/symbol-table/etc and add up the memory used), it
    >would be portable,


    If you examine the source for Ruby you'll find that the above is
    non-trivial and is implemented in C, in the form of the garbage
    collector. I have offered to write a C API to collect such stats (and
    will provide a simple Ruby API to provide other data to Ruby in the form
    of a snapshot) but have not yet had the time to do it. Is this portable?
    Yes if you regard compiling C on each platform to get the code working.

    Someone else (sorry can't remember who) did write a Ruby snippet to walk
    the object space. The main problem with this approach and any approach
    that tries to trace ruby memory usage using say a set_trace_func() type
    approach is the act of sending data to the ruby callback or walking the
    object space in ruby allocates more objects, thus distorting the
    picture, sometimes quite badly.

    >but you said it is windows only.


    It is written in C++ and assembly. It is impossible to do low level
    hooking of the type required to write this software in anything other
    than similar languages. You could not write these software tools in
    Ruby, Python, Java, Lua, etc.

    >Also, I
    >couldn't find any download link.


    I think I covered that when I wrote:

    >No website description at the present time but you can apply for the

    beta >at

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
    Stephen Kellett, Sep 23, 2005
    #7
  8. Eric Mahurin

    Eric Mahurin Guest

    --- Chris McGrath <> wrote:

    > On 9/23/05, Eric Mahurin <> wrote:
    > > --- Stephen Kellett <> wrote:

    > ...snipped...
    > > I don't care about graphs and such. Just a method (or

    > several)
    > > in GC/ObjectSpace to get some memory info. If the above

    > did
    > > what I'm thinking about (walk through each
    > > object/stack/symbol-table/etc and add up the memory used),

    > it
    > > would be portable, but you said it is windows only. Also,

    > I
    > > couldn't find any download link.

    >=20
    > You might find
    > http://rubyforge.org/snippet/detail.php?type=3Dsnippet&id=3D70
    > does some of what you want.


    It looks to gauge memory usage by counting objects. Far from
    accurate. And then it uses the same non-portable (linux only)
    method I've been using to get the real answer:
    /proc/<pid>/status.




    =09
    =09
    ______________________________________________________=20
    Yahoo! for Good=20
    Donate to the Hurricane Katrina relief effort.=20
    http://store.yahoo.com/redcross-donate3/=20
    Eric Mahurin, Sep 23, 2005
    #8
  9. In article <dNwpj3Pr+DNDFws$@objmedia.demon.co.uk>,
    Stephen Kellett <> wrote:

    > In message <>,
    > Eric Mahurin <> writes


    > >but you said it is windows only.

    >
    > It is written in C++ and assembly. It is impossible to do low level
    > hooking of the type required to write this software in anything other
    > than similar languages. You could not write these software tools in
    > Ruby, Python, Java, Lua, etc.


    None of the following is in any way a slight on what you have done - it
    seems to be the only way _to_ do it with the current state of Ruby.

    Java has a GC system that can track objects, return sizes, and all sorts
    of groovy stuff, and communicate it over a socket to an external
    profiler or debugger. Ruby, as best as I can read it, does not have
    such hooks built in, so you have to grope the native memory model.

    You can write such software tools in Java for Java. You will not be able
    to write them in Ruby for Ruby, until such hooks are added into the Ruby
    runtime.

    I, for one, would like to see such hooks added, as I believe a lack of
    memory profiling tools (built in at the lowest levels) is one of the
    bigger things in the way of Ruby scaling. If a webapp is going to run
    for months at a time, it really helps to be able to introspect long
    lived and large objects. (Some of my Java webapps have over a year of
    uptime, possible because we did that kind of memory leak testing. I
    would like to be able to do the same with Ruby on Rails.)

    Scott

    --
    Scott Ellsworth

    Java and database consulting for the life sciences
    Scott Ellsworth, Sep 23, 2005
    #9
  10. In message <>, Scott
    Ellsworth <> writes
    >You can write such software tools in Java for Java.


    Yes I appreciate that. I knew it when I wrote it. I was wondering who
    would spot the language specific mistake and how quickly. Bonus points
    for scoring on the first day! However for our tools it is nonsensical to
    rearchitect and reimplement a fully working Coverage/Profiler/Flow
    Tracer etc from C++ to Java just to support Java when we can do it
    better and faster (both in implementation time and execution time) from
    C++ in the first place (also, I'd much rather use JVMPI, JVMDI, JVMTI
    than any of that over the wire Java stuff). I've used Java on and off
    since 1996 and I just don't like it. It feels far too clumsy and lacks
    expression, although interfaces are useful. C++ and assembly I am happy
    with. Ruby I have a good feeling about although the syntax errors are
    obscure and often in the "wrong" place if you know what I mean. Main
    gripe about Ruby is that I don't have enough of a reason to use it for
    real work (it isn't suited for writing high performance software tools)
    as opposed to me experimenting with it - so I probably don't think quite
    in the Ruby way like those of you that get to use Ruby more than I. I
    digress.

    >I, for one, would like to see such hooks added, as I believe a lack of
    >memory profiling tools (built in at the lowest levels) is one of the
    >bigger things in the way of Ruby scaling.


    Well, I'll get around to the low level C side of things for the memory
    API in the near future. I know what to write and how to write it. It
    just a case of getting the time and having the energy to do it. It
    should be easy for someone to wrap a Ruby layer for sockets around what
    I write so that you can have the interface you describe.

    Stephen
    --
    Stephen Kellett
    Object Media Limited http://www.objmedia.demon.co.uk/software.html
    Computer Consultancy, Software Development
    Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
    Stephen Kellett, Sep 24, 2005
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mahesh Prasad
    Replies:
    1
    Views:
    695
    Tom Wells
    Feb 22, 2004
  2. Cy Huckaba
    Replies:
    1
    Views:
    1,136
    Xie Xiao
    Jun 26, 2003
  3. John Velman

    Tkinter measurements

    John Velman, Sep 28, 2004, in forum: Python
    Replies:
    3
    Views:
    537
    John Velman
    Sep 28, 2004
  4. Peter Hansen
    Replies:
    0
    Views:
    702
    Peter Hansen
    Feb 22, 2006
  5. Steve Howell

    medians for degree measurements

    Steve Howell, Jan 23, 2010, in forum: Python
    Replies:
    15
    Views:
    548
    Steve Howell
    Jan 25, 2010
Loading...

Share This Page