memory measurements

[Eric Mahurin]

In ruby, is there a portable way to measure memory usage? Any
of these would be useful:

- peak
- average
- current
- any of the above for just the memory that GC maintains

It would be nice if Benchmark did this if there was a portable
way. Many times there is a tradeoff between runtime and memory
usage and it would be good to see this easily.



=09
__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

 

[Stephen Kellett]

In ruby, is there a portable way to measure memory usage? Any
of these would be useful:

Not portable, but if you are using Windows you may want to take a look
at Ruby Memory Validator. No website description at the present time but
you can apply for the beta at http://www.softwareverify.com

Stephen

 

[Eric Mahurin]

I'm a linux guy. Here's what I do to get the memory of the
current ruby process:

IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display

Of course this is non-portable too.

I'd imagine to get a non-portable solution, you'd need
something in GC/ObjectSpace to figure out how much its objects
are taking up.

In message

=20
Not portable, but if you are using Windows you may want to
take a look=20
at Ruby Memory Validator. No website description at the
present time but=20
you can apply for the beta at http://www.softwareverify.com
=20
Stephen
--=20
Stephen Kellett
Object Media Limited =20
http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis,
Troubleshooting
=20
=20

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20

 

[Stephen Kellett]

I'd imagine to get a non-portable solution, you'd need
something in GC/ObjectSpace to figure out how much its objects
are taking up.

That is what Ruby Memory Validator does. Plus provide lots of view and
metrics to examine the reference graphs and so on.

Stephen

 

[Eric Mahurin]

In message

=20
That is what Ruby Memory Validator does. Plus provide lots of
view and=20
metrics to examine the reference graphs and so on.

I don't care about graphs and such. Just a method (or several)
in GC/ObjectSpace to get some memory info. If the above did
what I'm thinking about (walk through each
object/stack/symbol-table/etc and add up the memory used), it
would be portable, but you said it is windows only. Also, I
couldn't find any download link.

What would be ideal would be if the C call get_rusage worked on
linux (it returns 0 for various memory fields) and the
equivalent was available on various platforms. Of course you
could also make a method that did whatever was appropriate for
each platform too.




=09
__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

 

[Chris McGrath]

I don't care about graphs and such. Just a method (or several)
in GC/ObjectSpace to get some memory info. If the above did
what I'm thinking about (walk through each
object/stack/symbol-table/etc and add up the memory used), it
would be portable, but you said it is windows only. Also, I
couldn't find any download link.

You might find http://rubyforge.org/snippet/detail.php?type=3Dsnippet&id=3D=
70
does some of what you want.

Chris

 

[Stephen Kellett]

I don't care about graphs and such. Just a method (or several)
in GC/ObjectSpace to get some memory info. If the above did
what I'm thinking about (walk through each
object/stack/symbol-table/etc and add up the memory used), it
would be portable,

If you examine the source for Ruby you'll find that the above is
non-trivial and is implemented in C, in the form of the garbage
collector. I have offered to write a C API to collect such stats (and
will provide a simple Ruby API to provide other data to Ruby in the form
of a snapshot) but have not yet had the time to do it. Is this portable?
Yes if you regard compiling C on each platform to get the code working.

Someone else (sorry can't remember who) did write a Ruby snippet to walk
the object space. The main problem with this approach and any approach
that tries to trace ruby memory usage using say a set_trace_func() type
approach is the act of sending data to the ruby callback or walking the
object space in ruby allocates more objects, thus distorting the
picture, sometimes quite badly.

but you said it is windows only.

It is written in C++ and assembly. It is impossible to do low level
hooking of the type required to write this software in anything other
than similar languages. You could not write these software tools in
Ruby, Python, Java, Lua, etc.

Also, I
couldn't find any download link.

No website description at the present time but you can apply for the

beta >at

Stephen

 

[Eric Mahurin]

=20
You might find
http://rubyforge.org/snippet/detail.php?type=3Dsnippet&id=3D70
does some of what you want.

It looks to gauge memory usage by counting objects. Far from
accurate. And then it uses the same non-portable (linux only)
method I've been using to get the real answer:
/proc/<pid>/status.




=09
=09
______________________________________________________=20
Yahoo! for Good=20
Donate to the Hurricane Katrina relief effort.=20
http://store.yahoo.com/redcross-donate3/=20

 

[Scott Ellsworth]

In message <[email protected]>,


It is written in C++ and assembly. It is impossible to do low level
hooking of the type required to write this software in anything other
than similar languages. You could not write these software tools in
Ruby, Python, Java, Lua, etc.

None of the following is in any way a slight on what you have done - it
seems to be the only way _to_ do it with the current state of Ruby.

Java has a GC system that can track objects, return sizes, and all sorts
of groovy stuff, and communicate it over a socket to an external
profiler or debugger. Ruby, as best as I can read it, does not have
such hooks built in, so you have to grope the native memory model.

You can write such software tools in Java for Java. You will not be able
to write them in Ruby for Ruby, until such hooks are added into the Ruby
runtime.

I, for one, would like to see such hooks added, as I believe a lack of
memory profiling tools (built in at the lowest levels) is one of the
bigger things in the way of Ruby scaling. If a webapp is going to run
for months at a time, it really helps to be able to introspect long
lived and large objects. (Some of my Java webapps have over a year of
uptime, possible because we did that kind of memory leak testing. I
would like to be able to do the same with Ruby on Rails.)

Scott

 

[Stephen Kellett]

You can write such software tools in Java for Java.

Yes I appreciate that. I knew it when I wrote it. I was wondering who
would spot the language specific mistake and how quickly. Bonus points
for scoring on the first day! However for our tools it is nonsensical to
rearchitect and reimplement a fully working Coverage/Profiler/Flow
Tracer etc from C++ to Java just to support Java when we can do it
better and faster (both in implementation time and execution time) from
C++ in the first place (also, I'd much rather use JVMPI, JVMDI, JVMTI
than any of that over the wire Java stuff). I've used Java on and off
since 1996 and I just don't like it. It feels far too clumsy and lacks
expression, although interfaces are useful. C++ and assembly I am happy
with. Ruby I have a good feeling about although the syntax errors are
obscure and often in the "wrong" place if you know what I mean. Main
gripe about Ruby is that I don't have enough of a reason to use it for
real work (it isn't suited for writing high performance software tools)
as opposed to me experimenting with it - so I probably don't think quite
in the Ruby way like those of you that get to use Ruby more than I. I
digress.

I, for one, would like to see such hooks added, as I believe a lack of
memory profiling tools (built in at the lowest levels) is one of the
bigger things in the way of Ruby scaling.

Well, I'll get around to the low level C side of things for the memory
API in the near future. I know what to write and how to write it. It
just a case of getting the time and having the energy to do it. It
should be easy for someone to wrap a Ruby layer for sockets around what
I write so that you can have the interface you describe.

Stephen