Memory Leak/Profiling

[jeungster]

Hello,

I'm trying to track down a memory issue with a C++ application that
I'm working on:

In a nutshell, the resident memory usage of my program continues to
grow as the program runs. It starts off at a nice 4% of memory, then
slowly grows up to 50% and beyond. This translates to around 2 gigs
of physical memory, and that's really way more memory than this
program should be taking up.

I'm looking for a tool that can tell me where this memory is being
allocated to.

I've fiddled around with both purify and ccmalloc without much
success. I suspect that what I'm seeing here is not a memory leak in
the traditional sense -- by which I mean, memory that has been
allocated and no longer has any pointers pointing to it.

I have a suspicion that somewhere in my program, a vector is growing
out of control. From my understanding, this type of growth won't be
detected by programs like purify because purify is looking for "true"
memory leaks.

Essentially what I'm looking for is a memory profiler -- I want to be
able to look at the memory that has been allocated to my program and
see what that memory is.

Does anyone know of any tools that will do this? Thanks!

-Michael Jeung

 

[Branimir Maksimovic]

Hello,

I'm trying to track down a memory issue with a C++ application that
I'm working on:

In a nutshell, the resident memory usage of my program continues to
grow as the program runs. It starts off at a nice 4% of memory, then
slowly grows up to 50% and beyond. This translates to around 2 gigs
of physical memory, and that's really way more memory than this
program should be taking up.

I'm looking for a tool that can tell me where this memory is being
allocated to.

I've fiddled around with both purify and ccmalloc without much
success. I suspect that what I'm seeing here is not a memory leak in
the traditional sense -- by which I mean, memory that has been
allocated and no longer has any pointers pointing to it.

I have a suspicion that somewhere in my program, a vector is growing
out of control.

What you are describing looks like memory leak caused by
fragmentation of memory. This type of leak is probably not
caused by vector, but buy large number of alloc's/free's
of varying sizes. To solve this problem you have to actually
do same thing as vector. To preallocate pools for each alloc
size and in that way keep fragmentation at bay.
In order to solve this successfully, you have to find out
what sizes are allocated/freed most.
If you can't determine that, even pool allocator won't help
you, because it will just preallocate too much blocks
for different sizes.
Or, you can just restart application automatically
if memory usage reaches some predefined peak
(once I did that, because controlling allocations was
impossible as this was rpc like server, and people might
do anything there )

Greetings, Branimir.

 

[Ian Collins]

Hello,

I'm trying to track down a memory issue with a C++ application that
I'm working on:

In a nutshell, the resident memory usage of my program continues to
grow as the program runs. It starts off at a nice 4% of memory, then
slowly grows up to 50% and beyond. This translates to around 2 gigs
of physical memory, and that's really way more memory than this
program should be taking up.

Well depending on the operating system, that might be expected behavior
and what you are seeing is the natural result of memory being allocated
and freed. Freed memory may not be reclaimed by the operating system
until it is required elsewhere. This results in the illusion that your
program has a leek.

 

[Ian Collins]

Well depending on the operating system, that might be expected behavior
and what you are seeing is the natural result of memory being allocated
and freed. Freed memory may not be reclaimed by the operating system
until it is required elsewhere. This results in the illusion that your
program has a leek.

Whoops, not a vegetable, but a memory leak!

 

[xsws5638]

Hello,

I'm trying to track down a memory issue with a C++ application that
I'm working on:

In a nutshell, the resident memory usage of my program continues to
grow as the program runs. It starts off at a nice 4% of memory, then
slowly grows up to 50% and beyond. This translates to around 2 gigs
of physical memory, and that's really way more memory than this
program should be taking up.

I'm looking for a tool that can tell me where this memory is being
allocated to.

I've fiddled around with both purify and ccmalloc without much
success. I suspect that what I'm seeing here is not a memory leak in
the traditional sense -- by which I mean, memory that has been
allocated and no longer has any pointers pointing to it.

I have a suspicion that somewhere in my program, a vector is growing
out of control. From my understanding, this type of growth won't be
detected by programs like purify because purify is looking for "true"
memory leaks.

Essentially what I'm looking for is a memory profiler -- I want to be
able to look at the memory that has been allocated to my program and
see what that memory is.

Does anyone know of any tools that will do this? Thanks!

-Michael Jeung

It is deeply abnormal! And i suggest you trace out all the memory
usages in you application. Then you can see what type and size of
memories you allocated. I believe there might be some memory leaks you
have not detected!

 

[James Kanze]

On May 19, 2:03 am, (e-mail address removed) wrote:

What you are describing looks like memory leak caused by
fragmentation of memory. This type of leak is probably not
caused by vector, but buy large number of alloc's/free's
of varying sizes.

Not necessarily. Modern collectors are pretty good at reducing
fragmentation.

The most frequent cause of memory leaks that I've seen in Java
(which has a moving collector, so doesn't suffer from
fragmentation at all) is simply objects which register for
events, and forget to deregister when they turn the source of
events off. So the event dispatching map just grows and grows.
Instrumenting the standard containers (using, say, the decorator
pattern) to keep track of the number of entries might help.

To solve this problem you have to actually
do same thing as vector. To preallocate pools for each alloc
size and in that way keep fragmentation at bay.

That's a strategy used by some mallocs. If he's having such a
problem, switching to a different malloc/free might help. (For
the most part, the system operator new will just call malloc.)

In order to solve this successfully, you have to find out
what sizes are allocated/freed most.
If you can't determine that, even pool allocator won't help
you, because it will just preallocate too much blocks
for different sizes.

A good implementation of malloc will detect this automatically,
and adjust its strategy accordingly.

 

[Branimir Maksimovic]

Not necessarily. Modern collectors are pretty good at reducing
fragmentation.

Yes, those are moving collectors that can't be implemented for C++
because of raw memory pointers and pointer arithmetic.
Altough one can imeplement such easilly with disciplined programing
and usage of objects as pointers instead of raw ones.
But, non moving collectors have same problem as manual
memory management.

That's a strategy used by some mallocs. If he's having such a
problem, switching to a different malloc/free might help. (For
the most part, the system operator new will just call malloc.)

I have tried to replace malloc in different ways
with my implementation on linux (because of same problem), but
unsucessfully. LD_PRELOAD won;t work because libc functions
call malloc during init time (so nobody knows which private function
of malloc interface will be called), and also malloc hooks can;t be
used because malloc uses those for it's internal purposes,
even that linux docs say opposite.
Problem is that when programers play smart and don;t use
global operator new or some function that wraps calls to malloc, then
since C don;t have interface for replacing that function
this is sometimes not an option.

A good implementation of malloc will detect this automatically,
and adjust its strategy accordingly.

I never saw such malloc. I can imagine that allocs would be
very costly with such implementation.
But my colleague and I have solved problem by aligning
allocation for different sizes into predefined sizes.
This uses more memory but does not have fragmentation that
much. This wasn't satisfactory solution for applications
that hold lot of memory since memory usage was a bit high.
So I really can't imagine general purpose allocator that
can solve all problems. In practice application programmer
always have to think about allocation/deallocation strategy,
if writing apps that will allocate/free
non-stop and run for a long time.

Greetings, Branimir.

 

[James Kanze]

Yes, those are moving collectors that can't be implemented for C++
because of raw memory pointers and pointer arithmetic.
Altough one can imeplement such easilly with disciplined programing
and usage of objects as pointers instead of raw ones.
But, non moving collectors have same problem as manual
memory management.

Not necessarily.

I have tried to replace malloc in different ways
with my implementation on linux (because of same problem), but
unsucessfully.

I've never had any problem on Solaris. If the default malloc
has caused problems, switching to one of the others delivered
with the system generally solved them.

It's important to realize, too, that while fragmentation will
generally increase the memory footprint of the program, it
almost never causes real leaking; the footprint may be larger,
but it is stable.

LD_PRELOAD won;t work because libc functions
call malloc during init time (so nobody knows which private function
of malloc interface will be called), and also malloc hooks can;t be
used because malloc uses those for it's internal purposes,
even that linux docs say opposite.
Problem is that when programers play smart and don;t use
global operator new or some function that wraps calls to malloc, then
since C don;t have interface for replacing that function
this is sometimes not an option.

Which malloc/free you use is decided at link time. So there's
no problem with static initializers (unless the malloc/free
requires some special initialization).

I never saw such malloc. I can imagine that allocs would be
very costly with such implementation.

Not really. All it entails is maintaining a bit of statistical
data. But it's true that it is generally easier for the vendor
to provide several different mallocs, and let the user
explicitly choose at link time.

But my colleague and I have solved problem by aligning
allocation for different sizes into predefined sizes.
This uses more memory but does not have fragmentation that
much. This wasn't satisfactory solution for applications
that hold lot of memory since memory usage was a bit high.
So I really can't imagine general purpose allocator that
can solve all problems. In practice application programmer
always have to think about allocation/deallocation strategy,
if writing apps that will allocate/free
non-stop and run for a long time.

That hasn't been my experience. Of course, very few of my
applications run for more than about ten years; maybe that's not
long enough.

 

[Branimir Maksimovic]

It's important to realize, too, that while fragmentation will
generally increase the memory footprint of the program, it
almost never causes real leaking; the footprint may be larger,
but it is stable.

I performed bit of search, and this is first what I have found.
http://www.codeproject.com/cpp/MemLeakByFragmentation.asp,
though I don;t see why not code in article just implements
operator new in class.
Seems that I'll get same headaches on windows as on linux

That hasn't been my experience. Of course, very few of my
applications run for more than about ten years; maybe that's not
long enough.

Well, I've over generalized it. I said that based on
my experience with linux, but I guess there can be
apps that never overload either class or global new
and have not problems, even when doing large number
of allocations.

Greetings, Branimir.

 

[EventHelix.com]

Hello,

I'm trying to track down a memory issue with a C++ application that
I'm working on:

In a nutshell, the resident memory usage of my program continues to
grow as the program runs. It starts off at a nice 4% of memory, then
slowly grows up to 50% and beyond. This translates to around 2 gigs
of physical memory, and that's really way more memory than this
program should be taking up.

I'm looking for a tool that can tell me where this memory is being
allocated to.

I've fiddled around with both purify and ccmalloc without much
success. I suspect that what I'm seeing here is not a memory leak in
the traditional sense -- by which I mean, memory that has been
allocated and no longer has any pointers pointing to it.

I have a suspicion that somewhere in my program, a vector is growing
out of control. From my understanding, this type of growth won't be
detected by programs like purify because purify is looking for "true"
memory leaks.

Essentially what I'm looking for is a memory profiler -- I want to be
able to look at the memory that has been allocated to my program and
see what that memory is.

Does anyone know of any tools that will do this? Thanks!

-Michael Jeung

Have you looked into Valgrind? It sometimes works in problems where
Purify will fail.