Map erase does not free memory

[Ged85]

Hi there. I have a problem that is sticked in my mind by days. I hope you can help.

This snippet of code:

map<unsigned long,unsigned long> foo_map;
for (int i = 0;i<K;i++)
foo_map = i;

cout<<"done("<<foo_map.size()<<")"<<endl;
cin>>stop;

cout<<"start empting"<<endl;
for (int i = 0;i<K;i++)
foo_map.erase(i);
cout<<"done("<<foo_map.size()<<")"<<endl;
cin>>stop;

Should empty the map container.

In fact, the size that I go to print turns to be 0. Instead, when I check the OS memory it doesn't change nothing!

As K is huge (order of 10^9), I need that the memory allocated is freed when I erase an element.

Can you give me some snippets, hints, pages, links where I can find some help?

I really thank you all.

Best regards,
G.

 

[Anand Hariharan]

Hi there. I have a problem that is sticked in my mind by days. I hope youcan help.
(...)
Can you give me some snippets, hints, pages, links where I can find some help?

Take a look at Effective STL by Meyers.

For your immediate problem, you want to use std::map::swap

- Anand

 

[Gerald Breuer]

Am 07.02.2013 14:56, schrieb Ged85:

In fact, the size that I go to print turns to be 0. Instead, when I check the OS memory it doesn't change nothing!

That's because malloc obtains its memory from the operating system in
multiples of pages which often can't easily returned to the operating
system for different reasons. But the memory is available for your
application after being freed.

 

[Melzzzzz]

I'd allocate a pool of anonymous memory with mmap() on a POSIX system
and free it with munamp() when no longer needed.

That is what malloc does anyway...

 

[Alain Ketterlin]

[...]

map<unsigned long,unsigned long> foo_map;
for (int i = 0;i<K;i++)
foo_map = i; [...]
for (int i = 0;i<K;i++)
foo_map.erase(i); [...]
As K is huge (order of 10^9), I need that the memory allocated is
freed when I erase an element.

If you're on Linux, see malloc_trim() and friends.

-- Alain.

 

[Ged85]

You don't specify which operating system or compiler you are using. It is difficult

to offer any substantial advice absent that information.



You could have used map::clear() instead of looping and calling map::erase.



You could simply "delete foo_map;". That would ensure that the container

memory is returned to the application heap, but there is no guarantee that the

application heap allocator (e.g. malloc()) will return the free memory to

the operating system (it generally doesn't).



I'd allocate a pool of anonymous memory with mmap() on a POSIX system and free it

with munamp() when no longer needed.



scott

I'm using ubuntu 10.04 with gcc 4.4.3.

The problem is that I'm writing a network simulator. Simulations are fairly long, and I use maps for implementing its critical data structures.

Most of these structures take just few items, but each time that I store a new key/value pair ,deleting some other one, the memory is not deallocated. So maps keep growing filling all my 24GB server memory....

May you be clearer about mmmap()? You're suggesting a custom map implementation?

Thank anyway.

Regards,
G.

 

[Ged85]

[...]

map<unsigned long,unsigned long> foo_map;

for (int i = 0;i<K;i++)

foo_map = i;
[...]

for (int i = 0;i<K;i++)

foo_map.erase(i);

[...]

As K is huge (order of 10^9), I need that the memory allocated is

freed when I erase an element.

If you're on Linux, see malloc_trim() and friends.



-- Alain.

How can I exploit malloc_trim() function? Custom allocator for map classes?

G.

 

[Gerald Breuer]

Am 07.02.2013 16:28, schrieb Alain Ketterlin:

If you're on Linux, see malloc_trim() and friends.

It would need only a single allocation somewhere at the end of the
heap to prevent malloc_trim from returning any substantial amount
of memory.

 

[Gerald Breuer]

Am 07.02.2013 16:32, schrieb Ged85:

Most of these structures take just few items, ...

Memory fragmentation can lead to memory freed with delete
or free not to be reused by later allocations, but usually
such a blowup shouldn't happen. Can it be that you missed
something of your code to post?

 

[Melzzzzz]

Most of these structures take just few items, but each time that I store
a new key/value pair ,deleting some other one, the memory is not
deallocated. So maps keep growing filling all my 24GB server memory....

Are you sure you don't have memory leak somewhere?
map should reuse memory , it would grow to some extent but 24gb?

 

[Alain Ketterlin]

Most of these structures take just few items, but each time that I
store a new key/value pair ,deleting some other one, the memory is not
deallocated. So maps keep growing filling all my 24GB server
memory....

This should not happen, deallocated blocks should be reused for new
allocations.

May you be clearer about mmmap()? You're suggesting a custom map
implementation?

Scott probably thinks about a custom _allocator_ using a block of memory
obtained via mmap(). If you're really alternating allocations and
deallocations it will not help much. It would be useful if you had a big
map that you'd have to deallocate in one shot.

-- Alain.

 

[Alain Ketterlin]

How can I exploit malloc_trim() function? Custom allocator for map
classes?

If you use g++'s standard allocator, all variants of new use malloc(),
malloc_trim() should apply directly.

Make sure you read malloc_trim() manual and check whether it applies to
your case. It will help only in specific cases (basically, no allocated
block ends up "above" your map cells and is retained until after
deallocating the map cells).

-- Alain.

 

[Marcel Müller]

I'm using ubuntu 10.04 with gcc 4.4.3.

The problem is that I'm writing a network simulator. Simulations are fairly long, and I use maps for implementing its critical data structures.

Most of these structures take just few items, but each time that I store a new key/value pair ,deleting some other one, the memory is not deallocated. So maps keep growing filling all my 24GB server memory....

First of all you should check your application for memory leaks. Maybe
not the map itself leaks memory.

If you are really sure that have no memory leak, then such symptoms are
often caused by heap fragmentation. Allocation and deleting objects of
increasing size can cause the virtual address space to get fragmented.
Especially if the allocations are intercepted by a few long lived objects.
In worst case the memory will grow quadratic with the allocated bytes.
Building large matrices in Matlab row by row was a common example.

May you be clearer about mmmap()? You're suggesting a custom map implementation?

I do not think that you need to write your own map. It might be
sufficient to use a custom allocator class to separate long lived and
short lived objects in different heaps.


Marcel

 

[Jorgen Grahn]

That is what malloc does anyway...

On which system? In Linux it's optional, user-configurable, and it's
undocumented whether free() unmaps the memory. It's also only for
huge blocks of data, and unlikely to affect a std::map<K, T>.

/Jorgen

 

[Jorgen Grahn]

10 gives you one that's it. The map should grow when you put things in,
but if you drop things and then put new ones in your memory should stay
the same. Somewhere you are allocating memory (or possibly creating some
kind of OS resource?) and not freeing it.

Does your platform have any heap walking diagnostics that would allow
you to see what is allocated after you have emptied the map?

He said he was on Linux, so chances are he has 'valgrind --memcheck
--leak-check=full'. Should be enough. (No doubt there are other ways
too, but he might as well see his other valgrind-detectable bugs now.)

/Jorgen

 

[Öö Tiib]

The problem is that I'm writing a network simulator. Simulations are fairly
long, and I use maps for implementing its critical data structures.

Fork such long separate calculation as a separate process. After it has
simulated just terminate the process. Operating system gets everything
that you did leak back.

Most of these structures take just few items, but each time that I store a
new key/value pair ,deleting some other one, the memory is not deallocated.
So maps keep growing filling all my 24GB server memory....

Freeing dynamically allocated memory does not result with process
returning the memory to operating system. That is normal. Process that
needed memory may need it again. Put your program into cycle of 100 and
see if it takes 100 times more memory.

I am about 98% sure that your program has defects and leaks memory. Try to
nail down your own defects first and then search defects in std::map.