Map erase does not free memory

Discussion in 'C++' started by Ged85, Feb 7, 2013.

  1. Ged85

    Ged85 Guest

    Hi there. I have a problem that is sticked in my mind by days. I hope you can help.

    This snippet of code:

    map<unsigned long,unsigned long> foo_map;
    for (int i = 0;i<K;i++)
    foo_map = i;

    cout<<"done("<<foo_map.size()<<")"<<endl;
    cin>>stop;

    cout<<"start empting"<<endl;
    for (int i = 0;i<K;i++)
    foo_map.erase(i);
    cout<<"done("<<foo_map.size()<<")"<<endl;
    cin>>stop;

    Should empty the map container.

    In fact, the size that I go to print turns to be 0. Instead, when I check the OS memory it doesn't change nothing!

    As K is huge (order of 10^9), I need that the memory allocated is freed when I erase an element.

    Can you give me some snippets, hints, pages, links where I can find some help?

    I really thank you all.

    Best regards,
    G.
     
    Ged85, Feb 7, 2013
    #1
    1. Advertising

  2. On Feb 7, 7:56 am, Ged85 <> wrote:
    > Hi there. I have a problem that is sticked in my mind by days. I hope youcan help.
    >

    (...)
    > Can you give me some snippets, hints, pages, links where I can find some help?
    >


    Take a look at Effective STL by Meyers.

    For your immediate problem, you want to use std::map::swap

    - Anand
     
    Anand Hariharan, Feb 7, 2013
    #2
    1. Advertising

  3. Am 07.02.2013 14:56, schrieb Ged85:

    > In fact, the size that I go to print turns to be 0. Instead, when I check the OS memory it doesn't change nothing!


    That's because malloc obtains its memory from the operating system in
    multiples of pages which often can't easily returned to the operating
    system for different reasons. But the memory is available for your
    application after being freed.
     
    Gerald Breuer, Feb 7, 2013
    #3
  4. Ged85

    Melzzzzz Guest

    On Thu, 07 Feb 2013 15:19:04 +0000, Scott Lurndal wrote:

    >
    > I'd allocate a pool of anonymous memory with mmap() on a POSIX system
    > and free it with munamp() when no longer needed.


    That is what malloc does anyway...
     
    Melzzzzz, Feb 7, 2013
    #4
  5. Ged85 <> writes:

    [...]
    > map<unsigned long,unsigned long> foo_map;
    > for (int i = 0;i<K;i++)
    > foo_map = i;

    [...]
    > for (int i = 0;i<K;i++)
    > foo_map.erase(i);

    [...]
    > As K is huge (order of 10^9), I need that the memory allocated is
    > freed when I erase an element.


    If you're on Linux, see malloc_trim() and friends.

    -- Alain.
     
    Alain Ketterlin, Feb 7, 2013
    #5
  6. Ged85

    Ged85 Guest

    On Thursday, February 7, 2013 4:19:04 PM UTC+1, Scott Lurndal wrote:
    > Ged85 <> writes:
    >
    > >Hi there. I have a problem that is sticked in my mind by days. I hope you can help.

    >
    >
    >
    > You don't specify which operating system or compiler you are using. It is difficult
    >
    > to offer any substantial advice absent that information.
    >
    >
    >
    > You could have used map::clear() instead of looping and calling map::erase.
    >
    >
    >
    > You could simply "delete foo_map;". That would ensure that the container
    >
    > memory is returned to the application heap, but there is no guarantee that the
    >
    > application heap allocator (e.g. malloc()) will return the free memory to
    >
    > the operating system (it generally doesn't).
    >
    >
    >
    > I'd allocate a pool of anonymous memory with mmap() on a POSIX system and free it
    >
    > with munamp() when no longer needed.
    >
    >
    >
    > scott


    I'm using ubuntu 10.04 with gcc 4.4.3.

    The problem is that I'm writing a network simulator. Simulations are fairly long, and I use maps for implementing its critical data structures.

    Most of these structures take just few items, but each time that I store a new key/value pair ,deleting some other one, the memory is not deallocated. So maps keep growing filling all my 24GB server memory....

    May you be clearer about mmmap()? You're suggesting a custom map implementation?

    Thank anyway.

    Regards,
    G.
     
    Ged85, Feb 7, 2013
    #6
  7. Ged85

    Ged85 Guest

    On Thursday, February 7, 2013 4:28:58 PM UTC+1, Alain Ketterlin wrote:
    > Ged85 <> writes:
    >
    >
    >
    > [...]
    >
    > > map<unsigned long,unsigned long> foo_map;

    >
    > > for (int i = 0;i<K;i++)

    >
    > > foo_map = i;

    >
    > [...]
    >
    > > for (int i = 0;i<K;i++)

    >
    > > foo_map.erase(i);

    >
    > [...]
    >
    > > As K is huge (order of 10^9), I need that the memory allocated is

    >
    > > freed when I erase an element.

    >
    >
    >
    > If you're on Linux, see malloc_trim() and friends.
    >
    >
    >
    > -- Alain.


    How can I exploit malloc_trim() function? Custom allocator for map classes?

    G.
     
    Ged85, Feb 7, 2013
    #7
  8. Am 07.02.2013 16:28, schrieb Alain Ketterlin:

    > If you're on Linux, see malloc_trim() and friends.


    It would need only a single allocation somewhere at the end of the
    heap to prevent malloc_trim from returning any substantial amount
    of memory.
     
    Gerald Breuer, Feb 7, 2013
    #8
  9. Am 07.02.2013 16:32, schrieb Ged85:

    > Most of these structures take just few items, ...



    Memory fragmentation can lead to memory freed with delete
    or free not to be reused by later allocations, but usually
    such a blowup shouldn't happen. Can it be that you missed
    something of your code to post?
     
    Gerald Breuer, Feb 7, 2013
    #9
  10. Ged85

    Melzzzzz Guest

    On Thu, 07 Feb 2013 07:32:08 -0800, Ged85 wrote:

    >
    > Most of these structures take just few items, but each time that I store
    > a new key/value pair ,deleting some other one, the memory is not
    > deallocated. So maps keep growing filling all my 24GB server memory....


    Are you sure you don't have memory leak somewhere?
    map should reuse memory , it would grow to some extent but 24gb?
     
    Melzzzzz, Feb 7, 2013
    #10
  11. Ged85 <> writes:

    > On Thursday, February 7, 2013 4:19:04 PM UTC+1, Scott Lurndal wrote:

    [...]
    >> I'd allocate a pool of anonymous memory with mmap() on a POSIX system
    >> and free it with munamp() when no longer needed.


    > Most of these structures take just few items, but each time that I
    > store a new key/value pair ,deleting some other one, the memory is not
    > deallocated. So maps keep growing filling all my 24GB server
    > memory....


    This should not happen, deallocated blocks should be reused for new
    allocations.

    > May you be clearer about mmmap()? You're suggesting a custom map
    > implementation?


    Scott probably thinks about a custom _allocator_ using a block of memory
    obtained via mmap(). If you're really alternating allocations and
    deallocations it will not help much. It would be useful if you had a big
    map that you'd have to deallocate in one shot.

    -- Alain.
     
    Alain Ketterlin, Feb 7, 2013
    #11
  12. Ged85 <> writes:

    > On Thursday, February 7, 2013 4:28:58 PM UTC+1, Alain Ketterlin wrote:


    >> If you're on Linux, see malloc_trim() and friends.


    > How can I exploit malloc_trim() function? Custom allocator for map
    > classes?


    If you use g++'s standard allocator, all variants of new use malloc(),
    malloc_trim() should apply directly.

    Make sure you read malloc_trim() manual and check whether it applies to
    your case. It will help only in specific cases (basically, no allocated
    block ends up "above" your map cells and is retained until after
    deallocating the map cells).

    -- Alain.
     
    Alain Ketterlin, Feb 7, 2013
    #12
  13. On 07.02.13 16.32, Ged85 wrote:
    > I'm using ubuntu 10.04 with gcc 4.4.3.
    >
    > The problem is that I'm writing a network simulator. Simulations are fairly long, and I use maps for implementing its critical data structures.
    >
    > Most of these structures take just few items, but each time that I store a new key/value pair ,deleting some other one, the memory is not deallocated. So maps keep growing filling all my 24GB server memory....


    First of all you should check your application for memory leaks. Maybe
    not the map itself leaks memory.

    If you are really sure that have no memory leak, then such symptoms are
    often caused by heap fragmentation. Allocation and deleting objects of
    increasing size can cause the virtual address space to get fragmented.
    Especially if the allocations are intercepted by a few long lived objects.
    In worst case the memory will grow quadratic with the allocated bytes.
    Building large matrices in Matlab row by row was a common example.

    > May you be clearer about mmmap()? You're suggesting a custom map implementation?


    I do not think that you need to write your own map. It might be
    sufficient to use a custom allocator class to separate long lived and
    short lived objects in different heaps.


    Marcel
     
    Marcel Müller, Feb 7, 2013
    #13
  14. Ged85

    Jorgen Grahn Guest

    On Thu, 2013-02-07, Melzzzzz wrote:
    > On Thu, 07 Feb 2013 15:19:04 +0000, Scott Lurndal wrote:
    >
    >>
    >> I'd allocate a pool of anonymous memory with mmap() on a POSIX system
    >> and free it with munamp() when no longer needed.

    >
    > That is what malloc does anyway...


    On which system? In Linux it's optional, user-configurable, and it's
    undocumented whether free() unmaps the memory. It's also only for
    huge blocks of data, and unlikely to affect a std::map<K, T>.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
     
    Jorgen Grahn, Feb 7, 2013
    #14
  15. Ged85

    Jorgen Grahn Guest

    On Thu, 2013-02-07, Andy Champ wrote:
    > On 07/02/2013 15:58, Melzzzzz wrote:
    >> On Thu, 07 Feb 2013 07:32:08 -0800, Ged85 wrote:
    >>
    >>>
    >>> Most of these structures take just few items, but each time that I store
    >>> a new key/value pair ,deleting some other one, the memory is not
    >>> deallocated. So maps keep growing filling all my 24GB server memory....

    >>
    >> Are you sure you don't have memory leak somewhere?
    >> map should reuse memory , it would grow to some extent but 24gb?
    >>

    >
    > 10 gives you one that's it. The map should grow when you put things in,
    > but if you drop things and then put new ones in your memory should stay
    > the same. Somewhere you are allocating memory (or possibly creating some
    > kind of OS resource?) and not freeing it.
    >
    > Does your platform have any heap walking diagnostics that would allow
    > you to see what is allocated after you have emptied the map?


    He said he was on Linux, so chances are he has 'valgrind --memcheck
    --leak-check=full'. Should be enough. (No doubt there are other ways
    too, but he might as well see his other valgrind-detectable bugs now.)

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
     
    Jorgen Grahn, Feb 7, 2013
    #15
  16. Ged85

    Öö Tiib Guest

    On Thursday, 7 February 2013 17:32:08 UTC+2, Ged85 wrote:
    >
    > The problem is that I'm writing a network simulator. Simulations are fairly
    > long, and I use maps for implementing its critical data structures.


    Fork such long separate calculation as a separate process. After it has
    simulated just terminate the process. Operating system gets everything
    that you did leak back.

    > Most of these structures take just few items, but each time that I store a
    > new key/value pair ,deleting some other one, the memory is not deallocated.
    > So maps keep growing filling all my 24GB server memory....


    Freeing dynamically allocated memory does not result with process
    returning the memory to operating system. That is normal. Process that
    needed memory may need it again. Put your program into cycle of 100 and
    see if it takes 100 times more memory.

    I am about 98% sure that your program has defects and leaks memory. Try to
    nail down your own defects first and then search defects in std::map.
     
    Öö Tiib, Feb 8, 2013
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. erase vs. erase

    , Mar 25, 2006, in forum: C++
    Replies:
    7
    Views:
    369
    Pete Becker
    Mar 30, 2006
  2. Milind

    Map erase memory problem

    Milind, Jun 15, 2006, in forum: C++
    Replies:
    1
    Views:
    389
    Daniel T.
    Jun 15, 2006
  3. Replies:
    3
    Views:
    1,758
    Marcus Kwok
    Mar 14, 2007
  4. newbie
    Replies:
    2
    Views:
    298
    newbie
    Jul 27, 2007
  5. Panduranga Chary

    How memory function free() knows how much memory to free.

    Panduranga Chary, Dec 27, 2007, in forum: C Programming
    Replies:
    2
    Views:
    419
    Keith Thompson
    Dec 27, 2007
Loading...

Share This Page