How to detect that memory is used up?

Discussion in 'C++' started by Lambda, Jun 10, 2008.

  1. Lambda

    Lambda Guest

    I'd like to load a lot of data into a hashtable,
    when the memory is used up, I'll write the data to a file.
    I'm trying to use std::tr1::unordered_map to implement that.

    My question is if I insert too many data into the map, what will
    happen?
    Does unordered_map need continuous memory like vector?
    How to detect that there is no free memory available?
    Can I limit the amount of memory that is used by a container?
    And last, is it the right way to accomplish that, is there any
    alternative?

    Thanks!
    Lambda, Jun 10, 2008
    #1
    1. Advertising

  2. Lambda

    asterisc Guest

    On Jun 10, 10:40 am, Lambda <> wrote:
    > I'd like to load a lot of data into a hashtable,
    > when the memory is used up, I'll write the data to a file.
    > I'm trying to use std::tr1::unordered_map to implement that.
    >
    > My question is if I insert too many data into the map, what will
    > happen?
    > Does unordered_map need continuous memory like vector?
    > How to detect that there is no free memory available?
    > Can I limit the amount of memory that is used by a container?
    > And last, is it the right way to accomplish that, is there any
    > alternative?
    >
    > Thanks!


    The container will use as many memory as it can get, and throw an
    bad_alloc exception when it can't alloc anymore.
    You can limit the memory usage only by limiting the number of inserted
    'items'.
    asterisc, Jun 10, 2008
    #2
    1. Advertising

  3. Lambda wrote:
    > I'd like to load a lot of data into a hashtable,
    > when the memory is used up, I'll write the data to a file.
    > I'm trying to use std::tr1::unordered_map to implement that.
    >
    > My question is if I insert too many data into the map, what will
    > happen?
    > Does unordered_map need continuous memory like vector?
    > How to detect that there is no free memory available?
    > Can I limit the amount of memory that is used by a container?
    > And last, is it the right way to accomplish that, is there any
    > alternative?


    Sorry there is no such thing as 'memory is used up'.

    On a personal computer your program will first get slow (filling
    one or more gigabytes of RAM with your data), then it will get
    even slower as the operating system pages memory for you, and when
    you manage to fill up 2 or 4 Gb of data operator new will throw an
    exception. Then your program will either be terminated if you
    have no exception handling code, or one of your exception handlers
    will be reached.

    As I know standard template library containers can be instantiated
    with an allocator template parameter that will then be used whever
    the container allocates memory. Create such an allocator, make sure
    you control how much memory you allocate with it, and use it when
    you declare your container type. Have your allocator set a global
    flag when it has allocated a certain amount of memory
    (call this amount the buffer size if you want). Then in your outer
    (main) function watch for this flag and when it is set it is time
    to flush (write data to file), empty the container and reset the flag
    and the allocator. Make sure you know how to write a memory allocator
    before you do all this.

    You can also declare a 'new handler' function that the built-in
    operator new, provided by the C++ language, will call when memory is
    exhausted but before it throws an exception. You can use it to
    free some more memory, that you have previously allocated and
    reserved for this purpose, and set that global flag to trigger a
    flush. This is not a good idea though as it will trigger excessive
    paging (swapping memory on the hard drive's free space) on most
    personal computers, and will make your software run *really, really* slow.

    I can not tell you how much memory you should use for that purpose,
    as it is system-dependent and also depends on how loaded the
    machine is, but I would have no problem going for a 50/100 Mb buffer
    for personal computers, unless operator new throws and then I would
    try half, and then half, etc., and I would also make this a
    configuration setting, that is I would allow the user to set the buffer
    size on the command line, some configuration file, environment
    variable, system registry key, etc ... Also please make sure your data
    actually is at least 50 Mb in size before allocating such a buffer, as
    it would be a pity to allocate more memory than you need on some
    already loaded server or computer.


    Hope that helps.


    Timothy Madden
    Romania
    Timothy Madden, Oct 30, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Stephen Walch
    Replies:
    2
    Views:
    366
    Teemu Keiski
    Jan 17, 2004
  2. HK
    Replies:
    7
    Views:
    8,557
    John C. Bollinger
    Jun 7, 2005
  3. =?Utf-8?B?QmlzaG95?=
    Replies:
    1
    Views:
    297
    Mark Fitzpatrick
    Dec 24, 2006
  4. Sune
    Replies:
    14
    Views:
    852
    Chris Thomasson
    Aug 26, 2007
  5. Sune
    Replies:
    5
    Views:
    417
    Darko
    Jul 13, 2007
Loading...

Share This Page