Lambda said:
I'd like to load a lot of data into a hashtable,
when the memory is used up, I'll write the data to a file.
I'm trying to use std::tr1::unordered_map to implement that.
My question is if I insert too many data into the map, what will
happen?
Does unordered_map need continuous memory like vector?
How to detect that there is no free memory available?
Can I limit the amount of memory that is used by a container?
And last, is it the right way to accomplish that, is there any
alternative?
Sorry there is no such thing as 'memory is used up'.
On a personal computer your program will first get slow (filling
one or more gigabytes of RAM with your data), then it will get
even slower as the operating system pages memory for you, and when
you manage to fill up 2 or 4 Gb of data operator new will throw an
exception. Then your program will either be terminated if you
have no exception handling code, or one of your exception handlers
will be reached.
As I know standard template library containers can be instantiated
with an allocator template parameter that will then be used whever
the container allocates memory. Create such an allocator, make sure
you control how much memory you allocate with it, and use it when
you declare your container type. Have your allocator set a global
flag when it has allocated a certain amount of memory
(call this amount the buffer size if you want). Then in your outer
(main) function watch for this flag and when it is set it is time
to flush (write data to file), empty the container and reset the flag
and the allocator. Make sure you know how to write a memory allocator
before you do all this.
You can also declare a 'new handler' function that the built-in
operator new, provided by the C++ language, will call when memory is
exhausted but before it throws an exception. You can use it to
free some more memory, that you have previously allocated and
reserved for this purpose, and set that global flag to trigger a
flush. This is not a good idea though as it will trigger excessive
paging (swapping memory on the hard drive's free space) on most
personal computers, and will make your software run *really, really* slow.
I can not tell you how much memory you should use for that purpose,
as it is system-dependent and also depends on how loaded the
machine is, but I would have no problem going for a 50/100 Mb buffer
for personal computers, unless operator new throws and then I would
try half, and then half, etc., and I would also make this a
configuration setting, that is I would allow the user to set the buffer
size on the command line, some configuration file, environment
variable, system registry key, etc ... Also please make sure your data
actually is at least 50 Mb in size before allocating such a buffer, as
it would be a pity to allocate more memory than you need on some
already loaded server or computer.
Hope that helps.
Timothy Madden
Romania
