Map and memory issues

[Kelvin Moss]

Hi all,

I am writing to a map in a while loop and running into some issues.
Here is a pseudo code --

/* Standard headers included from namespace std*/

int main () {

string line;
size_t lines_written = 0;
map<Myobj, string> mymap;

try {

while (input file is being read) {

Myobj *obj;

...(read line)..
/* Construct obj on dynamic store based on the line read (if not
already created, else reuse)*/


itr = mymap.find(obj);
if (itr == mymap.end()) {
mymap.insert(std::make_pair(obj, line));
lines_written += line.length();
} else {
/* Append current record to existing value */
mymap[obj] += line;
lines_written += mymap[obj].length();
}
if (lines_written > BIG_VALUE) {
/* Write the map using the obj objects to some files (I get files
sorted by obj) */
...
/* Clear the map */
mymap.clear();
lines_written = 0;
}

} /* while loop */

} catch(...) {

/* Code reaching here */
}


}

I see that my code goes into the exception block after a while when I
execute it for big files with BIG_VALUE set to something like
(1024*1024). I am trying to write the files when maps have grown
sufficiently large. I then try to erase everything for the next
iteration. It works fine for small files.

It seems that map insertions are failing and map.clear is not helping
me here. I am probably running out of memory. Is there any bit that I
am missing or that I could improve?

Thanks ..
P.S. - I feel that I could go the C way and handle memory management
on my own by continually writing to a buffer on stack and later
sorting it based on obj (the not so recommended way).

 

[Alf P. Steinbach]

* Kelvin Moss:

Hi all,

I am writing to a map in a while loop and running into some issues.
Here is a pseudo code --

So, to sum up, this is not the code you're having problems with.

The problem code is totally different from what you present here.

That means that answers you get here may be totally unrelated to your
actual problem, and you're wasting people's time.

/* Standard headers included from namespace std*/

int main () {

string line;
size_t lines_written = 0;
map<Myobj, string> mymap;

try {

while (input file is being read) {

Myobj *obj;

..(read line)..
/* Construct obj on dynamic store based on the line read (if not
already created, else reuse)*/


itr = mymap.find(obj);

Searching for a pointer will only find a pointer, not an object with in
some sense same value as the pointed to object.

if (itr == mymap.end()) {
mymap.insert(std::make_pair(obj, line));
lines_written += line.length();

'lines_written' doesn't count lines, nor what's written, it apparently
counts characters stored.

} else {
/* Append current record to existing value */
mymap[obj] += line;
lines_written += mymap[obj].length();

Except that here the same characters are possibly counted again, with
'lines_written' a quadratic function of number of characters stored, in
the worst case.

}
if (lines_written > BIG_VALUE) {

Useful convention: don't use all uppercase except for macro names (which
should be all uppercase). Don't use macros for simple constants.

/* Write the map using the obj objects to some files (I get files
sorted by obj) */
...
/* Clear the map */
mymap.clear();

To release memory, use a swap with empty map.

lines_written = 0;
}

} /* while loop */

} catch(...) {

/* Code reaching here */
}


}

Please read the FAQ item about how to post a question about code that
doesn't work.

Cheers, & hth.,

- Alf

 

[David Harmon]

It seems that map insertions are failing and map.clear is not helping
me here. I am probably running out of memory. Is there any bit that I
am missing or that I could improve?

Did you call delete for every object you allocated with new?

 

[Daniel T.]

Hi all,

I am writing to a map in a while loop and running into some issues.
Here is a pseudo code --

/* Standard headers included from namespace std*/

int main () {

string line;
size_t lines_written = 0;
map<Myobj, string> mymap;

try {

while (input file is being read) {

Myobj *obj;

..(read line)..
/* Construct obj on dynamic store based on the line read (if not
already created, else reuse)*/


itr = mymap.find(obj);
if (itr == mymap.end()) {
mymap.insert(std::make_pair(obj, line));
lines_written += line.length();
} else {
/* Append current record to existing value */
mymap[obj] += line;
lines_written += mymap[obj].length();
}
if (lines_written > BIG_VALUE) {
/* Write the map using the obj objects to some files (I get files
sorted by obj) */
...
/* Clear the map */
mymap.clear();
lines_written = 0;
}

} /* while loop */

} catch(...) {

/* Code reaching here */
}


}

I see that my code goes into the exception block after a while when I
execute it for big files with BIG_VALUE set to something like
(1024*1024). I am trying to write the files when maps have grown
sufficiently large. I then try to erase everything for the next
iteration. It works fine for small files.

It seems that map insertions are failing and map.clear is not helping
me here. I am probably running out of memory. Is there any bit that I
am missing or that I could improve?

I agree with David Harmon, are probably newing objects without deleting
them.

Frankly though, we are all just guessing because your code as written is
inconsistent as to what is supposed to be stored in the map. (parts of
the above are written as if map stores Myobjs, while other parts are
written as if it stores Myobj*s.