Please Help: can't deallocate a vector of heap objects

[gogogo_1001]

Dear all,


I don't understand why "delete" works well on destructing a object, but
fails to destruct a vector of it. Any of your comment is highly
appreciated!


Following is the program that represent this problem:


=========================================
int main (int argc, char *argv[])
{
FILE *input_file = fopen64 (argv[1], "r"); //input file is 216Mb,
//10 millon lines
[... initialize some variables here ..]


vector<m_op*> mVec; // a vector of pointer to m_op objects


//this loop will parse each input line into a m_op object,
//and then insert them into a vector
while (fgets (intput_line, 30, input_file) != NULL) //read in a line
{
[... assign values a,b,c from input_line ...]


m_op* ptr = new m_op (a, b, c);
mVec.push_back(ptr);
}


[break point #1] <- process memory size: 801Mb
//size of 10 million m_op objects

//this loop will delete on each object in the vector
for (vector<m_op*>::iterator it = mVec.begin();
it != mVec.end(); it++)
{
delete (*it);
}
mVec.clear();


[break point #2] <- process memory size: 763Mb //shouldn't be 0Mb?


}


======================================================
The input file is 216Mb, with 10 million lines. The first loop will
create a m_op object for each input line. At [break point #1], I use
"top" to inspect the process, and find that it takes 801Mb memory
space. So I expect after running the second loop, it will be something
close to zero. However, after the second loop, at [break point #2], the
process still holds 763Mb. Obviously, the vector has been cleared, but
the 10 million objects still remain in the memory. I can not figure out
why the destructor didn't free up the memory space.

However, if I put
the statement "delete ptr" right after it's "new", into a single loop,
the memory will always remain 808Kb, which means that the destructor
has worked properply.


======================================================
while (fgets (intput_line, 30, input_file) != NULL)
{
[... assign values a,b,c from input_line ...]


m_op* ptr = new m_op (a, b, c);
delete (ptr);
}
======================================================


The constructor and destructor is simply:


m_op::m_op(int cyc, string &rw, string &addr)
:_cycle_no(cyc),
_rw(rw),
_m_addr(addr),
{
}


m_op::~m_op()
{
}


=====================================================


I got no clue why destructor works well on a single object, but failed
to deallocate a vector of object.


The environment is RedHat Linux, x86, gcc 3.2.2, compiled with:
g++ -o prog1 main.cc -g


Any of your comment or help is highly appreciated!


Many thanks,
Charlie

 

[Victor Bazarov]

I don't understand why "delete" works well on destructing a object,
but fails to destruct a vector of it. Any of your comment is highly
appreciated!


Following is the program that represent this problem:


=========================================
int main (int argc, char *argv[])
{
FILE *input_file = fopen64 (argv[1], "r"); //input file is 216Mb,
//10 millon lines
[... initialize some variables here ..]

[..]
[break point #1] <- process memory size: 801Mb
//size of 10 million m_op objects
[..]


[break point #2] <- process memory size: 763Mb //shouldn't be 0Mb?
[..]

Whatever conclusions you try to draw from "process memory size", it has
no bearing on whether 'delete' works or not. Those are two totally
unrelated things. The memory may become available inside your process
but not to other processes. Or the OS may decide to make it available
to everybody. It's not in the language, it's in the platform. Please
ask for assistance in comp.os.linux.development.apps.

V

 

[codigo]

Dear all,


I don't understand why "delete" works well on destructing a object, but
fails to destruct a vector of it. Any of your comment is highly
appreciated!


Following is the program that represent this problem:


=========================================
int main (int argc, char *argv[])
{
FILE *input_file = fopen64 (argv[1], "r"); //input file is 216Mb,
//10 millon lines
[... initialize some variables here ..]


vector<m_op*> mVec; // a vector of pointer to m_op objects


//this loop will parse each input line into a m_op object,
//and then insert them into a vector
while (fgets (intput_line, 30, input_file) != NULL) //read in a line
{
[... assign values a,b,c from input_line ...]


m_op* ptr = new m_op (a, b, c);
mVec.push_back(ptr);
}


[break point #1] <- process memory size: 801Mb
//size of 10 million m_op objects

//this loop will delete on each object in the vector
for (vector<m_op*>::iterator it = mVec.begin();
it != mVec.end(); it++)
{
delete (*it);
}
mVec.clear();


[break point #2] <- process memory size: 763Mb //shouldn't be 0Mb?

Look up the subjects "scope" and "lifetime of objects" and then the
difference between "stack" and "heap" and when and how these allocations are
recovered (thats not what delete does).

Better yet, compile a program that does nothing, allocates nothing and set a
breakpoint on the return. What are your findings?

Then explain to us why a running program's process should somehow not be
occupying a given chunk of memory. Or is it your beleif that breakpoint #2
does not occur in a running program?

 

[Old Wolf]

I don't understand why "delete" works well on destructing a
object, but fails to destruct a vector of it.

The objects are destructed correctly. But you are misinterpreting
what your debugger is showing you.

int main (int argc, char *argv[])
{
while (fgets (intput_line, 30, input_file) != NULL)
{
m_op* ptr = new m_op (a, b, c);
mVec.push_back(ptr);
}

[break point #1] <- process memory size: 801Mb
//size of 10 million m_op objects

//this loop will delete on each object in the vector
for (vector<m_op*>::iterator it = mVec.begin();
it != mVec.end(); it++)
delete (*it);
mVec.clear();

[break point #2] <- process memory size: 763Mb //shouldn't be 0Mb?

This is an operating system issue. When you deallocate memory,
there is no requirement that the process return it to the
operating system.

However, if I put the statement "delete ptr" right after it's
"new", into a single loop, the memory will always remain 808Kb

Because the process only ever has one object in existence at
once, it only needs to request memory for one object from
the operating system. When you allocate the second object,
it re-uses the memory from the first object.

If you don't allocate any objects at all, you will probably
find the memory used is slightly less than 808Kb.

Any of your comment or help is highly appreciated!

Have you considered storing the objects in the vector, rather
than pointers? This will save you four bytes per object
(ie. 40Mb), and makes your job as a coder a lot easier.

Also, technically you should remove a pointer from the
vector before you delete it. If you delete a pointer while
it is in a vector, then the vector contains an indeterminate
value, and this could cause undefined behaviour (even if
the only thing you do to the vector is clear it).

For example:
for (vector<m_op*>::iterator it = mVec.begin();
it != mVec.end(); it++)
{
m_op *ptr = 0;
std::swap(*it, ptr);
delete ptr;
}
mVec.clear();

Also note that clear() on a vector isn't required to free
the memory used by the vector. To do that, you have to destroy
the vector; one convenient way to do that is:

mVec.swap( std::vector<m_op*>() );

 

[Samee Zahur]

I know this doesn't answer your question, but couldn't you just say
mVec.push_back(m_op(a,b,c)) where mVec is a simple vector<m_op> ??

That way you would never have to delete anything.

Samee

 

[Abecedarian]

Also, technically you should remove a pointer from the
vector before you delete it. If you delete a pointer while
it is in a vector, then the vector contains an indeterminate
value, and this could cause undefined behaviour (even if
the only thing you do to the vector is clear it).

For example:
for (vector<m_op*>::iterator it = mVec.begin();
it != mVec.end(); it++)
{
m_op *ptr = 0;
std::swap(*it, ptr);
delete ptr;
}
mVec.clear();

Also note that clear() on a vector isn't required to free
the memory used by the vector. To do that, you have to destroy
the vector; one convenient way to do that is:

mVec.swap( std::vector<m_op*>() );

Why sooo complicated?

const T* t = vector.back();
vector.pop_back();
delete t;


::A::

 

[Old Wolf]

Why sooo complicated?

const T* t = vector.back();
vector.pop_back();
delete t;

Your code only deletes one item, and causes UB if the vector
is empty. My code deletes all items safely.
If you extend your code to work correctly, then I think
it is at least as 'complicated' as mine.

 

[Abecedarian]

Your code only deletes one item, and causes UB if the vector
is empty. My code deletes all items safely.
If you extend your code to work correctly, then I think
it is at least as 'complicated' as mine.

while (!vector.empty()) {
const T* t = vector.back();
vector.pop_back();
delete t;
}

At least it doesn't copy (swap) objects unnecessarily.

::A::

 

[gogogo_1001]

Thank you sooo much! Eventually, I got this problem fixed. I stored
the objects directly into the vector.

My first thought was to use the code like this:
==========================
while (fgets (intput_line, 30, input_file) != NULL) //read in a line
{
[... assign values a,b,c from input_line ...]

m_op tmp(a, b, c);
mVec.push_back(tmp);
}
============================
After a reading Samee Zahur's post, I realised that it is incorrect, as
the push_back(const &T), is call by reference, so the next iteration of
the loop will overide tmp, so mVec[1] is referencing the same object as
mVec[0].
Creating a nameless object is better. Later on if I want to refer to
it, I can use mVec's iterator.
================
After "swap" the vector with a empty vector, the memory size has
greatly shrinked, to 380Mb. So I was confused how this couldn't free
them up? I also tried clear(), = emptyVec()... None worked.

Eventually, I found that there is another problem with the string, as
mVec has 2 string members, after calling ~string(), they still do not
free up the memory. So I changed to Char[], now, eventually, everyting
is fine.

This is the hardest bug I've ever met. And gave me lots of lessons.
- always prefer premitive data type than STL
- push_back is call by &, so if you call it multiple times in a loop,
create anonymous object, so the object names wouldn't be overiden.
- process wouldn't return claimed heap memory back to OS, same true for
malloc and free. But things are different for stack objects.
- Not to be superstition, but sometime, the bug might really not come
from your code
- use swap to free vector
- ...

Many thanks all of you for your kind suggestions!!!
Best regards,
Charlie

 

[Larry I Smith]

Thank you sooo much! Eventually, I got this problem fixed. I stored
the objects directly into the vector.

My first thought was to use the code like this:
==========================
while (fgets (intput_line, 30, input_file) != NULL) //read in a line
{
[... assign values a,b,c from input_line ...]

m_op tmp(a, b, c);
mVec.push_back(tmp);
}
============================
After a reading Samee Zahur's post, I realised that it is incorrect, as
the push_back(const &T), is call by reference, so the next iteration of
the loop will overide tmp, so mVec[1] is referencing the same object as
mVec[0].

When you put an object in an STL container (list, vector, set, etc)
The STL makes a copy of the object and stores that copy; it does
not store the original. That's why objects to be stored in
containers need to have (at least):

- a public copy constructor
- a public default constructor
- a public destructor
- a public assignment operator.

Regards,
Larry

 

[Old Wolf]

while (!vector.empty()) {
const T* t = vector.back();
vector.pop_back();
delete t;
}

At least it doesn't copy (swap) objects unnecessarily.

Right. But yours changes the size of the vector unnecessarily.
Is swapping any more expensive? Probably not..

Let's call a truce and say that both are good solutions ?