Wierd behavior of gc.collect

[Bodhi]

I have a python process that does some operations and is supposed to release memory after those. The issue is that memory is not released (as seen through top). So I do a gc.collect() to see if there is any cycle etc. Immediately after doing the collect memory usage drops as expected, but strangely gc.collect() returns 0.
This means I cannot find out what the problem is by setting the debug option on gc which is what I usually do to figure out issues like this.

Maybe its that my understanding about it is incorrect, but if gc.collect returned 0, how come some memory was freed?

 

[Dave Angel]

I have a python process that does some operations and is supposed to release memory after those. The issue is that memory is not released (as seen through top). So I do a gc.collect() to see if there is any cycle etc. Immediately after doing the collect memory usage drops as expected, but strangely gc.collect() returns 0.
This means I cannot find out what the problem is by setting the debug option on gc which is what I usually do to figure out issues like this.

Maybe its that my understanding about it is incorrect, but if gc.collect returned 0, how come some memory was freed?

To put it simply, top won't in general show you that things are freed.
The C libraries for malloc and free will reuse the memory, but not
usually release it to the operating system. So it's not usually going
to show up in 'top.'

There was a long thread on this quite recently, but I can't seem to find
it right now.

 

[Bodhi]

I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case.

 

[Bodhi]

I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case.

 

[Dave Angel]

I know this, but my question is what does gc.collect do which results in the c library to free memory? Usually it is because of unreferenced objects in a cycle or something, but here that doesn't seem to be the case.

As I said, python calls the C free() function, whether it's when an
object's ref-count goes to zero, or whether it's during a gc call, where
circular refs are freed.

But free() does not necessarily release the memory to the OS. And the
times it does depends on which C library is being used, and what OS it's
running on.

If the freed memory affects top in some situations, it's a C library
detail. I've written a replacement C allocator in the past for Windows
that used a different scheme for blocks over a certain threshold, and
when those blocks were freed, it gave them back to the OS. But such
blocks were multiples of 64k, which was the increment for VirtualAlloc.

 

[Bodhi]

Thanks for the info.
I now suspect that the free lists are taking up the memory which won't be released unless we do a collect. I'm verifying that.

 

[Bodhi]

Thanks for the info.
I now suspect that the free lists are taking up the memory which won't be released unless we do a collect. I'm verifying that.