A
ashwin.u.rao
Hi,
We are currently trying to identify and fix all the memory leaks by just doing Py_Initialize-PyRun_SimpleFile(some simple script)-Py_Finalize and found that there are around 70 malloc-ed blocks which are not freed. One of the significant contributor to this number is the 'files' object in ZipImporter. I am not able to identify the reason for this leak and was wondering if anyone on this list would help me out here.
So, here goes :
Since we have a zip file in our sys.path, this object is initialized and added to the zip_directory_cache dict during Py_Initialize. One point to note here is that there is no DECREF on the 'files' object after adding it to the zip_directory_cache dict.
When a module in a directory is imported(encoding.alias) then the reference count of 'files' is increased. When this module is unloaded during Py_Finalize-PyImport_Cleanup, the ref count of files object is decremented properly. So at the end of Py_Finalize the files object still has one reference count which is a result of it being an entry in the zip_directory_cache.
To summarize :
ZipImporter->files - zipimporter_init function:
Py_Initialize -
Read files from zip - ref_count -1
Add files to zip_directory_cache - ref count - 2
Import some modules from different directory in zip - ref count - 3
Py_Finalze -
PyImport_Cleanup - sub-directory - ref-count - 2
PyImport_Cleanup - main directory - ref-count - 1
So the reference count of 'files' is still 1 after Py_Finalize. The zip_directory_cache is the dict that has this 'files' object and this dict is added to the module during zipimport module initialization. This dict is destroyed during _PyModule_Clear(called by PyImport_Cleanup) but the 'files' object's reference count is not decremented.
Please excuse me if I have misinterpreted the whole logic. Any pointers towards solving this would be great.
Thank you.
~Ashwin
We are currently trying to identify and fix all the memory leaks by just doing Py_Initialize-PyRun_SimpleFile(some simple script)-Py_Finalize and found that there are around 70 malloc-ed blocks which are not freed. One of the significant contributor to this number is the 'files' object in ZipImporter. I am not able to identify the reason for this leak and was wondering if anyone on this list would help me out here.
So, here goes :
Since we have a zip file in our sys.path, this object is initialized and added to the zip_directory_cache dict during Py_Initialize. One point to note here is that there is no DECREF on the 'files' object after adding it to the zip_directory_cache dict.
When a module in a directory is imported(encoding.alias) then the reference count of 'files' is increased. When this module is unloaded during Py_Finalize-PyImport_Cleanup, the ref count of files object is decremented properly. So at the end of Py_Finalize the files object still has one reference count which is a result of it being an entry in the zip_directory_cache.
To summarize :
ZipImporter->files - zipimporter_init function:
Py_Initialize -
Read files from zip - ref_count -1
Add files to zip_directory_cache - ref count - 2
Import some modules from different directory in zip - ref count - 3
Py_Finalze -
PyImport_Cleanup - sub-directory - ref-count - 2
PyImport_Cleanup - main directory - ref-count - 1
So the reference count of 'files' is still 1 after Py_Finalize. The zip_directory_cache is the dict that has this 'files' object and this dict is added to the module during zipimport module initialization. This dict is destroyed during _PyModule_Clear(called by PyImport_Cleanup) but the 'files' object's reference count is not decremented.
Please excuse me if I have misinterpreted the whole logic. Any pointers towards solving this would be great.
Thank you.
~Ashwin