ctypes free memory which is allocated in C DLL

[zlchen.ken]

Hi Guys,

I have a DLL which written in C language, one of the function is to allocate a structure, fill the members and then return the pointer of the structure.

After Python called this function, and done with the returned structure, I would like to free the returned structure. How can I achieve this ? Basically, I tried that I wrote a corresponding free interface in the DLL, it works, but calling the libc.free in Python doesn't work.

my_dll.return_structure_ptr.restypes = POINTER(Dummy)
res_ptr = my_dll.return_structure_ptr()
windll.msvcrt.free(res_ptr) <==== doesn't work, memory violation
my_dll.free_dummy_struture(res_ptr) <== This works.

Thanks !

 

[Chris Angelico]

Hi Guys,

I have a DLL which written in C language, one of the function is to allocate a structure, fill the members and then return the pointer of the structure.

After Python called this function, and done with the returned structure, I would like to free the returned structure. How can I achieve this ? Basically, I tried that I wrote a corresponding free interface in the DLL, it works, but calling the libc.free in Python doesn't work.

As a general rule, you should always match your allocs and frees. Use
the same library to free the memory as was used to allocate it.
Nothing else is safe. If your DLL exposes an API that allocates
memory, your DLL should expose a corresponding API to free it. So what
you have is the best way

ChrisA

 

[Ken Chen]

As a general rule, you should always match your allocs and frees. Use

the same library to free the memory as was used to allocate it.

Nothing else is safe. If your DLL exposes an API that allocates

memory, your DLL should expose a corresponding API to free it. So what

you have is the best way



ChrisA

Thanks a lot, Chris!
Yes, I agree writing a corresponding API to free the memory is the best practice and best bet.
Sometimes, the third party API may not provide that.

After digging the Python manual again and again.
I finally figure out why windll.msvcrt.free is failing.

As the manual stated below, the DLL is using another version of msvcrt lib which is different than the builtin windll.msvcrt. After I explicitly load the msvcrt which built the DLL, things are getting function now.

"ctypes.util.find_msvcrt()
Windows only: return the filename of the VC runtype library used by Python,and by the extension modules. If the name of the library cannot be determined, None is returned.

If you need to free memory, for example, allocated by an extension module with a call to the free(void *), it is important that you use the function in the same library that allocated the memory."

 

[Ken Chen]

As a general rule, you should always match your allocs and frees. Use

the same library to free the memory as was used to allocate it.

Nothing else is safe. If your DLL exposes an API that allocates

memory, your DLL should expose a corresponding API to free it. So what

you have is the best way



ChrisA

Thanks a lot, Chris!
Yes, I agree writing a corresponding API to free the memory is the best practice and best bet.
Sometimes, the third party API may not provide that.

After digging the Python manual again and again.
I finally figure out why windll.msvcrt.free is failing.

As the manual stated below, the DLL is using another version of msvcrt lib which is different than the builtin windll.msvcrt. After I explicitly load the msvcrt which built the DLL, things are getting function now.

"ctypes.util.find_msvcrt()
Windows only: return the filename of the VC runtype library used by Python,and by the extension modules. If the name of the library cannot be determined, None is returned.

If you need to free memory, for example, allocated by an extension module with a call to the free(void *), it is important that you use the function in the same library that allocated the memory."

 

[Chris Angelico]

Yes, I agree writing a corresponding API to free the memory is the best practice and best bet.
Sometimes, the third party API may not provide that.

Then that's a majorly dangerous third party API. The only time it's
safe to provide a half-only API like that is when the code gets
statically linked with the application, so there's a guarantee that
malloc() in one place corresponds to free() in another.

After digging the Python manual again and again.
I finally figure out why windll.msvcrt.free is failing.

As the manual stated below, the DLL is using another version of msvcrt lib which is different than the builtin windll.msvcrt. After I explicitly load the msvcrt which built the DLL, things are getting function now.

That's still vulnerable. There's really no guarantee about _anything_
with mismatched memory allocators; it's theoretically possible for
compiler switches to change the behaviour of malloc/free such that
compiling malloc in debug mode and free in release mode would crash
your program. (I don't know off-hand of any compilers/libraries that
do that specifically, but there are similar changes done in other
ways.)

If a DLL allocates memory and doesn't deallocate it, lean on its
author to complete the job.

ChrisA

 

[Nobody]

I have a DLL which written in C language, one of the function is to
allocate a structure, fill the members and then return the pointer of
the structure.

After Python called this function, and done with the returned structure,
I would like to free the returned structure. How can I achieve this ?
Basically, I tried that I wrote a corresponding free interface in the
DLL, it works, but calling the libc.free in Python doesn't work.

my_dll.return_structure_ptr.restypes = POINTER(Dummy) res_ptr =
my_dll.return_structure_ptr() windll.msvcrt.free(res_ptr) <==== doesn't
work, memory violation my_dll.free_dummy_struture(res_ptr) <== This
works.

On Windows, a process may have multiple versions of the MSVCRT DLL (which
provides malloc/free). If an executable or DLL is linked against multiple
DLLs, each DLL could be using a different version of MSVCRT.

Different versions of MSVCRT may have separate heaps, so anything which
is allocated with malloc() (or calloc() or realloc()) from a specific
version of MSVCRT must be passed to free() from the same version of MSVCRT.
windll.msvcrt refers to the version of MSVCRT against which the Python DLL
is linked, which isn't necessarily the version against which my_dll is
linked.

If a function in a DLL returns a pointer to memory which it allocated
with malloc(), that DLL must also provide a function which can be used to
free that memory. It can't leave it to the application (or higher-level
DLL) to call free(), because the application may not be using the same
version of MSVCRT as the DLL.

 

[zlchen.ken]

On Windows, a process may have multiple versions of the MSVCRT DLL (which

provides malloc/free). If an executable or DLL is linked against multiple

DLLs, each DLL could be using a different version of MSVCRT.



Different versions of MSVCRT may have separate heaps, so anything which

is allocated with malloc() (or calloc() or realloc()) from a specific

version of MSVCRT must be passed to free() from the same version of MSVCRT.

windll.msvcrt refers to the version of MSVCRT against which the Python DLL

is linked, which isn't necessarily the version against which my_dll is

linked.



If a function in a DLL returns a pointer to memory which it allocated

with malloc(), that DLL must also provide a function which can be used to

free that memory. It can't leave it to the application (or higher-level

DLL) to call free(), because the application may not be using the same

version of MSVCRT as the DLL.

Thank you for the details.
This is really useful!