ctypes from_param() truncating 64 bit pointers to 32 bit

[Joakim Hove]

Hello,

I wonder if I am encountering a bug in ctypes; or if I am doing things
wrong? I have written a small Python class wrapping a C library, the
Python class has a from_param() method which passes a "raw pointer
value" from Python to C. When reaching C the pointer has been
truncated to 32 bits.

I have written a quite extensive post about this on Stackoverflow:
http://www.stackoverflow.com/q/5029341/421889/ - so far without
response. Would have been very glad if anyone could indicate if this
was a bug on my part, or if it is a ctypes bug.

I have tried with Python version 2.4 and 2.6 with identical results.

Regards Joakim

 

[Stefan Behnel]

Joakim Hove, 18.02.2011 09:29:

I wonder if I am encountering a bug in ctypes; or if I am doing things
wrong? I have written a small Python class wrapping a C library, the
Python class has a from_param() method which passes a "raw pointer
value" from Python to C. When reaching C the pointer has been
truncated to 32 bits.

I have written a quite extensive post about this on Stackoverflow:
http://www.stackoverflow.com/q/5029341/421889/ - so far without
response. Would have been very glad if anyone could indicate if this
was a bug on my part, or if it is a ctypes bug.

I rarely use ctypes and I actually don't know how this specific case works.
However, you are not providing a function signature for the "print_addr" C
function in the BugTest case, so my guess is that it assumes the standard
signature for C functions, which is "int func(int)". That would explain the
32bit truncation.

Stefan

 

[Joakim Hove]

Thanks for answering;

I rarely use ctypes

You should Apart from my current struggle I think ctypes is really
good.

and I actually don't know how this specific case works.
However, you are not providing a function signature for the "print_addr" C
function in the BugTest case, so my guess is that it assumes the standard
signature for C functions, which is "int func(int)". That would explain the
32bit truncation.

Well; I think you are wrong. First of all because what I showed in the
Stackoverflow post was just a contrived example - the real code where
the problem initially arose does indeed have the proper function
signature. Secondly: <Uncertain>The use function signatures will be
relevant for C-C linkage, but ctypes library only relates to the fully
linked shared librarry lib.so where these signatures have been long
lost anyway; that is why we have to specify the argtypes and restype
attributes of the ctypes function objects?</Uncertain>.

Joakim

 

[Stefan Behnel]

Joakim Hove, 18.02.2011 09:46:

Thanks for answering;


You should Apart from my current struggle I think ctypes is really
good.

I'm well aware that it's a very good solution for the cases where you can
trade performance for minimizing your external dependencies. But being a
Cython developer, those use cases are truly rare for me.

Well; I think you are wrong.

Nicely put.

First of all because what I showed in the
Stackoverflow post was just a contrived example - the real code where
the problem initially arose does indeed have the proper function
signature.

Ok, then take care to post proper code next time. Details can be important,
especially when asking about non-obvious problems.

Secondly:<Uncertain>The use function signatures will be
relevant for C-C linkage, but ctypes library only relates to the fully
linked shared librarry lib.so where these signatures have been long
lost anyway; that is why we have to specify the argtypes and restype
attributes of the ctypes function objects?</Uncertain>.

Hmm, so you think that to call a C function, ctypes doesn't need to know
what the exact signature is? That sounds a bit naive to me.

It's correct that all compile time information about function signatures is
gone in the library. So you must provide ctypes with the exact signature to
allow it to convert your input and output data correctly, and to put the
values on the call stack in the correct calling layout.

Stefan

 

[Stefan Krah]

Well; I think you are wrong. First of all because what I showed in the
Stackoverflow post was just a contrived example - the real code where
the problem initially arose does indeed have the proper function
signature.

I suggest that you reconsider, since this appears to work:

def from_param(self):
return ctypes.c_void_p(self.c_ptr)



Stefan Krah

 

[Joakim Hove]

Ok, then take care to post proper code next time. Details can be important,

especially when asking about non-obvious problems.

Point taken - however the original context where the problem arose
consists of several massive libraries, which would be quite
impractical to post on a forum. The code I did post was proper in the
sense that it compiles and demonstrates the problem.

Hmm, so you think that to call a C function, ctypes doesn't need to know
what the exact signature is? That sounds a bit naive to me.

I fully agree that ctypes needs to know the type signature of the C
functions it wishes to call, my point is that this particular
information is not present in the shared library, and for that reason
we must set the input / return attributes of the ctypes functions
manually. In the Python code I do this like:

class BugTest:
....
def from_param(self):
return self.c_ptr

def __init__(self)
self.c_ptr =

lib_handle = ctypes.CDLL( "lib.so" )
....
func = getattr( lib_handle , "print_addr")

# Set the return type:
func.restype = None

# Set the types of the arguments:
func.argtypes = [ BugTest ]

Now - the point is that if I omit the ctypes method from_param() and
instead set the argtype as:

func.argtypes = [ ctypes.c_void_p ]

And manually passing the internalized C - pointer to C things work as
I expect; I believe that the problem is somewhere along the
from_param() function path.

I did not make it really clear that if the original C allocation
points to memory in "32 bit region" everything works fine (as I
expected).

Thank you for your time.

Joakim

 

[Joakim Hove]

I suggest that you reconsider, since this appears to work:

    def from_param(self):
        return ctypes.c_void_p(self.c_ptr)

Stefan Krah

That works - thank you very much!

Joakim