How do I pass structures using a C extension?

[timothy.williams]

Hi.

I trying to write an extension module to call some C libraries so I can
use them in Python. Several of the library functions pass pointers to
structures as arguments. I was thinking that I could create a class for
each structure, but I'm not sure how to get the data back and forth.
The example the "Extending and Embedding the Python Interpreter" manual
has examples of passing strings and ints using
PyArg_ParseTupleAndKeywords(), but it's not clear to me if I can even
pass structs around like this.

I'm just learning how to do extensions, so any help is greatly
appreciated.

Thanks.

 

[Jaime Wyant]

You need to check out swig. It is the *only* way to setup a `c'
library for use with python.

http://www.swig.org/

jw

 

[Diez B. Roggisch]

Hi.

I trying to write an extension module to call some C libraries so I can
use them in Python. Several of the library functions pass pointers to
structures as arguments. I was thinking that I could create a class for
each structure, but I'm not sure how to get the data back and forth.
The example the "Extending and Embedding the Python Interpreter" manual
has examples of passing strings and ints using
PyArg_ParseTupleAndKeywords(), but it's not clear to me if I can even
pass structs around like this.

I'm just learning how to do extensions, so any help is greatly
appreciated.

There are several ways you can accomplish this. You could use the module
"struct" to create byte-string representations of your structs and pass
these to your extension. In C, you just cast the string-pointer to your
struct and pass it. Disadvantages are: low security, and cumbersome on the
python side. Another way would be to pass the structures as python
structures - tuples and/or lists. Then manually disassemble these is the
extension (PyArg_ParseTupleAndKeywords and co. help you a great deal here)
and create an instance of the desired struct. A third way could be to
create an extension type that resembles your struct and use that to create
the "real" struct by getting the types instance values.

HTH

 

[qui le bile]

(e-mail address removed) a écrit :

Hi.

I trying to write an extension module to call some C libraries so I can
use them in Python. Several of the library functions pass pointers to
structures as arguments. I was thinking that I could create a class for
each structure, but I'm not sure how to get the data back and forth.
The example the "Extending and Embedding the Python Interpreter" manual
has examples of passing strings and ints using
PyArg_ParseTupleAndKeywords(), but it's not clear to me if I can even
pass structs around like this.

I'm just learning how to do extensions, so any help is greatly
appreciated.

Thanks.

You might be interested in the recently announced ctypes 0.9.5

<http://starship.python.net/crew/theller/ctypes/>

 

[timothy.williams]

Thanks for all the replies so far. I'm starting to look at SWIG, but
the libraries I want access to are all static. I created a small
interface file and a setup.py file, but when I build it, I get
undefined symbols. It sounds like the pack/unpack struct method is a
little to messy for my tastes. I was looking at the example for
defining a new type, but I wasn't sure how to pass the structure. I
need to look more into the PyArg_ParseTupleAndKeywords and creating a
list to pass into that.

I haven't looked at the ctypes module yet, but it looks like it not one
of the modules Python comes with, so I'm a little reluctant to use it.

Here's my set.py and interface file for my module:

##### setup.py ########
#!/bin/env python
import sys, os
from distutils.core import setup, Extension

OTB_HOME='/vps/otbknox/williams/OTB_2.0'
OTB_INCLDIR=[
os.path.join(OTB_HOME, 'include', 'global'),
os.path.join(OTB_HOME, 'include', 'libinc'),
os.path.join(sys.prefix,'include','python2.3'),
OTB_HOME
]
OTB_LIBDIR=[os.path.join(OTB_HOME, 'lib'),
os.path.join(sys.prefix, 'lib', 'python2.3')]
OTB_LIBS=['cmdline']

setup (name = "OTB_cmdline",
version="1.0",
author="Tim Williams",
ext_modules=[Extension('_OTB_cmdline',
sources=['OTB_cmdline.i'],
include_dirs=OTB_INCLDIR,
library_dirs=OTB_LIBDIR,
libraries=OTB_LIBS,

runtime_library_dirs=[os.path.join(OTB_HOME,'lib')]
)]
)
#############

##### module.i ##########
%module OTB_cmdline
%{
#include <libcmdline.h>
#include <stdext.h>
#include <stdio.h>
%}

extern void cmd_process_options (
int argc,
argv_t argv,
int *leftover_argc,
argv_t * leftover_argv,
CMD_OPTION * options,
uint32 sizeof_options,
int32 verbose);

###############

Here's the output of the build:
####################
python setup.py build --force
running build
running build_ext
building '_OTB_cmdline' extension
swigging OTB_cmdline.i to OTB_cmdline_wrap.c
swig -python -o OTB_cmdline_wrap.c OTB_cmdline.i
/usr/bin/gcc33 -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -fPIC
-I/vps/otbknox/williams/OTB_2.0/include/global
-I/vps/otbknox/williams/OTB_2.0/include/libinc
-I/project/c4i/Users_Share/williams/Linux/include/python2.3
-I/vps/otbknox/williams/OTB_2.0
-I/project/c4i/Users_Share/williams/Linux/include/python2.3 -c
OTB_cmdline_wrap.c -o build/temp.linux-i686-2.3/OTB_cmdline_wrap.o
OTB_cmdline_wrap.c:187: warning: `SWIG_Python_TypeDynamicCast' defined
but notused
OTB_cmdline_wrap.c:199: warning: `SWIG_Python_TypeName' defined but not
used
OTB_cmdline_wrap.c:205: warning: `SWIG_Python_TypeQuery' defined but
not used
OTB_cmdline_wrap.c:444: warning: `SWIG_Python_addvarlink' defined but
not used
OTB_cmdline_wrap.c:551: warning: `SWIG_Python_MustGetPtr' defined but
not used
OTB_cmdline_wrap.c:559: warning: `SWIG_Python_ConvertPacked' defined
but not used
gcc -pthread -shared build/temp.linux-i686-2.3/OTB_cmdline_wrap.o
-L/vps/otbknox/williams/OTB_2.0/lib
-L/project/c4i/Users_Share/williams/Linux/lib/python2.3-Wl,-R/vps/otbknox/williams/OTB_2.0/lib
-lcmdline -o build/lib.linux-i686-2.3/_OTB_cmdline.so
################

After copying the _OTB_cmdline.so file to the directory where
OTB_cmdline.py is, here's what happens when I try to import it:

#############
Python 2.3.2 (#10, Feb 24 2005, 10:59:54)
[GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "OTB_cmdline.py", line 5, in ?
import _OTB_cmdline
ImportError: ./_OTB_cmdline.so: undefined symbol: cmd_process_options
################

Can I only make extensions with shared libraries?

 

[Thomas Heller]

Thanks for all the replies so far. I'm starting to look at SWIG, but
the libraries I want access to are all static. I created a small
interface file and a setup.py file, but when I build it, I get
undefined symbols. It sounds like the pack/unpack struct method is a
little to messy for my tastes. I was looking at the example for
defining a new type, but I wasn't sure how to pass the structure. I
need to look more into the PyArg_ParseTupleAndKeywords and creating a
list to pass into that.

I haven't looked at the ctypes module yet, but it looks like it not one
of the modules Python comes with, so I'm a little reluctant to use it.

ctypes does only support shared libraries, not static libraries.

Thomas

 

[timothy.williams]

I have access to the source, so it seems that I can create shareable
libs for the libraries I want to use using distutils. I do a
glob.glob() on the *.c files in the libscr directory. If I copy the
main library.h file over to where my module.i file is, I can do a
%include on it and things seem to get built fine. (I still need to
actually call something to see if it works.)

Is there a way to change my setup.py file to look in the locations
specified by the 'include_dirs' argument? This argument works for my C
compiling, but it doesn't get passed to swig.


#!/bin/env python
import sys, os, glob
from distutils.core import setup, Extension

py_version='python%d.%d' % (sys.version_info[0],sys.version_info[1])
OTB_HOME='/vps/otbknox/williams/OTB_2.0'
OTB_INCLDIR=[
os.path.join(OTB_HOME, 'include', 'global'),
os.path.join(OTB_HOME, 'include', 'libinc'),
os.path.join(sys.prefix,'include',py_version),
OTB_HOME
]
libsrcs=glob.glob(os.path.join(OTB_HOME,'libsrc','libcmdline','*.c'))

setup (name = 'OTB_libs',
version='1.0',
author="Tim Williams",
## packages=['cmdline'],
## ext_package='OTB_libs',
ext_modules=[Extension('_cmdline',
sources=['cmdline.i']+ libsrcs,
include_dirs=OTB_INCLDIR
)
]
)


running build
running build_ext
building '_cmdline' extension
swigging cmdline.i to cmdline_wrap.c
swig -python -o cmdline_wrap.c cmdline.i
cmdline.i:9: Unable to find 'libcmdline.h'
error: command 'swig' failed with exit status 1

 

[Jaime Wyant]

I've never built a swig extension module using distutils. Heck, i've
only rolled up pure python code using it... that's pretty slick..

Oh, anyway, make sure that libcmdline.h is in swigs search path.
Either copy libcmdline.h to $CWD or figure out how to pass -I to swig
from distutils:

-I<dir> - Look for SWIG files in <dir>

You may find it easier to roll the swig wrappers by hand... It's not
that hard really...

hth,
jw

 

[timothy.williams]

I'm slowly learning how to use distutils. What I have now is a setup.py
and a subdirectory containing the extension I'm playing around with.
########## setup.py ##############
#!/bin/env python
import sys, os, glob
from distutils.core import setup, Extension

py_version='python%d.%d' % (sys.version_info[0],sys.version_info[1])
OTB_HOME='/vps/otbknox/williams/OTB_2.0'
OTB_INCLDIR=[
os.path.join(OTB_HOME, 'include', 'global'),
os.path.join(OTB_HOME, 'include', 'libinc'),
os.path.join(sys.prefix,'include',py_version),
OTB_HOME
]
cmdline_src=glob.glob(os.path.join(OTB_HOME,'libsrc','libcmdline','*.c'))

setup (name = 'OTB_libs',
version='1.0',
author="Tim Williams",
packages=['OTB_libs'],
py_modules=['OTB_libs.cmdline'],
ext_package='OTB_libs',
ext_modules=[Extension('_cmdline',
sources=['OTB_libs/cmdline.i']+
cmdline_src,
include_dirs=OTB_INCLDIR
)
]
)

#####################

ls -R setup.py build/lib.linux-i686-2.3 OTB_libs

setup.py

build/lib.linux-i686-2.3:
OTB_libs/

build/lib.linux-i686-2.3/OTB_libs:
cmdline.py _cmdline.so* __init__.py

OTB_libs:
cmdline.i cmdline.pyc __init__.py __init__.pyc setup.py~
cmdline.py cmdline_wrap.c __init__.py~ libcmdline.h
########################

I'll have a list of the <module>.py files that swig generates in a list
of py_modules in my package that's also a ext_package. 'python
setup.py install' puts everything in
site-packages/OTB_libs, and I can just do an
Copying the libcmdline.h file to the directory where my cmdline.i file
works, but I haven't figured out how to get distutils to tell swig to
look in the OTB_INCLDIR list.

 

[Yevgen Muntyan]

You need to check out swig. It is the *only* way to setup a `c'
library for use with python.

It's not.

Regards,
Yevgen