Extending Python with C or C++

[Ryan]

I've been using Python for many years now. It's a wonderful language
that I enjoy using everyday. I'm now interested in getting to know
more about the guts (C/C++) and extending it. But, extending python
still seems like a black art to me. Is there anymore docs or info on
extending it besides the standard sparse ones (http://www.python.org/
doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
class available? How can I learn more about the guts of python? How
would one go about following an interest in contributing to the
development of python.

Thanks,

Ryan

 

[Ralf Schoenian]

I've been using Python for many years now. It's a wonderful language
that I enjoy using everyday. I'm now interested in getting to know
more about the guts (C/C++) and extending it. But, extending python
still seems like a black art to me. Is there anymore docs or info on
extending it besides the standard sparse ones (http://www.python.org/
doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
class available? How can I learn more about the guts of python? How
would one go about following an interest in contributing to the
development of python.

Thanks,

Ryan

It is not exactly what you are looking for but nevertheless I am
thinking the article "Automatic C Library Wrapping -- Ctypes from the
Trenches" may be interesting for you. You can find it in the latest
Python Papers issue or simply following the link:
http://ojs.pythonpapers.org/index.php/tpp/article/view/71

Regards,
Ralf

 

[Terry Reedy]

I've been using Python for many years now. It's a wonderful language
that I enjoy using everyday. I'm now interested in getting to know
more about the guts

The 'guts' of Python the language include the object model, namespaces
(including modules), and the statement and infix-expression syntax.

(C/C++) and extending it.

Now you are asking about CPython, the leading computer implementation.

But, extending python still seems like a black art to me.
> Is there anymore docs or info on
extending it besides the standard sparse ones (http://www.python.org/
doc/2.5.2/ext/intro.html) that may give me more insight? Is there a
class available?

If you want to connect CPython to Python-oblivious code written in C,
Swig (with C code) and Ctypes (with Python code) are the main choices.
If you want to write new Python-aware (and specific) code, you can use
the CPython C-API functions. Extensions in C are written as importable
modules. The interface for such is not difficult; existing examples
should be a good guide.

> How can I learn more about the guts of python?

The 'guts' of an implementation follow from the 'guts' of the language.
There must be a syntax parser and compiler to internal form,
evaluation loop, and implemenations of built-in constants, functions,
classes, and modules. CPython's source tree begins as
http://svn.python.org/view/
You might actually want to start at
http://svn.python.org/view/python/trunk/
Note: if you click a filename, such as 'setup.py', you get the entire
revision history with checkin messages.
If you click the displayed revision number, such as '67978', you get the
latest checkin message and the current version of the file.

How would one go about following an interest in contributing to the
development of python.

Read
http://python.org/dev/
and start following the pydev list, mirrored to gmane.comp.python.devel
at news.gmane.org.

Terry Jan Reedy

 

[Ryan]

The 'guts' of Python the language include the object model, namespaces
(including modules), and the statement and infix-expression syntax.


Now you are asking about CPython, the leading computer implementation.




If you want to connect CPython to Python-oblivious code written in C,
Swig (with C code) and Ctypes (with Python code) are the main choices.
If you want to write new Python-aware (and specific) code, you can use
the CPython C-API functions. Extensions in C are written as importable
modules. The interface for such is not difficult; existing examples
should be a good guide.


The 'guts' of an implementation follow from the 'guts' of the language.
There must be a syntax parser and compiler to internal form,
evaluation loop, and implemenations of built-in constants, functions,
classes, and modules. CPython's source tree begins ashttp://svn.python.org/view/
You might actually want to start athttp://svn.python.org/view/python/trunk/
Note: if you click a filename, such as 'setup.py', you get the entire
revision history with checkin messages.
If you click the displayed revision number, such as '67978', you get the
latest checkin message and the current version of the file.


Readhttp://python.org/dev/
and start following the pydev list, mirrored to gmane.comp.python.devel
at news.gmane.org.

Terry Jan Reedy

Thanks Terry! This clarifies many of the concepts that I want to get
started to dive deeper into CPython.

1. The abstract Python Language (not specific to any implementation)
2. The CPython implementation (http://svn.python.org/view/python/
trunk/)
3. Extending CPython by connecting it to Python-oblivious code written
in C with Ctypes (Ralf's suggestion is good for this)
4. Extending CPython by connecting it to Python-aware (and specific)
code using the CPython C-API functions (http://docs.python.org/c-api/)

 

[Stefan Behnel]

3. Extending CPython by connecting it to Python-oblivious code written
in C with Ctypes (Ralf's suggestion is good for this)
4. Extending CPython by connecting it to Python-aware (and specific)
code using the CPython C-API functions (http://docs.python.org/c-api/)

For extending CPython (and possibly connecting it to external C
code/libraries) without learning too many new things (Python + a little C
should be enough), you can use ctypes if you can accept the runtime
overhead and the dependency on the ctypes module (which does not exist in
all CPython releases). However, I'd recommend Cython instead, as it gives
you very fast and portable code that works with all CPython versions. And
Cython code is not any harder to write at all.

http://cython.org/

Stefan

 

[sturlamolden]

I've been using Python for many years now. It's a wonderful language
that I enjoy using everyday. I'm now interested in getting to know
more about the guts (C/C++) and extending it. But, extending python
still seems like a black art to me.

There are several alternatives to using the Python C API by hand,
depending on
your particular need. It will save you a lot of greif:

Pyrex: Python-like language for creating extension modules.

*Cython: clone of Pyrex

*ctypes: call DLLs from python

comtypes: call COM DLLs from Python

*f2py: generate wrappers for Fortran or C libraries. Knowledge of
Fortran required.

pywin32: use or implement ActiveX objects with Python

*scipy.weave: inline C++ code in Python

Swig: generate wrappers for C libraries (some C++ support)

CXX: C++ wrapper for the Python C API

Boost.Python: C++ wrapper for the Python C API

SIP: generate wrappers for C++ libraries

(* Works easily with NumPy arrays.)

 

[Thomas Heller]

Interesting - I didn't know about h2xml and xml2py before and I've
done lots of ctypes wrapping! Something to help with the initial
drudge work of converting the structures would be very helpful.

( http://pypi.python.org/pypi/ctypeslib/ )

I gave it a quick go and it worked fine. I had to edit the XML in one
place to make it acceptable (removing a u from a hex number). The
output of xml2py was an excellent place to start for the conversion,
though I don't think I'd want to use an automated process like in the
paper above as its output needed tweaking.

...

If you are using a recent version of gccxml, then you should use the
ctypeslib-gccxml-0.9 branch of ctypeslib instead of the trunk. (I should
really merge the gccxml-0.9 branch into the trunk;-)

If you are already using the branch and the XML file is not accepted,
then could you please provide a short C code snippet that reproduces
the problem so that I can fix it?

Here are my thoughts on the conversion :-

It converted an interface which looked like this (the inotify interface)

struct inotify_event {
int wd; /* Watch descriptor */
uint32_t mask; /* Mask of events */
uint32_t cookie; /* Unique cookie associating related
events (for rename(2)) */
uint32_t len; /* Size of name field */
char name[]; /* Optional null-terminated name */
};

Into this

class inotify_event(Structure):
pass
inotify_event._fields_ = [
('wd', c_int),
('mask', uint32_t),
('cookie', uint32_t),
('len', uint32_t),
('name', c_char * 0),
]

Which is a very good start. However it defined these which clearly
aren't portable

int32_t = c_int
uint32_t = c_uint

Whereas it should have been using the ctypes inbuilt types

c_int32
c_uint32

IMO this would be difficult to achive automatically. There are cases
wher c_int is the correct type, in other cases c_int32 is correct.

Also I don't think c_char * 0 does anything sensible in ctypes,
c_byte * 0 is what is required plus a bit of casting. This is a
non-standard GNU extension to C though.

All that said though, it looks like a great time saver.

Thanks,
Thomas

 

[Thomas Heller]

I'm using gccxml from debian testing 0.9.0+cvs20080525-1 which I guess
doesn't have the code from the branch in.

I meant the branch in the repository where ctypeslib lives in.
Anyway, it doesn't matter anymore since I merged that branch
into the ctypeslib trunk.

Now the problem with the 'u' suffix that you mentioned should be fixed, and
I also made a quick change so that the sized integer types from stdint.h are generated
correctly. So, please update your ctypeslib installation and trey again;-)

Thomas