Using C++ and ctypes together: a vast conspiracy? ;)

J

Joseph Garvin

So I was curious whether it's possible to use the ctypes module with
C++ and if so how difficult it is. I figure in principal it's possible
if ctypes knows about each compiler's name mangling scheme. So I
searched for "ctypes c++" on Google.

The third link will be "Using ctypes to Wrap C++ Libraries". If you
follow the link, it's broken. If you view the cache of the link, it's
someone pointing to another blog, retrograde-orbit.blogspot.com,
saying they discovered a way to do it easily. If you follow that link,
you get taken a page does not exist error.

Clearly there's some way to use ctypes with C++ and there's a vast
conspiracy preventing it from reaching the masses ;) What's even
stranger is that this link, despite being broken, has seemingly been
near the top of google's results for these terms for a couple weeks
(that's when I last tried), as if there were some underground group of
rebels trying to hint the truth to us... ;)

More seriously -- how difficult is it to use ctypes instead of saying,
boost::python, and why isn't this in a FAQ somewhere? ;)
 
D

Diez B. Roggisch

Joseph said:
So I was curious whether it's possible to use the ctypes module with
C++ and if so how difficult it is. I figure in principal it's possible
if ctypes knows about each compiler's name mangling scheme. So I
searched for "ctypes c++" on Google.

The third link will be "Using ctypes to Wrap C++ Libraries". If you
follow the link, it's broken. If you view the cache of the link, it's
someone pointing to another blog, retrograde-orbit.blogspot.com,
saying they discovered a way to do it easily. If you follow that link,
you get taken a page does not exist error.

Clearly there's some way to use ctypes with C++ and there's a vast
conspiracy preventing it from reaching the masses ;) What's even
stranger is that this link, despite being broken, has seemingly been
near the top of google's results for these terms for a couple weeks
(that's when I last tried), as if there were some underground group of
rebels trying to hint the truth to us... ;)

More seriously -- how difficult is it to use ctypes instead of saying,
boost::python, and why isn't this in a FAQ somewhere? ;)

Because it's much more needed than name-mangling. Name mangling is
(amongst other things) one thing to prevent
C++-inter-compiler-interoperability which results from differing C++ ABIs.

I'm not an expert on this, but AFAIK no common ABI exists, especially
not amongst VC++ & G++. Which means that to work for C++, ctypes would
need to know about the internals of both compilers, potentially in
several versions, and possibly without prior notice to changes.

Instead of dealing with this truly daunting task, tools such as SIP and
SWIG or boost::python deal with this by utilizing the one tool that
really knows about the ABI in use - the compiler itself.

So while I'd loved to be proven wrong, I fear your curiosity won't be
satisfied - or at least not in the positive sense you certainly envisioned.

Diez
 
R

Roy Smith

More seriously -- how difficult is it to use ctypes instead of saying,
boost::python, and why isn't this in a FAQ somewhere? ;)

Because it's much more needed than name-mangling. Name mangling is
(amongst other things) one thing to prevent
C++-inter-compiler-interoperability which results from differing C++ ABIs.[/QUOTE]

Indeed. Name mangling is the most trivial way in which different C++
compilers do not interoperate. Some other thorny issues include exception
handling, template instantiation, physical layout of struct and class
elements (i.e. padding/alignment), and how references are returned (i.e.
Return Value Optimization). I'm sure I've left out several critical items.

It's an ugly world out there.
 
L

Lawrence D'Oliveiro

No wonder, you have never actually used C++ with C types. An extern "C"
clause tells the compiler to generate C functions (more precisely,
functions that conform to the C ABI conventions), so effectively you're
calling into C, not into C++.

Seems like the only sane way to do it. In all other directions lies madness.
 
K

Kay Schluehr

Seems like the only sane way to do it. In all other directions lies madness.

Yes but creating C stubs is also hard in presence of everything that
is not basic C++. How would you wrap the STL? I suspect one ends up in
creating a variant of SWIG and I wonder if it's not a good idea to
just use SWIG then.
 
L

Lawrence D'Oliveiro

In message <ad634d5d-
(e-mail address removed)>, Kay Schluehr
wrote:
Yes but creating C stubs is also hard in presence of everything that
is not basic C++. How would you wrap the STL?

What does the STL offer that Python doesn't already do more flexibly and
more simply?
 
A

A. Cavallo

No wonder, you have never actually used C++ with C types. An extern
What does the STL offer that Python doesn't already do more flexibly and
more simply?


Couldn't agree more:)

The following is the STL equivalent of:

print [ x*2 for range(10) in data if (x%2 == 0) ]

It will take more than digging into the STL documentation.....


#include <iostream>
#include <vector>
#include <iterator>
#include <algorithm>
#include <functional>
#include <ext/functional>
using __gnu_cxx::compose1;

class Range
{
public:
Range(int start=0) : m_start(start) {}
virtual ~Range() {}
int operator()() { return m_start++; }
private:
int m_start;
};
class Times
{
public:
Times(int val) : m_val(val) {}
int operator()(int x) const { return x*m_val; }
private:
int m_val;
};

int main(int argc, char * argv[])
{
std::vector<int> data(10);
std::generate(data.begin(), data.end(), Range(0));

std::vector<int>::iterator new_end = std::remove_if(data.begin(), data.end(), compose1(std::bind2nd(std::equal_to<int>(), 0), std::bind2nd(std::modulus<int>(), 2)));
data.erase(new_end, data.end());
std::transform(data.begin(), data.end(), data.begin(), Times(2));
std::copy(data.begin(), data.end(), std::eek:stream_iterator<int>(std::cout, "\n"));

return 0;
}
 
K

Kay Schluehr

In message <ad634d5d-
(e-mail address removed)>, Kay Schluehr
wrote:





What does the STL offer that Python doesn't already do more flexibly and
more simply?

I do not quite understand your concern? Wasn't the whole point of
Josephs post that he intended to create C++ bindings effectively s.t.
ctypes can be used?
 
R

Roy Smith

"A. Cavallo said:
The following is the STL equivalent of:

print [ x*2 for range(10) in data if (x%2 == 0) ]

Are you sure about that? I haven't tested the following code, but I
believe it is a much more direct match the the behavior of your Python code
(at least on Python 2.5.1).

#include <iostream>
int main(int argc, char * argv[])
{
std::cout << "SyntaxError: can't assign to function call";
std::cout << endl;
}
 
A

A. Cavallo

A. Cavallo said:
The following is the STL equivalent of:

print [ x*2 for range(10) in data if (x%2 == 0) ]

Are you sure about that? I haven't tested the following code, but I
believe it is a much more direct match the the behavior of your Python code
(at least on Python 2.5.1).

#include <iostream>
int main(int argc, char * argv[])
{
std::cout << "SyntaxError: can't assign to function call";
std::cout << endl;
}

Ahahaha you're right!

print [ x*2 for range(10) in data if (x%2 == 0) ] and it should work just
fine: I've still left 9 minutes to match what I've spent in order to write the
C++ version including wandering through the STL docs, compiling, fix the
syntax errors, finding the gnu STL hasn't compose1 in the expected header
file........



Regards,
Antonio

btw.
 
A

A. Cavallo

#include <iostream>
int main(int argc, char * argv[])
{
std::cout << "SyntaxError: can't assign to function call";
std::cout << endl;
}

Ops,
I've forgotten ....

a.cpp: In function ‘int main(int, char**)’:
a.cpp:5: error: ‘endl’ was not declared in this scope

Oh C++ joy!



Regards,
Antonio
 
L

Lou Pecora

Lawrence D'Oliveiro said:
Seems like the only sane way to do it. In all other directions lies madness.

Agreed. I've done this several times and it works fine. Once I'm in C
I'm really in C++ and can use all my C++ code and libraries. Not a big
problem really. But maybe I'm missing something.
 
C

Carl Banks

In message <ad634d5d-
(e-mail address removed)>, Kay Schluehr
wrote:







What does the STL offer that Python doesn't already do more flexibly and
more simply?

The opportunity to type several lines of ASCII line noise just to do
something really simple like iterate through a vector.


Carl Banks
 
R

Roy Smith

"A. Cavallo said:
#include <iostream>
int main(int argc, char * argv[])
{
std::cout << "SyntaxError: can't assign to function call";
std::cout << endl;
}

Ops,
I've forgotten ....

a.cpp: In function ‘int main(int, char**)’:
a.cpp:5: error: ‘endl’ was not declared in this scope

Oh C++ joy!

I suppose I *did* deserve that :)
 
T

Thomas Heller

Joseph said:
So I was curious whether it's possible to use the ctypes module with
C++ and if so how difficult it is.

There have been some attempts to use ctypes to access C++ objects.
We (Roman Yakovenko and myself) made some progress. We were able to
handle C++ name mangling, the special C++ calling convention,
access virtual, non-virtual, overloaded functions, but finally gave up
because the binary layout (function tables, member variables, and so on)
of C++ objects is way too complicated and undocumented.

Our attempts are documented in posts to the ctypes-users mailing list,
most of them have the word 'cpptypes' in the subject line.

Thomas
 
T

Thomas Heller

[Please keep the discussion on the list]

Joseph said:
Have you read the book Inside The C++ Object Model?:

I haven't, but considered to buy it ;-)
http://www.amazon.com/Inside-Object...=sr_1_1?ie=UTF8&s=books&qid=1244139929&sr=8-1

It's probably out of date now, but I'm about halfway through it and it
documents a ton of the little unexpected optimizations and such that
cause the binary layout to be complex. How did you get as far as you
did without having figured out the layout? (e.g. if you could access
virtual functions you must have known how to get at the virtual table)

I found a lot of material on the web, also I used the (very good) visual
studio debugger, and finally I did a lot of experimentation. We were only
able to access some very simple C++ objects.

There is also a patent or patents from MS about the vtable.

All in all, as I said, IMO it is too complicated to figure out the binary
layout of the C++ objects (without using a C++ compiler), also there are
quite some Python packages for accessing them.
 
P

Philip Semanchuk

[Please keep the discussion on the list]

Joseph said:
Have you read the book Inside The C++ Object Model?:
It's probably out of date now, but I'm about halfway through it and it
documents a ton of the little unexpected optimizations and such that
cause the binary layout to be complex. How did you get as far as you
did without having figured out the layout? (e.g. if you could access
virtual functions you must have known how to get at the virtual
table)

I found a lot of material on the web, also I used the (very good)
visual
studio debugger, and finally I did a lot of experimentation. We
were only
able to access some very simple C++ objects.

There is also a patent or patents from MS about the vtable.

All in all, as I said, IMO it is too complicated to figure out the
binary
layout of the C++ objects (without using a C++ compiler), also there
are
quite some Python packages for accessing them.

Hi Thomas,
We're weighing options for accessing C++ objects via Python. I know of
SIWG and Boost; are there others that you think deserve consideration?

I've been happy with ctypes in my limited exposure to it and would
love to find a way to make that work. This thread has been very
interesting to me.

Thanks
Philip
 
T

Thomas Heller

Philip said:
Hi Thomas,
We're weighing options for accessing C++ objects via Python. I know of
SIWG and Boost; are there others that you think deserve consideration?

I haven't used any of them myself. A common suggestion is SIP,
less known are pybindgen and Robin. But there may be many more,
and others with more experience might have much more to say about them.
Also there is Roman Yokavenko's pygccxml which has a lot of stuff.
I've been happy with ctypes in my limited exposure to it and would
love to find a way to make that work. This thread has been very
interesting to me.

Unfortunately there is no solution in sight, with ctypes.
 
P

Philip Semanchuk

What is the goal of this conversation that goes above and beyond what
Boost.Python + pygccxml achieve? Boost has published a variety of
libraries
that will be included into the next c++ standard. It's hard to
imagine a
better designed python/c++ interface library than Boost.Python.
Further,
pygccxml is amazing with regards to scanning your source code using
gcc
itself and then using that xml representation to write out the
Boost.Python
code. What more do people in this conversation want?

Hi Brian,
I've only experimented with SWIG (and ctypes for wrapping a C
library). We're not yet sold on using SWIG to wrap our C++ libraries
and so we're exploring alternatives. Before I started with SWIG, I did
some research to see what other folks were using. The arguments I
recall reading that swayed me to try SWIG before Boost were --
- Boost involves more "magic" than SWIG which means it does a bit more
work for you, but when things go wrong (i.e. interface won't compile
or doesn't work as expected) it is very difficult to debug.
- Boost-ed code requires a Boost runtime library. This isn't a
showstopper problem, obviously, but it's a mark against since it adds
yet another prerequisite to our install process.
- Boost's generated code can take a long time to compile.

I don't know what pygccxml adds to the equation, so perhaps some of
those disadvantages disappear with Boost.Python + pygccxml versus just
plain Boost.Python. If you'd like to expound on this, feel free. I'd
appreciate the education.

I don't know about what Boost (or any other tool) generates, but the
interface I got out of SWIG was not pretty. That's no knock on SWIG --
I'm amazed at what it can do. Nevertheless the generated interface has
all the charm of a Babelfish translation. I imagine lots of
autogenerated code looks "babelfish-ed": meaning is preserved, albeit
crudely, but all the idioms are lost and it's eminently clear that it
was not created by a native speaker.

Until there's an interface generation tool that can build an interface
that makes the wrapped library look completely Pythonic, then choosing
a tool will be a matter of choosing which compromises you want to
make. As long as that's true, I think there's room for multiple
library-wrapping packages, just like there's room for more than one
programming language in the world.

Cheers
Philip
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top