Cpp + Python: static data dynamic initialization in *nix shared lib?

  • Thread starter Alf P. Steinbach /Usenet
  • Start date
A

Alf P. Steinbach /Usenet

[Cross-posted comp.lang.python and comp.lang.c++]

I lack experience with shared libraries in *nix and so I need to ask...

This is about "cppy", some support for writing Python extensions in C++ that I
just started on (some days ago almost known as "pynis" (not funny after all)).

For an extension module it seems that Python requires each routine to be defined
as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern "C"' and C++
linkage, using a routine declared as 'static' in a class as a C callback,
formally they're two different kinds, and I seem to recall that /some/ C++
compiler balks at that kind of mixing unless specially instructed to allow it.
Perhaps it was the Sun compiler?

Anyway, to be formally correct I cannot generate the required C routines via
templating, and I ended up using macros that the user must explicitly invoke,
like, here the Py doc's first extension module example recoded using cppy,


------------------------------------------------------------------
<code file="spam.cpp">
#include <progrock/cppx/devsupport/better_experience.h>
#include <progrock/cppy/Module.h>
using namespace progrock;

class Spam: public cppy::Module
{
public:
Spam(): cppy::Module( "spam" )
{
setDocString( L"blåbærsyltetøy er blått" );
}

PyObject* system( PyObject* args )
{
const char *command;
int sts;

if( !PyArg_ParseTuple( args, "s", &command ) )
{
return NULL;
}
sts = ::system( command );
return Py_BuildValue( "i", sts );
}
};

CPPY_MODULE_CROUTINE( Spam, system, "Execute a shell command" )

PyMODINIT_FUNC PyInit_spam()
{
return cppy::init< Spam >();
}
</code>
------------------------------------------------------------------


It works in Windows.

But here CPPY_MODULE_CROUTINE does three things:

A Defining the 'extern "C"' routine.
I cannot think of any problem here.

B Defining installation data for that routine.
Possible problem: initializing a static with address of routine?

C -> Adding that install data record into a linked list!
Possible problem: are dynamic initialization actions guaranteed
to be performed in *nix shared library?

Problem (C) is outside the realm of the C++ standard, since the C++ standard
doesn't support shared libraries, and I've never actually used *nix shared
libraries so I don't /know/...

Is such dynamic initialization guaranteed?

For completeness, the macro definition (the 0 in there is a list next-pointer):


<code>
#define CPPY_MODULE_CROUTINE_DEF( cppClassName, name ) \
extern "C" \
static PyObject* cppClassName##_##name( PyObject*, PyObject* args ) \
{ \
return ::progrock::cppy::module<cppClassName>().name( args ); \
}

#define CPPY_MODULE_CROUTINE_INSTALLDATA( cppClassName, name, docString ) \
static ::progrock::cppy::detail::ModuleRoutineDescriptor \
cppClassName##_##name##_descriptor = { \
0, \
#name, \
docString, \
&cppClassName##_##name \
}; \
\
static bool cppClassName##_##name##_descriptor_installed = \
::progrock::cppy::detail::addToList< cppClassName >( \
cppClassName##_##name##_descriptor \
);

#define CPPY_MODULE_CROUTINE( cppClassName, name, docString ) \
CPPY_MODULE_CROUTINE_DEF( cppClassName, name ) \
CPPY_MODULE_CROUTINE_INSTALLDATA( cppClassName, name, docString )
</code>


TIA.,

- Alf
 
I

Ian Collins

[Cross-posted comp.lang.python and comp.lang.c++]

I lack experience with shared libraries in *nix and so I need to ask...

This is about "cppy", some support for writing Python extensions in C++
that I just started on (some days ago almost known as "pynis" (not funny
after all)).

For an extension module it seems that Python requires each routine to be
defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern
"C"' and C++ linkage, using a routine declared as 'static' in a class as
a C callback, formally they're two different kinds, and I seem to recall
that /some/ C++ compiler balks at that kind of mixing unless specially
instructed to allow it. Perhaps it was the Sun compiler?

Yes, it will (correctly) issue a warning.

As the is a bit OT, contact me directly and we can work through it. I
have had similar fun and games adding PHP modules!
 
A

Alf P. Steinbach /Usenet

* Ian Collins, on 09.07.2010 23:22:
[Cross-posted comp.lang.python and comp.lang.c++]

I lack experience with shared libraries in *nix and so I need to ask...

This is about "cppy", some support for writing Python extensions in C++
that I just started on (some days ago almost known as "pynis" (not funny
after all)).

For an extension module it seems that Python requires each routine to be
defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern
"C"' and C++ linkage, using a routine declared as 'static' in a class as
a C callback, formally they're two different kinds, and I seem to recall
that /some/ C++ compiler balks at that kind of mixing unless specially
instructed to allow it. Perhaps it was the Sun compiler?

Yes, it will (correctly) issue a warning.

As the is a bit OT, contact me directly and we can work through it. I
have had similar fun and games adding PHP modules!

Thanks. I'm mailing you a zip with the code... <g>

The question, of course, whether it works in *nix.


Cheers,

- Alf
 
A

Alf P. Steinbach /Usenet

* geremy condra, on 09.07.2010 23:43:
[Cross-posted comp.lang.python and comp.lang.c++]

I lack experience with shared libraries in *nix and so I need to ask...

This is about "cppy", some support for writing Python extensions in C++
that I just started on (some days ago almost known as "pynis" (not funny
after all)).

For an extension module it seems that Python requires each routine to be
defined as 'extern "C"'. And although e.g. MSVC is happy to mix 'extern
"C"' and C++ linkage, using a routine declared as 'static' in a class as
a C callback, formally they're two different kinds, and I seem to recall
that /some/ C++ compiler balks at that kind of mixing unless specially
instructed to allow it. Perhaps it was the Sun compiler?

Yes, it will (correctly) issue a warning.

As the is a bit OT, contact me directly and we can work through it. I have
had similar fun and games adding PHP modules!

I'd appreciate it if you'd either leave this on-list or cc me in on this, as
I'm working through a similar issue.

Well, we got no further, but I know of three solutions:

A) Punting: just say that the compiler has to support C++/C function type
mingling.
-> Perhaps the practical solution, but formally unsafe.

B) On the script side of things, delegate all calls to single Mother Of All
C func downcaller that supplies as extra arg an id of the C++ function.
-> Micro-level inefficient but easy to use and formally correct.

C) Let the user define the C linkage function wrappers via macros.
-> Efficient and formally correct but exposes ugly macro names.

I chose (C).

I believe Boost's Python binding uses (A), or perhaps (B).


Cheers,

- Alf

PS: You (the reader) may be wondering, why why why Yet Another Python/C++
binding? Well, because I had this great name for it, "pyni", unfortunately
already in use. But cppy is very different from Boost: Boost is large, cppy is
tiny; Boost has as main goal to expose arbitrary C++ code to Python, automating
argument conversion etc., while with cppy your Python design is exposed to C++
with no enforced arg conversions and such; Boost relies on canned magic,
difficult to subvert when it doesn't do what you want, while with cppy you are
(or, so far, I am) in control; and I suspect that the Boost Python binding,
relying on dynamic registries and stuff, is not all that efficient, while cppy
is as efficient as using the Python C API to create an extension. And besides,
cppy supports national characters in doc strings etc. And I'm Norwegian. So. :)
 
J

Jonathan Lee

Problem (C) is outside the realm of the C++ standard, since the C++ standard
doesn't support shared libraries, and I've never actually used *nix shared
libraries so I don't /know/...

Is such dynamic initialization guaranteed?

Not guaranteed, though I think there's a combination of dlopen options
and gcc command line parameters that invoke this behavior. See the
second page of

http://www.linuxjournal.com/article/3687

about auto-registration.

Personally, though, it never worked for me :/

--Jonathan
 
A

Alf P. Steinbach /Usenet

* Jonathan Lee, on 13.07.2010 16:41:
Not guaranteed, though I think there's a combination of dlopen options
and gcc command line parameters that invoke this behavior. See the
second page of

http://www.linuxjournal.com/article/3687

about auto-registration.

Personally, though, it never worked for me :/

Ah, well. :-( Thanks for the info! OK, I'll just have to replace the
auto-registration with some C++ magic. For which I think I'll simply /require/
that the compiler supports mixing of C and C++ linkage, that is, that ...


<code language="Not quite standard C++!">
#include <iostream>

extern "C"
{
typedef int (*Callback)( int );
}

void foo( Callback f ) { std::cout << "foo!" << f( 42 ) << std::endl; }

int a( int ) { return 1; }
extern "C" int b( int ) { return 2; }

int main()
{
foo( a ); // Unholy Mix of C++ and C linkage, formally not OK.
foo( b ); // Should be OK with any compiler.
}
</code>


.... compiles, and works.


Cheers, & thanks,

- Alf
 
S

sturlamolden

For an extension module it seems that Python requires each routine to be defined
as 'extern "C"'.

That is strange. PyMethodDef is just a jump table. So why should
'extern "C"' matter?

Good luck on re-inventing the wheel (you've probably heared about
Swig, SIP, Boost.Python, PyCXX, scipy.weave and Cython...)
 
A

Alf P. Steinbach /Usenet

* sturlamolden, on 13.07.2010 22:03:
That is strange. PyMethodDef is just a jump table. So why should
'extern "C"' matter?

Formally because they're incompatible function pointer types.

C++98 standard §7.5/1: "Two function types with different language linkages are
distinct types even if they are otherwise identical". Add to that §7.5/4 "A
linkage-specification shall occur only in namespace scope". And add to that
§14-4 "A template, a template explicit specialization, or a class-template
partial specialization shall not have C linkage." This means that formally
correct code that generates callbacks by templating, is ruled out.

In practice, 'extern "C"' matters for the jump tables because for those few
compilers if any where it really matters (not just the compiler emitting a
warning like reportedly Sun CC does), different linkage can imply different
machine code level calling convention. For example, who's responsible for
cleaning up the stack, the order in which arguments are pushed or which
registers they're passed in, and so forth. Ignoring such matters your code gets
into la-la land pretty fast, but, it's a different matter when one /understands/
this and places a requirement on the compiler.

Good luck on re-inventing the wheel (you've probably heared about
Swig, SIP, Boost.Python, PyCXX, scipy.weave and Cython...)

Yes, I know Boost.Python in more detail and I've heard of all the rest except
SIP, but then regarding SIP I really don't like QT (QT makes eminent sense in
the context of Python, they're both essentially dynamically typed, but that
means QT is not very nice as C++, plus there is the ugly preprocessor).

And as you'd guess if you were not in silly ignoramus assertion-mode, I'm not
reinventing the wheel.


Cheers & hth.,

- Alf
 
D

Dilip

Well, we got no further, but I know of three solutions:

   A) Punting: just say that the compiler has to support C++/C function type
      mingling.
      -> Perhaps the practical solution, but formally unsafe.

   B) On the script side of things, delegate all calls to single Mother Of All
      C func downcaller that supplies as extra arg an id of the C++ function.
      -> Micro-level inefficient but easy to use and formally correct.

   C) Let the user define the C linkage function wrappers via macros.
      -> Efficient and formally correct but exposes ugly macro names.

I chose (C).

Alf

This may or may not be what you are looking for but the middleware Ice
provides language mapping to enable Python to call into the Ice
libraries which are basically written in C++. You can take a look at
this: http://www.zeroc.com/icepy.html

However that page may not be very descriptive. The codebase, though,
is freely downloadable. You can take a look at it if that will help
although you need to wade around a little bit to figure out what is
where.
 
S

sturlamolden

In practice, 'extern "C"' matters for the jump tables because for those few
compilers if any where it really matters (not just the compiler emitting a
warning like reportedly Sun CC does), different linkage can imply different
machine code level calling convention.

I see. Just stick to MSVC and GNU and that never happens, just do a C
style cast.
Yes, I know Boost.Python in more detail and I've heard of all the rest except
SIP, but then regarding SIP I really don't like QT

You don't have to use Qt to use SIP. It's just a tool to automatically
wrap C++ for Python (like Swig, except designed for C++).

And as you'd guess if you were not in silly ignoramus assertion-mode, I'm not
reinventing the wheel.

It seems you are re-inventing PyCXX (or to a lesser extent
Boost.Python).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,770
Messages
2,569,583
Members
45,074
Latest member
StanleyFra

Latest Threads

Top