Embedding/Extending Python in/with C++: non-static members?

[dmoore]

Hi Folks:

I have a question about the use of static members in Python/C
extensions. Take the simple example from the "Extending and Embedding
the Python Interpreter" docs:

A simple module method:

static PyObject *
spam_system(PyObject *self, PyObject *args)
{
...
}

listed in a method table:

static PyMethodDef SpamMethods[] = {
...
{"system", spam_system, METH_VARARGS,
"Execute a shell command."},
...
{NULL, NULL, 0, NULL} /* Sentinel */
};

that is registered with:

PyMODINIT_FUNC
initspam(void)
{
(void) Py_InitModule("spam", SpamMethods);
}

I have an application that embed multiple interpreters. I want to be
able to register methods for each interpreter that access C++ state
data relevant to that particular interpreter. In other words I want to
be able to register non-static class members in a class like the
following:

class PythonSpamModuleInstance
{
PythonSpamModuleInstance()
{
SpamMethods[0]={"system", spam_system, METH_VARARGS,"Execute
a shell command."};
SpamMethods[1]={NULL, NULL, 0, NULL};
PyObject *spammodule=Py_InitModule("spam", SpamMethods); //
Needs an INCREF call?
}
~PythonSpamModuleInstance()
{
PyObject_Del("spam", SpamMethods);
}
PyObject *spam_system(PyObject *self, PyObject *args)
{
// accesses non-static data in this class
}
PyMethodDef SpamMethods[2];
PyObject *spammodule;
// Other data specific to this instance of the module
};

The idea is that each time my app launches an embedded Interpreter it
will create an accompanying instance of PythonSpamModuleInstance that
allows the Interpreter to call the registered non-static C++ member
function spam_system. Calls to that member function will also affect
state information.

So the questions: will this work? In particular, will the use of non-
static member functions cause problems? Is there a better way of
creating extension modules for multiple embedded interpreters? If I am
forced to use static methods, how do i figure out which interpreter is
calling them?

 

[dmoore]

(I thought I'd follow up on this post so as not to send unsuspecting
readers down a hopeless path)

duh!

I've obviously spent too much time with dynamic languages. The problem
with what I'm trying to do is obvious: In C++ you simply can't pass
pointers to call a particular instance of a C++ class method so there
is no way to initialize the PyMethodDef with class methods instances.
In other words, there is no callback mechanism built into the language
and you are stuck with template based approaches, which obviously
don't mesh very well with the Python/C API (without SWIG,
Boost:ython etc).

So to get a python interpreter instance to communicate with pre-
existing dynamically allocated C++ objects, it looks like I need to
implement some kind of lookup table approach. The static nature of the
Python/C API is just too cumbersome for use with multiple interpreters
and making me thing it might be better to embed just a sinlge
interpreter inside my app and use multi-process communication with
either pipes or sockets for the extra interpreters...

 

[Nick Craig-Wood]

I've obviously spent too much time with dynamic languages. The problem
with what I'm trying to do is obvious: In C++ you simply can't pass
pointers to call a particular instance of a C++ class method so there
is no way to initialize the PyMethodDef with class methods instances.

It is possible in theory. You can do what python does when it makes a
bound method and build a custom function at run time with the instance
and method built in.

How you build that function is tricky - you've basically got to poke
object code onto the heap which implements the method call to that
particular instance.

Looking at this page might give you some ideas

http://gcc.gnu.org/onlinedocs/gccint/Trampolines.html

This probably isn't a good approach in reality though as it is very
architecture / compiler dependent!

 

[AnonMail2005]

(I thought I'd follow up on this post so as not to send unsuspecting
readers down a hopeless path)

duh!

I've obviously spent too much time with dynamic languages. The problem
with what I'm trying to do is obvious: In C++ you simply can't pass
pointers to call a particular instance of a C++ class method so there
is no way to initialize the PyMethodDef with class methods instances.
In other words, there is no callback mechanism built into the language
and you are stuck with template based approaches, which obviously
don't mesh very well with the Python/C API (without SWIG,
Boost:ython etc).

So to get a python interpreter instance to communicate with pre-
existing dynamically allocated C++ objects, it looks like I need to
implement some kind of lookup table approach. The static nature of the
Python/C API is just too cumbersome for use with multiple interpreters
and making me thing it might be better to embed just a sinlge
interpreter inside my app and use multi-process communication with
either pipes or sockets for the extra interpreters...

I'm doing a similar thing, and I would imagine others are also.

1. In a python file, set up wrapper functions that the python user
actually uses (e.g FunctionA). Each function corresponds to a
particular C/C++ extension function that you are exposing (e.g.
CFunctionA).

2. Each wrapper function takes the same number of args as the C/C++
extension function except it adds one additional argument - the
interperter
identifier.

3. The interpreter identifier is just a global id (e.g. an integer)
that
is unique for each interpreter and is set before hand. We set the id
by
first loading the wrapper function python file (in C/C++) and calling
a
predefined function that sets the id (you just pass the id as an
argument
to the function).

4. Now when you enter you C/C++ function you can find your object by
looking
it up in some predefined table using the id. What we actually do is
not use
an integer but instead use a void pointer which is cast appropriately
once
inside C/C++. This avoids the lookup.

Hope that helps.

 

[AnonMail2005]

Hi Folks:

I have a question about the use of static members in Python/C
extensions. Take the simple example from the "Extending and Embedding
the Python Interpreter" docs:

A simple module method:

static PyObject *
spam_system(PyObject *self, PyObject *args)
{
...

}

listed in a method table:

static PyMethodDef SpamMethods[] = {
...
{"system", spam_system, METH_VARARGS,
"Execute a shell command."},
...
{NULL, NULL, 0, NULL} /* Sentinel */

};

that is registered with:

PyMODINIT_FUNC
initspam(void)
{
(void) Py_InitModule("spam", SpamMethods);

}

I have an application that embed multiple interpreters. I want to be
able to register methods for each interpreter that access C++ state
data relevant to that particular interpreter. In other words I want to
be able to register non-static class members in a class like the
following:

class PythonSpamModuleInstance
{
PythonSpamModuleInstance()
{
SpamMethods[0]={"system", spam_system, METH_VARARGS,"Execute
a shell command."};
SpamMethods[1]={NULL, NULL, 0, NULL};
PyObject *spammodule=Py_InitModule("spam", SpamMethods); //
Needs an INCREF call?
}
~PythonSpamModuleInstance()
{
PyObject_Del("spam", SpamMethods);
}
PyObject *spam_system(PyObject *self, PyObject *args)
{
// accesses non-static data in this class
}
PyMethodDef SpamMethods[2];
PyObject *spammodule;
// Other data specific to this instance of the module

};

The idea is that each time my app launches an embedded Interpreter it
will create an accompanying instance of PythonSpamModuleInstance that
allows the Interpreter to call the registered non-static C++ member
function spam_system. Calls to that member function will also affect
state information.

So the questions: will this work? In particular, will the use of non-
static member functions cause problems? Is there a better way of
creating extension modules for multiple embedded interpreters? If I am
forced to use static methods, how do i figure out which interpreter is
calling them?

test posting.

 

[dmoore]

thanks for the responses Nick and "AnonMail"

I'm doing a similar thing, and I would imagine others are also.

1. In a python file, set up wrapper functions that the python user
actually uses (e.g FunctionA). Each function corresponds to a
particular C/C++ extension function that you are exposing (e.g.
CFunctionA).
....
4. Now when you enter you C/C++ function you can find your object by
looking
it up in some predefined table using the id. What we actually do is
not use
an integer but instead use a void pointer which is cast appropriately
once
inside C/C++. This avoids the lookup.

step 4 is smart

It's good to know that there is at least one data point of success
using the approach. I won't give up just yet.