Using a dictionary to pass data to/from embedded python functions

[wardm]

I have created a Dict object in a C++ App that calls (embedded) Python
functions.
The Dict is to be used to pass variable data between the C++ App and the
python functions.

However I cannot get the Python functions to 'see' the Dict created in the
C++ App.

The C++ app creates the Dict using the following functions:

// Create the dict for the viable data
m_pVarDictionary = PyDict_New();

// Create interface module
m_pInterfaceModule = PyImport_AddModule("InterfaceModule");


// Add the Dictionary to the interface Module
int status = PyModule_AddObject(m_pInterfaceModule, "VarDictionary" ,
m_pVarDictionary);


// Load the script module
m_pScriptModule = PyImport_ImportModule("MyScriptModule");

The C++ app calls the functions using the following functions:

PyObject* func = PyObject_GetAttrString(m_pScriptModule, "functionName");

if (func && PyCallable_Check(func))
{

PyObject* ret = PyObject_CallObject(func, NULL);

}


My test python function looks like this:


"""----------------------------------------------------------------------
Simple scripts for testing Dict Access
-------------------------------------------------------------------------"""
from InterfaceModule import VarDictionary

def hello():


# dumps the contents of VarDictionary
d2 = VarDictionary
f1.write(len(d2))
f1.write('\n')

for k, v in d2.iteritems():
f1.write(k)
f1.write (':')
f1.write(str(v))
f1.write('\n')

f1.close()


This python code throws an exception when it attempts to access the
"VarDictionary".
Does anyone know why this fails ?

I have tried adding code that dumps the Dict returned from vars(),
and could not see "VarDictionary" as an entry in the Vars() dict.

 

[Alex Martelli]

I have created a Dict object in a C++ App that calls (embedded) Python
functions. [[snip snip]]
This python code throws an exception when it attempts to access the
"VarDictionary".
Does anyone know why this fails ?

It fails due to some code of yours that you have not included: in other
words, the minimal application which embeds Python using _only_ the code
you show us does not exhibit the failure.

I wrote the following file za.c:

#include "Python.h"
#include "stdio.h"
#include "stdlib.h"

int main()
{
printf("start\n");
putenv("PYTHONPATH=.");
Py_Initialize();
printf("inited\n");
PyObject* m_pVarDictionary = PyDict_New();
printf("dict is %p\n", m_pVarDictionary);
PyObject* m_pInterfaceModule =
PyImport_AddModule("InterfaceModule");
printf("modu is %p\n", m_pInterfaceModule);
int status = PyModule_AddObject(m_pInterfaceModule, "VarDictionary"
,
m_pVarDictionary);
printf("stat is %d\n", status);
PyObject* m_pScriptModule = PyImport_ImportModule("MyScriptModule");
printf("impo is %p\n", m_pScriptModule);

PyObject* func = PyObject_GetAttrString(m_pScriptModule,
"functionName");
printf("func is %p\n", func);
if (func && PyCallable_Check(func)) {
PyObject* ret = PyObject_CallObject(func, NULL);
printf("retu is %p\n", ret);
}
printf("done\n");
return 0;
}

and the following file MyScriptModule.py:

import InterfaceModule

def functionName():
print "hello"
print dir(InterfaceModule)
print "that's all"
return

and proceeded to compile and execute as follows: [[Note: it does not
matter that I'm using 2.5, the code is just as fine with previous
versions -- it just happens that 2.5 is what I'm using right now in
order to help out with 2.5's beta testing]]:

brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
-lpython2.5
brain:~/pyex alex$ ./za

and observed exactly the kind of output I predicted [[Note: the exact
addresses printed of course do not matter]]:

start
inited
dict is 0x51c780
modu is 0x520230
stat is 0
impo is 0x5202d0
func is 0x513770
hello
['VarDictionary', '__doc__', '__name__']
that's all
retu is 0xe57c0
done
brain:~/pyex alex$

As you see, in particular, VarDictionary is right up there in
InterfaceModule's dir.


There's a well-known essay by Eric Raymond, "How to ask questions the
smart way", at <http://catb.org/~esr/faqs/smart-questions.html> -- it
seems to me that you, quite commendably, follow most of Eric's advice,
but it's still worth reading -- the key point by which you could help us
to help you is what Eric mentions as "If you have a large, complicated
test case that is breaking a program, try to trim it and make it as
small as possible".

In this case, you should try to trim your code down to the smallest
program using this approach, which you believe should work in a certain
way but actually doesn't. The exact code you posted plus the minimal
additions to make it a compilable program does work the way you appear
to desire, as I show above; therefore, there must be something else in
your code that is breaking things. Do tiny steps of addition and
restructuring to move this minimal skeleton towards the direction of
your bigger program (that does not behave this way) until you've found
exactly the "largest" version that still succeeds, and the minisculely
larger "smallest" version that fails -- the tiny difference between the
two must then be the root of the problem, and if the reason is not clear
at that point then posting here again is exactly the right thing to do.

Perhaps you're falling afoul of the fact that (as documented e.g. at
<http://docs.python.org/api/moduleObjects.html>) PyModule_AddObject is a
"convenience function" that steals a reference to the value -- so you
end up with just one reference to the dictionary object, and if for some
reason you decref it, the object gets garbage collected. But, that's
just a wild guess on my part, since you have not shown us any code
performing (for example) any decrefs.


Alex

 

[wardm]

Thanks Alex for your help, (and advice on focusing the point of my
question).

I was able to compile and run your example OK, but when I try to use the
"VarDictionary" in the
MyScriptModule.py code, I get an exception.

I added the following code to the C app just to add two entries to the
Dictionary

PyDict_SetItemString( m_pVarDictionary, "tk1",
Py_BuildValue("s","test1Val"));
PyDict_SetItemString( m_pVarDictionary, "tk2",
Py_BuildValue("s","test2Val"));

Then tried various things in the Python code to display the contents of the
"VarDictionary",
such as adding the "print VarDictionary" below.

import InterfaceModule

def functionName():
print "hello"
print dir(InterfaceModule)
print "that's all"
print VarDictionary
return

Even though "VarDictionary " is in the Dir, every time I try to use the
"VarDictionary" the program fails.
Am I doing something wrong when I try and reference "VarDictionary" in
Python ?
I need to be able to get/set entries in VarDictionary from the Python
function.

I have created a Dict object in a C++ App that calls (embedded) Python
functions. [[snip snip]]
This python code throws an exception when it attempts to access the
"VarDictionary".
Does anyone know why this fails ?

It fails due to some code of yours that you have not included: in other
words, the minimal application which embeds Python using _only_ the code
you show us does not exhibit the failure.

I wrote the following file za.c:

#include "Python.h"
#include "stdio.h"
#include "stdlib.h"

int main()
{
printf("start\n");
putenv("PYTHONPATH=.");
Py_Initialize();
printf("inited\n");
PyObject* m_pVarDictionary = PyDict_New();
printf("dict is %p\n", m_pVarDictionary);
PyObject* m_pInterfaceModule =
PyImport_AddModule("InterfaceModule");
printf("modu is %p\n", m_pInterfaceModule);
int status = PyModule_AddObject(m_pInterfaceModule, "VarDictionary"
,
m_pVarDictionary);
printf("stat is %d\n", status);
PyObject* m_pScriptModule = PyImport_ImportModule("MyScriptModule");
printf("impo is %p\n", m_pScriptModule);

PyObject* func = PyObject_GetAttrString(m_pScriptModule,
"functionName");
printf("func is %p\n", func);
if (func && PyCallable_Check(func)) {
PyObject* ret = PyObject_CallObject(func, NULL);
printf("retu is %p\n", ret);
}
printf("done\n");
return 0;
}

and the following file MyScriptModule.py:

import InterfaceModule

def functionName():
print "hello"
print dir(InterfaceModule)
print "that's all"
return

and proceeded to compile and execute as follows: [[Note: it does not
matter that I'm using 2.5, the code is just as fine with previous
versions -- it just happens that 2.5 is what I'm using right now in
order to help out with 2.5's beta testing]]:

brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
-lpython2.5
brain:~/pyex alex$ ./za

and observed exactly the kind of output I predicted [[Note: the exact
addresses printed of course do not matter]]:

start
inited
dict is 0x51c780
modu is 0x520230
stat is 0
impo is 0x5202d0
func is 0x513770
hello
['VarDictionary', '__doc__', '__name__']
that's all
retu is 0xe57c0
done
brain:~/pyex alex$

As you see, in particular, VarDictionary is right up there in
InterfaceModule's dir.


There's a well-known essay by Eric Raymond, "How to ask questions the
smart way", at <http://catb.org/~esr/faqs/smart-questions.html> -- it
seems to me that you, quite commendably, follow most of Eric's advice,
but it's still worth reading -- the key point by which you could help us
to help you is what Eric mentions as "If you have a large, complicated
test case that is breaking a program, try to trim it and make it as
small as possible".

In this case, you should try to trim your code down to the smallest
program using this approach, which you believe should work in a certain
way but actually doesn't. The exact code you posted plus the minimal
additions to make it a compilable program does work the way you appear
to desire, as I show above; therefore, there must be something else in
your code that is breaking things. Do tiny steps of addition and
restructuring to move this minimal skeleton towards the direction of
your bigger program (that does not behave this way) until you've found
exactly the "largest" version that still succeeds, and the minisculely
larger "smallest" version that fails -- the tiny difference between the
two must then be the root of the problem, and if the reason is not clear
at that point then posting here again is exactly the right thing to do.

Perhaps you're falling afoul of the fact that (as documented e.g. at
<http://docs.python.org/api/moduleObjects.html>) PyModule_AddObject is a
"convenience function" that steals a reference to the value -- so you
end up with just one reference to the dictionary object, and if for some
reason you decref it, the object gets garbage collected. But, that's
just a wild guess on my part, since you have not shown us any code
performing (for example) any decrefs.


Alex

 

[Alex Martelli]

Thanks Alex for your help, (and advice on focusing the point of my
question).

I was able to compile and run your example OK, but when I try to use the
"VarDictionary" in the
MyScriptModule.py code, I get an exception.

I added the following code to the C app just to add two entries to the
Dictionary

PyDict_SetItemString( m_pVarDictionary, "tk1",
Py_BuildValue("s","test1Val"));
PyDict_SetItemString( m_pVarDictionary, "tk2",
Py_BuildValue("s","test2Val"));

Then tried various things in the Python code to display the contents of the
"VarDictionary",
such as adding the "print VarDictionary" below.

import InterfaceModule

def functionName():
print "hello"
print dir(InterfaceModule)
print "that's all"
print VarDictionary

Note the wrong indentation in this latter print statement: this would
already cause a syntax error (unless the leading 'p' happened to be
aligned with the leading 'd' of 'def', in which case the function would
be terminated, the latest print would happen at import-time, and the
FOLLOWING statement:

return

....would then be a syntax error (return outside of function). But,
there's more:

Even though "VarDictionary " is in the Dir, every time I try to use the
"VarDictionary" the program fails.

"VarDictionary" is in the dir(...) *** of InterfaceModule ***, of
course, so you need to refer to it as InterfaceModule.VarDictionary in
your Python code -- the barename, nor qualified by modulename, just will
not work, of course!!!

Adding the two C code lines you quote, and changing the Python example
code to:

def functionName():
print "hello"
print dir(InterfaceModule)
print "VarDictionary is:", InterfaceModule.VarDictionary
print "that's all"

changes that part of the output to:

hello
['VarDictionary', '__doc__', '__name__']
VarDictionary is: {'tk2': 'test2Val', 'tk1': 'test1Val'}
that's all


With all due respect, it looks like you're trying to run before you can
walk -- or more specifically, to embed Python in C++ before you become
familiar with the most elementary and fundamental aspects of Python,
such as indentation and the need to qualify compound names. You might
want to consider getting a good Python book -- such as, my own Python in
A Nutshell (2nd ed), Aahz and Stef Maruch's Python For Dummies, Wesley
Chun's Core Python Programming (2nd ed) -- they're all very recent (mine
is just out, Aahz's and Stef's I believe is due to hit bookstores in
September or October), and any of them might serve you well (if you're
OK with books not necessarily covering the very latest release of Python
[and the issues you're having suggest that this is not really the
problem!], there are many other good books, such as Magnus Lie Hetland's
"Beginning Python", Beazley's "Python Essential Reference", Lutz and
Ascher's "Learning Python",

 

[wardm]

Thanks again for your help, I agree, it seems I need to read a good book on
Python.

One last question, will Python allow me to add new items to
InterfaceModule.VarDictionary
from the Python functions I call ?

Thanks Alex for your help, (and advice on focusing the point of my
question).

I was able to compile and run your example OK, but when I try to use the
"VarDictionary" in the
MyScriptModule.py code, I get an exception.

I added the following code to the C app just to add two entries to the
Dictionary

PyDict_SetItemString( m_pVarDictionary, "tk1",
Py_BuildValue("s","test1Val"));
PyDict_SetItemString( m_pVarDictionary, "tk2",
Py_BuildValue("s","test2Val"));

Then tried various things in the Python code to display the contents of
the
"VarDictionary",
such as adding the "print VarDictionary" below.

import InterfaceModule

def functionName():
print "hello"
print dir(InterfaceModule)
print "that's all"
print VarDictionary

Note the wrong indentation in this latter print statement: this would
already cause a syntax error (unless the leading 'p' happened to be
aligned with the leading 'd' of 'def', in which case the function would
be terminated, the latest print would happen at import-time, and the
FOLLOWING statement:

return

...would then be a syntax error (return outside of function). But,
there's more:

Even though "VarDictionary " is in the Dir, every time I try to use the
"VarDictionary" the program fails.

"VarDictionary" is in the dir(...) *** of InterfaceModule ***, of
course, so you need to refer to it as InterfaceModule.VarDictionary in
your Python code -- the barename, nor qualified by modulename, just will
not work, of course!!!

Adding the two C code lines you quote, and changing the Python example
code to:

def functionName():
print "hello"
print dir(InterfaceModule)
print "VarDictionary is:", InterfaceModule.VarDictionary
print "that's all"

changes that part of the output to:

hello
['VarDictionary', '__doc__', '__name__']
VarDictionary is: {'tk2': 'test2Val', 'tk1': 'test1Val'}
that's all


With all due respect, it looks like you're trying to run before you can
walk -- or more specifically, to embed Python in C++ before you become
familiar with the most elementary and fundamental aspects of Python,
such as indentation and the need to qualify compound names. You might
want to consider getting a good Python book -- such as, my own Python in
A Nutshell (2nd ed), Aahz and Stef Maruch's Python For Dummies, Wesley
Chun's Core Python Programming (2nd ed) -- they're all very recent (mine
is just out, Aahz's and Stef's I believe is due to hit bookstores in
September or October), and any of them might serve you well (if you're
OK with books not necessarily covering the very latest release of Python
[and the issues you're having suggest that this is not really the
problem!], there are many other good books, such as Magnus Lie Hetland's
"Beginning Python", Beazley's "Python Essential Reference", Lutz and
Ascher's "Learning Python",

 

[Alex Martelli]

Thanks again for your help, I agree, it seems I need to read a good book on
Python.

One last question, will Python allow me to add new items to
InterfaceModule.VarDictionary
from the Python functions I call ?

Yes, no problem with that.


Alex

 

[Bill Pursell]

I wrote the following file za.c:

and proceeded to compile and execute as follows: [[Note: it does not
matter that I'm using 2.5, the code is just as fine with previous
versions -- it just happens that 2.5 is what I'm using right now in
order to help out with 2.5's beta testing]]:

brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
-lpython2.5
brain:~/pyex alex$ ./za

I needed to include a lot more flags to make the example compile.
In particular:
-L /usr/local/lib/python2.4/config/ -lpython2.4 -lrt -lm -ldl -lutil

Why did you not need them? Is this a misconfiguration on my
box, or something entirely different? I've seen this type of thing
come up a lot, and I can't tell if it is simply the author omitting
flags for the sake of brevity, or if I'm actually missing something.

 

[Alex Martelli]

I wrote the following file za.c:

and proceeded to compile and execute as follows: [[Note: it does not
matter that I'm using 2.5, the code is just as fine with previous
versions -- it just happens that 2.5 is what I'm using right now in
order to help out with 2.5's beta testing]]:

brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
-lpython2.5
brain:~/pyex alex$ ./za

I needed to include a lot more flags to make the example compile.
In particular:
-L /usr/local/lib/python2.4/config/ -lpython2.4 -lrt -lm -ldl -lutil

You no doubt needed that for linking, not for compiling (I needed the -L
and -l too for linking, see above, just not the "other" libraries).

Why did you not need them? Is this a misconfiguration on my
box, or something entirely different? I've seen this type of thing
come up a lot, and I can't tell if it is simply the author omitting
flags for the sake of brevity, or if I'm actually missing something.

Neither: we're just talking about different platforms -- probably both
Unix variants, but with different linking-loader behavior and/or
dependencies among libraries. Considering that my platform's ld is
doubtlessly the single most prominent difference between it and other
Unix variants from a developer's viewpoint (see
<http://developer.apple.com/documentation/Darwin/Reference/ManPages/man1
/ld.1.html> for more details) I was remiss in not mentioning my
platform; I apologize, I was concentrating on the C++ and Python sources
rather than on the build process (but I should have mentioned it in the
Note, just as I mentioned that I was using 2.5 but it didn't matter!).


Alex