Using a dictionary to pass data to/from embedded python functions

Discussion in 'Python' started by wardm, Aug 11, 2006.

  1. wardm

    wardm Guest

    I have created a Dict object in a C++ App that calls (embedded) Python
    functions.
    The Dict is to be used to pass variable data between the C++ App and the
    python functions.

    However I cannot get the Python functions to 'see' the Dict created in the
    C++ App.

    The C++ app creates the Dict using the following functions:

    // Create the dict for the viable data
    m_pVarDictionary = PyDict_New();

    // Create interface module
    m_pInterfaceModule = PyImport_AddModule("InterfaceModule");


    // Add the Dictionary to the interface Module
    int status = PyModule_AddObject(m_pInterfaceModule, "VarDictionary" ,
    m_pVarDictionary);


    // Load the script module
    m_pScriptModule = PyImport_ImportModule("MyScriptModule");

    The C++ app calls the functions using the following functions:

    PyObject* func = PyObject_GetAttrString(m_pScriptModule, "functionName");

    if (func && PyCallable_Check(func))
    {

    PyObject* ret = PyObject_CallObject(func, NULL);

    }


    My test python function looks like this:


    """----------------------------------------------------------------------
    Simple scripts for testing Dict Access
    -------------------------------------------------------------------------"""
    from InterfaceModule import VarDictionary

    def hello():


    # dumps the contents of VarDictionary
    d2 = VarDictionary
    f1.write(len(d2))
    f1.write('\n')

    for k, v in d2.iteritems():
    f1.write(k)
    f1.write (':')
    f1.write(str(v))
    f1.write('\n')

    f1.close()


    This python code throws an exception when it attempts to access the
    "VarDictionary".
    Does anyone know why this fails ?

    I have tried adding code that dumps the Dict returned from vars(),
    and could not see "VarDictionary" as an entry in the Vars() dict.
    wardm, Aug 11, 2006
    #1
    1. Advertising

  2. wardm <> wrote:

    > I have created a Dict object in a C++ App that calls (embedded) Python
    > functions.

    [[snip snip]]
    > This python code throws an exception when it attempts to access the
    > "VarDictionary".
    > Does anyone know why this fails ?


    It fails due to some code of yours that you have not included: in other
    words, the minimal application which embeds Python using _only_ the code
    you show us does not exhibit the failure.

    I wrote the following file za.c:

    #include "Python.h"
    #include "stdio.h"
    #include "stdlib.h"

    int main()
    {
    printf("start\n");
    putenv("PYTHONPATH=.");
    Py_Initialize();
    printf("inited\n");
    PyObject* m_pVarDictionary = PyDict_New();
    printf("dict is %p\n", m_pVarDictionary);
    PyObject* m_pInterfaceModule =
    PyImport_AddModule("InterfaceModule");
    printf("modu is %p\n", m_pInterfaceModule);
    int status = PyModule_AddObject(m_pInterfaceModule, "VarDictionary"
    ,
    m_pVarDictionary);
    printf("stat is %d\n", status);
    PyObject* m_pScriptModule = PyImport_ImportModule("MyScriptModule");
    printf("impo is %p\n", m_pScriptModule);

    PyObject* func = PyObject_GetAttrString(m_pScriptModule,
    "functionName");
    printf("func is %p\n", func);
    if (func && PyCallable_Check(func)) {
    PyObject* ret = PyObject_CallObject(func, NULL);
    printf("retu is %p\n", ret);
    }
    printf("done\n");
    return 0;
    }

    and the following file MyScriptModule.py:

    import InterfaceModule

    def functionName():
    print "hello"
    print dir(InterfaceModule)
    print "that's all"
    return

    and proceeded to compile and execute as follows: [[Note: it does not
    matter that I'm using 2.5, the code is just as fine with previous
    versions -- it just happens that 2.5 is what I'm using right now in
    order to help out with 2.5's beta testing]]:

    brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
    brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
    -lpython2.5
    brain:~/pyex alex$ ./za

    and observed exactly the kind of output I predicted [[Note: the exact
    addresses printed of course do not matter]]:

    start
    inited
    dict is 0x51c780
    modu is 0x520230
    stat is 0
    impo is 0x5202d0
    func is 0x513770
    hello
    ['VarDictionary', '__doc__', '__name__']
    that's all
    retu is 0xe57c0
    done
    brain:~/pyex alex$

    As you see, in particular, VarDictionary is right up there in
    InterfaceModule's dir.


    There's a well-known essay by Eric Raymond, "How to ask questions the
    smart way", at <http://catb.org/~esr/faqs/smart-questions.html> -- it
    seems to me that you, quite commendably, follow most of Eric's advice,
    but it's still worth reading -- the key point by which you could help us
    to help you is what Eric mentions as "If you have a large, complicated
    test case that is breaking a program, try to trim it and make it as
    small as possible".

    In this case, you should try to trim your code down to the smallest
    program using this approach, which you believe should work in a certain
    way but actually doesn't. The exact code you posted plus the minimal
    additions to make it a compilable program does work the way you appear
    to desire, as I show above; therefore, there must be something else in
    your code that is breaking things. Do tiny steps of addition and
    restructuring to move this minimal skeleton towards the direction of
    your bigger program (that does not behave this way) until you've found
    exactly the "largest" version that still succeeds, and the minisculely
    larger "smallest" version that fails -- the tiny difference between the
    two must then be the root of the problem, and if the reason is not clear
    at that point then posting here again is exactly the right thing to do.

    Perhaps you're falling afoul of the fact that (as documented e.g. at
    <http://docs.python.org/api/moduleObjects.html>) PyModule_AddObject is a
    "convenience function" that steals a reference to the value -- so you
    end up with just one reference to the dictionary object, and if for some
    reason you decref it, the object gets garbage collected. But, that's
    just a wild guess on my part, since you have not shown us any code
    performing (for example) any decrefs.


    Alex
    Alex Martelli, Aug 12, 2006
    #2
    1. Advertising

  3. wardm

    wardm Guest

    Thanks Alex for your help, (and advice on focusing the point of my
    question).

    I was able to compile and run your example OK, but when I try to use the
    "VarDictionary" in the
    MyScriptModule.py code, I get an exception.

    I added the following code to the C app just to add two entries to the
    Dictionary

    PyDict_SetItemString( m_pVarDictionary, "tk1",
    Py_BuildValue("s","test1Val"));
    PyDict_SetItemString( m_pVarDictionary, "tk2",
    Py_BuildValue("s","test2Val"));

    Then tried various things in the Python code to display the contents of the
    "VarDictionary",
    such as adding the "print VarDictionary" below.

    import InterfaceModule

    def functionName():
    print "hello"
    print dir(InterfaceModule)
    print "that's all"
    print VarDictionary
    return

    Even though "VarDictionary " is in the Dir, every time I try to use the
    "VarDictionary" the program fails.
    Am I doing something wrong when I try and reference "VarDictionary" in
    Python ?
    I need to be able to get/set entries in VarDictionary from the Python
    function.


    "Alex Martelli" <> wrote in message
    news:1hjy7rx.1sxrihq1jtyy68N%...
    > wardm <> wrote:
    >
    >> I have created a Dict object in a C++ App that calls (embedded) Python
    >> functions.

    > [[snip snip]]
    >> This python code throws an exception when it attempts to access the
    >> "VarDictionary".
    >> Does anyone know why this fails ?

    >
    > It fails due to some code of yours that you have not included: in other
    > words, the minimal application which embeds Python using _only_ the code
    > you show us does not exhibit the failure.
    >
    > I wrote the following file za.c:
    >
    > #include "Python.h"
    > #include "stdio.h"
    > #include "stdlib.h"
    >
    > int main()
    > {
    > printf("start\n");
    > putenv("PYTHONPATH=.");
    > Py_Initialize();
    > printf("inited\n");
    > PyObject* m_pVarDictionary = PyDict_New();
    > printf("dict is %p\n", m_pVarDictionary);
    > PyObject* m_pInterfaceModule =
    > PyImport_AddModule("InterfaceModule");
    > printf("modu is %p\n", m_pInterfaceModule);
    > int status = PyModule_AddObject(m_pInterfaceModule, "VarDictionary"
    > ,
    > m_pVarDictionary);
    > printf("stat is %d\n", status);
    > PyObject* m_pScriptModule = PyImport_ImportModule("MyScriptModule");
    > printf("impo is %p\n", m_pScriptModule);
    >
    > PyObject* func = PyObject_GetAttrString(m_pScriptModule,
    > "functionName");
    > printf("func is %p\n", func);
    > if (func && PyCallable_Check(func)) {
    > PyObject* ret = PyObject_CallObject(func, NULL);
    > printf("retu is %p\n", ret);
    > }
    > printf("done\n");
    > return 0;
    > }
    >
    > and the following file MyScriptModule.py:
    >
    > import InterfaceModule
    >
    > def functionName():
    > print "hello"
    > print dir(InterfaceModule)
    > print "that's all"
    > return
    >
    > and proceeded to compile and execute as follows: [[Note: it does not
    > matter that I'm using 2.5, the code is just as fine with previous
    > versions -- it just happens that 2.5 is what I'm using right now in
    > order to help out with 2.5's beta testing]]:
    >
    > brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
    > brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
    > -lpython2.5
    > brain:~/pyex alex$ ./za
    >
    > and observed exactly the kind of output I predicted [[Note: the exact
    > addresses printed of course do not matter]]:
    >
    > start
    > inited
    > dict is 0x51c780
    > modu is 0x520230
    > stat is 0
    > impo is 0x5202d0
    > func is 0x513770
    > hello
    > ['VarDictionary', '__doc__', '__name__']
    > that's all
    > retu is 0xe57c0
    > done
    > brain:~/pyex alex$
    >
    > As you see, in particular, VarDictionary is right up there in
    > InterfaceModule's dir.
    >
    >
    > There's a well-known essay by Eric Raymond, "How to ask questions the
    > smart way", at <http://catb.org/~esr/faqs/smart-questions.html> -- it
    > seems to me that you, quite commendably, follow most of Eric's advice,
    > but it's still worth reading -- the key point by which you could help us
    > to help you is what Eric mentions as "If you have a large, complicated
    > test case that is breaking a program, try to trim it and make it as
    > small as possible".
    >
    > In this case, you should try to trim your code down to the smallest
    > program using this approach, which you believe should work in a certain
    > way but actually doesn't. The exact code you posted plus the minimal
    > additions to make it a compilable program does work the way you appear
    > to desire, as I show above; therefore, there must be something else in
    > your code that is breaking things. Do tiny steps of addition and
    > restructuring to move this minimal skeleton towards the direction of
    > your bigger program (that does not behave this way) until you've found
    > exactly the "largest" version that still succeeds, and the minisculely
    > larger "smallest" version that fails -- the tiny difference between the
    > two must then be the root of the problem, and if the reason is not clear
    > at that point then posting here again is exactly the right thing to do.
    >
    > Perhaps you're falling afoul of the fact that (as documented e.g. at
    > <http://docs.python.org/api/moduleObjects.html>) PyModule_AddObject is a
    > "convenience function" that steals a reference to the value -- so you
    > end up with just one reference to the dictionary object, and if for some
    > reason you decref it, the object gets garbage collected. But, that's
    > just a wild guess on my part, since you have not shown us any code
    > performing (for example) any decrefs.
    >
    >
    > Alex
    wardm, Aug 12, 2006
    #3
  4. wardm <> wrote:

    > Thanks Alex for your help, (and advice on focusing the point of my
    > question).
    >
    > I was able to compile and run your example OK, but when I try to use the
    > "VarDictionary" in the
    > MyScriptModule.py code, I get an exception.
    >
    > I added the following code to the C app just to add two entries to the
    > Dictionary
    >
    > PyDict_SetItemString( m_pVarDictionary, "tk1",
    > Py_BuildValue("s","test1Val"));
    > PyDict_SetItemString( m_pVarDictionary, "tk2",
    > Py_BuildValue("s","test2Val"));
    >
    > Then tried various things in the Python code to display the contents of the
    > "VarDictionary",
    > such as adding the "print VarDictionary" below.
    >
    > import InterfaceModule
    >
    > def functionName():
    > print "hello"
    > print dir(InterfaceModule)
    > print "that's all"
    > print VarDictionary


    Note the wrong indentation in this latter print statement: this would
    already cause a syntax error (unless the leading 'p' happened to be
    aligned with the leading 'd' of 'def', in which case the function would
    be terminated, the latest print would happen at import-time, and the
    FOLLOWING statement:

    > return


    ....would then be a syntax error (return outside of function). But,
    there's more:


    > Even though "VarDictionary " is in the Dir, every time I try to use the
    > "VarDictionary" the program fails.


    "VarDictionary" is in the dir(...) *** of InterfaceModule ***, of
    course, so you need to refer to it as InterfaceModule.VarDictionary in
    your Python code -- the barename, nor qualified by modulename, just will
    not work, of course!!!

    Adding the two C code lines you quote, and changing the Python example
    code to:

    def functionName():
    print "hello"
    print dir(InterfaceModule)
    print "VarDictionary is:", InterfaceModule.VarDictionary
    print "that's all"

    changes that part of the output to:

    hello
    ['VarDictionary', '__doc__', '__name__']
    VarDictionary is: {'tk2': 'test2Val', 'tk1': 'test1Val'}
    that's all


    With all due respect, it looks like you're trying to run before you can
    walk -- or more specifically, to embed Python in C++ before you become
    familiar with the most elementary and fundamental aspects of Python,
    such as indentation and the need to qualify compound names. You might
    want to consider getting a good Python book -- such as, my own Python in
    A Nutshell (2nd ed), Aahz and Stef Maruch's Python For Dummies, Wesley
    Chun's Core Python Programming (2nd ed) -- they're all very recent (mine
    is just out, Aahz's and Stef's I believe is due to hit bookstores in
    September or October), and any of them might serve you well (if you're
    OK with books not necessarily covering the very latest release of Python
    [and the issues you're having suggest that this is not really the
    problem!], there are many other good books, such as Magnus Lie Hetland's
    "Beginning Python", Beazley's "Python Essential Reference", Lutz and
    Ascher's "Learning Python",
    Alex Martelli, Aug 13, 2006
    #4
  5. wardm

    wardm Guest

    Thanks again for your help, I agree, it seems I need to read a good book on
    Python.

    One last question, will Python allow me to add new items to
    InterfaceModule.VarDictionary
    from the Python functions I call ?


    "Alex Martelli" <> wrote in message
    news:1hjytek.1gh1ur81so4q0tN%...
    > wardm <> wrote:
    >
    >> Thanks Alex for your help, (and advice on focusing the point of my
    >> question).
    >>
    >> I was able to compile and run your example OK, but when I try to use the
    >> "VarDictionary" in the
    >> MyScriptModule.py code, I get an exception.
    >>
    >> I added the following code to the C app just to add two entries to the
    >> Dictionary
    >>
    >> PyDict_SetItemString( m_pVarDictionary, "tk1",
    >> Py_BuildValue("s","test1Val"));
    >> PyDict_SetItemString( m_pVarDictionary, "tk2",
    >> Py_BuildValue("s","test2Val"));
    >>
    >> Then tried various things in the Python code to display the contents of
    >> the
    >> "VarDictionary",
    >> such as adding the "print VarDictionary" below.
    >>
    >> import InterfaceModule
    >>
    >> def functionName():
    >> print "hello"
    >> print dir(InterfaceModule)
    >> print "that's all"
    >> print VarDictionary

    >
    > Note the wrong indentation in this latter print statement: this would
    > already cause a syntax error (unless the leading 'p' happened to be
    > aligned with the leading 'd' of 'def', in which case the function would
    > be terminated, the latest print would happen at import-time, and the
    > FOLLOWING statement:
    >
    >> return

    >
    > ...would then be a syntax error (return outside of function). But,
    > there's more:
    >
    >
    >> Even though "VarDictionary " is in the Dir, every time I try to use the
    >> "VarDictionary" the program fails.

    >
    > "VarDictionary" is in the dir(...) *** of InterfaceModule ***, of
    > course, so you need to refer to it as InterfaceModule.VarDictionary in
    > your Python code -- the barename, nor qualified by modulename, just will
    > not work, of course!!!
    >
    > Adding the two C code lines you quote, and changing the Python example
    > code to:
    >
    > def functionName():
    > print "hello"
    > print dir(InterfaceModule)
    > print "VarDictionary is:", InterfaceModule.VarDictionary
    > print "that's all"
    >
    > changes that part of the output to:
    >
    > hello
    > ['VarDictionary', '__doc__', '__name__']
    > VarDictionary is: {'tk2': 'test2Val', 'tk1': 'test1Val'}
    > that's all
    >
    >
    > With all due respect, it looks like you're trying to run before you can
    > walk -- or more specifically, to embed Python in C++ before you become
    > familiar with the most elementary and fundamental aspects of Python,
    > such as indentation and the need to qualify compound names. You might
    > want to consider getting a good Python book -- such as, my own Python in
    > A Nutshell (2nd ed), Aahz and Stef Maruch's Python For Dummies, Wesley
    > Chun's Core Python Programming (2nd ed) -- they're all very recent (mine
    > is just out, Aahz's and Stef's I believe is due to hit bookstores in
    > September or October), and any of them might serve you well (if you're
    > OK with books not necessarily covering the very latest release of Python
    > [and the issues you're having suggest that this is not really the
    > problem!], there are many other good books, such as Magnus Lie Hetland's
    > "Beginning Python", Beazley's "Python Essential Reference", Lutz and
    > Ascher's "Learning Python",
    wardm, Aug 13, 2006
    #5
  6. wardm <> wrote:

    > Thanks again for your help, I agree, it seems I need to read a good book on
    > Python.
    >
    > One last question, will Python allow me to add new items to
    > InterfaceModule.VarDictionary
    > from the Python functions I call ?


    Yes, no problem with that.


    Alex
    Alex Martelli, Aug 13, 2006
    #6
  7. wardm

    Bill Pursell Guest

    Alex Martelli wrote:
    >
    > I wrote the following file za.c:


    <sample code snipped>

    > and proceeded to compile and execute as follows: [[Note: it does not
    > matter that I'm using 2.5, the code is just as fine with previous
    > versions -- it just happens that 2.5 is what I'm using right now in
    > order to help out with 2.5's beta testing]]:
    >
    > brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
    > brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
    > -lpython2.5
    > brain:~/pyex alex$ ./za



    I needed to include a lot more flags to make the example compile.
    In particular:
    -L /usr/local/lib/python2.4/config/ -lpython2.4 -lrt -lm -ldl -lutil

    Why did you not need them? Is this a misconfiguration on my
    box, or something entirely different? I've seen this type of thing
    come up a lot, and I can't tell if it is simply the author omitting
    flags for the sake of brevity, or if I'm actually missing something.

    --
    Bill Pursell
    Bill Pursell, Aug 13, 2006
    #7
  8. Bill Pursell <> wrote:

    > Alex Martelli wrote:
    > >
    > > I wrote the following file za.c:

    >
    > <sample code snipped>
    >
    > > and proceeded to compile and execute as follows: [[Note: it does not
    > > matter that I'm using 2.5, the code is just as fine with previous
    > > versions -- it just happens that 2.5 is what I'm using right now in
    > > order to help out with 2.5's beta testing]]:
    > >
    > > brain:~/pyex alex$ gcc -c za.c -I/usr/local/include/python2.5
    > > brain:~/pyex alex$ gcc -o za za.o -L/usr/local/lib/python2.5/config/
    > > -lpython2.5
    > > brain:~/pyex alex$ ./za

    >
    >
    > I needed to include a lot more flags to make the example compile.
    > In particular:
    > -L /usr/local/lib/python2.4/config/ -lpython2.4 -lrt -lm -ldl -lutil


    You no doubt needed that for linking, not for compiling (I needed the -L
    and -l too for linking, see above, just not the "other" libraries).


    > Why did you not need them? Is this a misconfiguration on my
    > box, or something entirely different? I've seen this type of thing
    > come up a lot, and I can't tell if it is simply the author omitting
    > flags for the sake of brevity, or if I'm actually missing something.


    Neither: we're just talking about different platforms -- probably both
    Unix variants, but with different linking-loader behavior and/or
    dependencies among libraries. Considering that my platform's ld is
    doubtlessly the single most prominent difference between it and other
    Unix variants from a developer's viewpoint (see
    <http://developer.apple.com/documentation/Darwin/Reference/ManPages/man1
    /ld.1.html> for more details) I was remiss in not mentioning my
    platform; I apologize, I was concentrating on the C++ and Python sources
    rather than on the build process (but I should have mentioned it in the
    Note, just as I mentioned that I was using 2.5 but it didn't matter!).


    Alex
    Alex Martelli, Aug 13, 2006
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ilias Lazaridis
    Replies:
    6
    Views:
    425
    Ilias Lazaridis
    Feb 21, 2006
  2. james_027
    Replies:
    1
    Views:
    307
    Marc 'BlackJack' Rintsch
    Aug 22, 2007
  3. BigHand
    Replies:
    3
    Views:
    307
    BigHand
    Mar 7, 2009
  4. Michael Kohout
    Replies:
    1
    Views:
    289
    Carl Banks
    Aug 12, 2009
  5. Navkirat Singh
    Replies:
    6
    Views:
    2,965
    Navkirat Singh
    Jul 29, 2010
Loading...

Share This Page