private data stashed in local/global execution context of PyEval_EvalCode disappears down the execut

Discussion in 'Python' started by sndive@gmail.com, Nov 7, 2007.

  1. Guest

    i naively created execution context:
    PyObject *execcontext = PyDict_New();
    stuffed a handle in it:
    PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    int st= PyDict_SetItemString(res, "interp", ih);

    and later on in a function for a module that i defined
    expected to extract that handle:
    PyObject *dict = PyEval_GetGlobals(); // or GetLocals
    depending on where
    // execcontext ended up in PyEval_EvalCode
    PyObject *co = PyDict_GetItemString(dict, "interp");
    assert(PyCObject_Check(co));
    but i get null co either way and the dict does not match execcontext
    that is passed into PyEval_EvalCode.

    any ideas how to fix this?
    , Nov 7, 2007
    #1
    1. Advertising

  2. Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack

    En Tue, 06 Nov 2007 23:25:17 -0300, <> escribió:

    > i naively created execution context:
    > PyObject *execcontext = PyDict_New();
    > stuffed a handle in it:
    > PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    > int st= PyDict_SetItemString(res, "interp", ih);


    What's `res`?
    One should make a lot of assumptions about your code because it's not
    complete. Please post a minimal complete example showing your problem.

    --
    Gabriel Genellina
    Gabriel Genellina, Nov 10, 2007
    #2
    1. Advertising

  3. Guest

    On Nov 9, 5:36 pm, "Gabriel Genellina" <> wrote:
    > En Tue, 06 Nov 2007 23:25:17 -0300, <> escribió:
    >
    > > i naively created execution context:
    > > PyObject *execcontext = PyDict_New();
    > > stuffed a handle in it:
    > > PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    > > int st= PyDict_SetItemString(res, "interp", ih);

    >
    > What's `res`?


    duh!
    it should be execcontext:
    int st= PyDict_SetItemString(execcontext, "interp", ih);
    then i'd
    PyEval_EvalCode(somecallable, execcontext, g_maindict);
    where g_maindict is a dictionary from __main__ module
    ....

    later as part of PyEval_EvalCode
    a method in a module extending python is executed by python runtime
    and i want to get "interp" assuming it would be in the global context:
    PyObject *dict = PyEval_GetGlobals();
    PyObject *co = PyDict_GetItemString(dict, "interp");
    but apparently PyEval_GetGlobals(); returns the dictionary
    that has nothing to do with the execcontext passed into
    PyEval_EvalCode higher up on the call stack
    and i do not know how to get to the execcontext dictionary that
    i passed in PyEval_EvalCode.
    If each module has its own globals what the point of passing
    global and local namespace into PyEval_EvalCode?

    > One should make a lot of assumptions about your code because it's not
    > complete. Please post a minimal complete example showing your problem.
    >

    It's a rather large program. My assumption was that just posting the
    snippet around the call site and the callee pathetic attempt
    to extract interp would be sufficient :(
    , Nov 13, 2007
    #3
  4. Chris Mellon Guest

    Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack

    On Nov 13, 2007 3:18 PM, <> wrote:
    > On Nov 9, 5:36 pm, "Gabriel Genellina" <> wrote:
    > > En Tue, 06 Nov 2007 23:25:17 -0300, <> escribió:


    >
    > > One should make a lot of assumptions about your code because it's not
    > > complete. Please post a minimal complete example showing your problem.
    > >

    > It's a rather large program. My assumption was that just posting the
    > snippet around the call site and the callee pathetic attempt
    > to extract interp would be sufficient :(
    >


    The creation of a minimal runnable sample is a fantastic way to find
    any bugs in your code, and has the benefit of (if the bug is not in
    your code) giving other people a simple way to recreate the bug. If I
    were to check this (and I'm not, but I would if you'd posted runnable
    code) I'll have to write the code myself from your textual
    description. Then, if the code works, I'll have to post the code that
    I wrote as well as my negative response, and go through several back
    and forths trying to isolate any differences between what I wrote and
    what you wrote but didn't show. That's way more work than I'm willing
    to do to solve someone else's problem.

    In my experience, creating a minimal sample that demonstrates the bug
    will lead you to the actual bug more than half the time. That's a lot
    of time (yours and other peoples) that can be saved if you do it.
    Chris Mellon, Nov 13, 2007
    #4
  5. Guest

    On Nov 13, 3:31 pm, "Chris Mellon" <> wrote:
    > On Nov 13, 2007 3:18 PM, <> wrote:
    >
    > > On Nov 9, 5:36 pm, "Gabriel Genellina" <> wrote:
    > > > En Tue, 06 Nov 2007 23:25:17 -0300, <> escribió:

    >
    > > > One should make a lot of assumptions about your code because it's not
    > > > complete. Please post a minimal complete example showing your problem.

    >
    > > It's a rather large program. My assumption was that just posting the
    > > snippet around the call site and the callee pathetic attempt
    > > to extract interp would be sufficient :(

    >
    > The creation of a minimal runnable sample is a fantastic way to find
    > any bugs in your code, and has the benefit of (if the bug is not in
    > your code) giving other people a simple way to recreate the bug. If I
    > were to check this (and I'm not, but I would if you'd posted runnable
    > code) I'll have to write the code myself from your textual
    > description. Then, if the code works, I'll have to post the code that
    > I wrote as well as my negative response, and go through several back
    > and forths trying to isolate any differences between what I wrote and
    > what you wrote but didn't show. That's way more work than I'm willing
    > to do to solve someone else's problem.
    >
    > In my experience, creating a minimal sample that demonstrates the bug
    > will lead you to the actual bug more than half the time. That's a lot
    > of time (yours and other peoples) that can be saved if you do it.


    working on a smaller example. i could not get pyNode_root invoked yet
    and
    PyRun_String("import node\nprint node.root()\n",
    Py_file_input,
    exec, g_maindict);
    triggers Py_CompileString
    failure with
    Traceback (most recent call last):
    File "<string>", line 1, in ?
    ImportError: __import__ not found

    without RyRun_String call program runs but the pyNode_root is not
    invoked

    #undef _POSIX_C_SOURCE
    #include <Python.h>

    #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif

    PyObject *g_mainmod;
    PyObject *g_maindict;
    bool worked = false;
    static
    PyObject *
    pyNode_root(PyObject *self, PyObject *args)
    {
    PyObject *dict = PyEval_GetGlobals();
    PyObject *co = PyDict_GetItemString(dict, "interp");
    assert(PyCObject_Check(co));
    void *interp = PyCObject_AsVoidPtr(co);
    assert(interp);
    // ...
    printf("root() worked\n");
    worked=true;
    return 0;
    }

    static PyMethodDef module_methods[] = {
    /* no need to create pyNode from python programs
    {"new", pyNode_new, METH_VARARGS,
    PyDoc_STR("new() -> new Node object")},
    */
    {"root", pyNode_root, METH_VARARGS,
    PyDoc_STR("root('dm') -> wrapper for the rootnode")},

    {NULL} /* Sentinel */
    };

    int
    main()
    {
    Py_Initialize();

    g_mainmod = PyImport_AddModule("__main__");
    assert(g_mainmod);
    g_maindict = PyModule_GetDict(g_mainmod);
    assert(g_maindict);
    Py_INCREF(g_maindict); // it was a borrowed reference

    PyObject* m = Py_InitModule("node", module_methods);
    if (m == NULL)
    return 1;

    PyObject *exec = PyDict_New();
    void *handle = (void*)0xdeadc0ed;
    PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    int st= PyDict_SetItemString(exec, "interp", ih);
    assert(!st);


    PyRun_String("import node\nprint node.root()\n",
    Py_file_input,
    exec, g_maindict);

    PyObject *mScriptHandle=
    Py_CompileString("import node\nprint node.root()\n",
    "comp", Py_file_input);
    if(!mScriptHandle) {
    PyErr_Print();
    return 2;
    }

    PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
    exec, g_maindict);

    assert(res=Py_None); // compiled as file_input
    assert(worked);
    }
    , Nov 13, 2007
    #5
  6. Guest

    On Nov 9, 7:36 pm, "Gabriel Genellina" <> wrote:
    > En Tue, 06 Nov 2007 23:25:17 -0300, <> escribió:
    >
    > > i naively created execution context:
    > > PyObject *execcontext = PyDict_New();
    > > stuffed a handle in it:
    > > PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    > > int st= PyDict_SetItemString(res, "interp", ih);

    >
    > What's `res`?
    > One should make a lot of assumptions about your code because it's not
    > complete. Please post a minimal complete example showing your problem.
    >
    > --
    > Gabriel Genellina


    for some reason i can't get compilestring/run_string to work in my
    example.
    compile the code below
    and from the command line type

    import node
    print node.root()

    you'd get:
    Program received signal SIGSEGV, Segmentation fault.
    [Switching to Thread -1218588544 (LWP 13539)]
    0x08048e37 in pyNode_root (self=0x0, args=0xb759d02c) at main.cpp:17
    17 assert(PyCObject_Check(co));
    (gdb) p co
    $1 = (PyObject *) 0x0

    because interp is not on globals

    #undef _POSIX_C_SOURCE
    #include <Python.h>

    #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif

    PyObject *g_mainmod;
    PyObject *g_maindict;
    bool worked = false;
    static
    PyObject *
    pyNode_root(PyObject *self, PyObject *args)
    {
    PyObject *dict = PyEval_GetGlobals();
    PyObject *co = PyDict_GetItemString(dict, "interp");
    assert(PyCObject_Check(co));
    void *interp = PyCObject_AsVoidPtr(co);
    assert(interp);
    // ...
    printf("root() worked\n");
    worked=true;
    return 0;
    }

    static PyMethodDef module_methods[] = {
    /* no need to create pyNode from python programs
    {"new", pyNode_new, METH_VARARGS,
    PyDoc_STR("new() -> new Node object")},
    */
    {"root", pyNode_root, METH_VARARGS,
    PyDoc_STR("root('dm') -> wrapper for the rootnode")},

    {NULL} /* Sentinel */
    };

    int
    main()
    {
    Py_Initialize();

    g_mainmod = PyImport_AddModule("__main__");
    assert(g_mainmod);
    g_maindict = PyModule_GetDict(g_mainmod);
    assert(g_maindict);
    Py_INCREF(g_maindict); // it was a borrowed reference

    PyObject* m = Py_InitModule("node", module_methods);
    if (m == NULL)
    return 1;

    PyObject *exec = PyDict_New();
    void *handle = (void*)0xdeadc0ed;
    PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    //Py_INCREF(ih);
    int st= PyDict_SetItemString(exec, "interp", ih);
    assert(!st);

    PyRun_InteractiveLoop(stdin, "<stdin>");
    }
    , Nov 13, 2007
    #6
  7. Guest

    On Nov 13, 3:31 pm, "Chris Mellon" <> wrote:
    > On Nov 13, 2007 3:18 PM, <> wrote:
    >
    > > On Nov 9, 5:36 pm, "Gabriel Genellina" <> wrote:
    > > > En Tue, 06 Nov 2007 23:25:17 -0300, <> escribió:

    >
    > > > One should make a lot of assumptions about your code because it's not
    > > > complete. Please post a minimal complete example showing your problem.

    >
    > > It's a rather large program. My assumption was that just posting the
    > > snippet around the call site and the callee pathetic attempt
    > > to extract interp would be sufficient :(

    >
    > The creation of a minimal runnable sample is a fantastic way to find
    > any bugs in your code, and has the benefit of (if the bug is not in
    > your code) giving other people a simple way to recreate the bug. If I
    > were to check this (and I'm not, but I would if you'd posted runnable
    > code) I'll have to write the code myself from your textual
    > description. Then, if the code works, I'll have to post the code that
    > I wrote as well as my negative response, and go through several back
    > and forths trying to isolate any differences between what I wrote and
    > what you wrote but didn't show. That's way more work than I'm willing
    > to do to solve someone else's problem.
    >
    > In my experience, creating a minimal sample that demonstrates the bug
    > will lead you to the actual bug more than half the time. That's a lot
    > of time (yours and other peoples) that can be saved if you do it.



    save as testimp.py:
    import node

    print "import test worked"

    def runtest():
    print "runtest()!!!"
    res = node.root()

    the program reproducing the problem:
    #undef _POSIX_C_SOURCE
    #include <Python.h>

    #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif

    PyObject *g_mainmod;
    PyObject *g_maindict;
    bool worked = false;
    static
    PyObject *
    pyNode_root(PyObject *self, PyObject *args)
    {
    PyObject *dict = PyEval_GetGlobals();
    PyObject *co = PyDict_GetItemString(dict, "interp");
    assert(PyCObject_Check(co));
    void *interp = PyCObject_AsVoidPtr(co);
    assert(interp);
    // ...
    printf("root() worked\n");
    worked=true;
    return 0;
    }

    static PyMethodDef module_methods[] = {
    /* no need to create pyNode from python programs
    {"new", pyNode_new, METH_VARARGS,
    PyDoc_STR("new() -> new Node object")},
    */
    {"root", pyNode_root, METH_VARARGS,
    PyDoc_STR("root('dm') -> wrapper for the rootnode")},

    {NULL} /* Sentinel */
    };

    int
    main()
    {
    Py_Initialize();
    PyRun_SimpleString("import sys\n"
    "sys.path.append('.')\n"
    "print ': ',sys.path\n"
    );

    g_mainmod = PyImport_AddModule("__main__");
    assert(g_mainmod);
    g_maindict = PyModule_GetDict(g_mainmod);
    assert(g_maindict);
    Py_INCREF(g_maindict); // it was a borrowed reference

    PyObject* m = Py_InitModule("node", module_methods);
    if (m == NULL)
    return 1;

    PyObject *eglobal = PyDict_New();
    void *handle = (void*)0xdeadc0ed;
    PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    int st= PyDict_SetItemString(eglobal, "interp", ih);
    assert(!st);

    PyObject *import = PyImport_ImportModule("testimp");
    if(!import) {
    PyErr_Print();
    return 3;
    }
    Py_INCREF(import);
    PyModule_AddObject(g_mainmod, "testimp", import);

    PyObject *mScriptHandle=
    Py_CompileString("testimp.runtest()",
    "comp", Py_eval_input);
    if(!mScriptHandle) {
    if (PyErr_Occurred()) {
    PyErr_Print();
    }
    return 2;
    }

    PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
    eglobal, g_maindict);
    if (PyErr_Occurred()) {
    PyErr_Print();
    }

    Py_DECREF(res);
    assert(worked);
    }
    , Nov 14, 2007
    #7
  8. Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack

    En Tue, 13 Nov 2007 19:59:56 -0300, <> escribió:

    > working on a smaller example. i could not get pyNode_root invoked yet
    > and
    > PyRun_String("import node\nprint node.root()\n",
    > Py_file_input,
    > exec, g_maindict);


    The globals argument should contain (at least) a key "__builtins__"
    pointing to the __builtin__ module; else, no builtin functions (like
    __import__) will be found. PyEval_Globals (you used it somewhere) returns
    a suitable globals argument.

    --
    Gabriel Genellina
    Gabriel Genellina, Nov 14, 2007
    #8
  9. Guest

    On Nov 14, 12:57 am, "Gabriel Genellina" <>
    wrote:
    > En Tue, 13 Nov 2007 19:59:56 -0300, <> escribió:
    >
    > > working on a smaller example. i could not get pyNode_root invoked yet
    > > and
    > > PyRun_String("import node\nprint node.root()\n",
    > > Py_file_input,
    > > exec, g_maindict);

    >
    > The globals argument should contain (at least) a key "__builtins__"
    > pointing to the __builtin__ module; else, no builtin functions (like
    > __import__) will be found. PyEval_Globals (you used it somewhere) returns
    > a suitable globals argument.
    >

    thank you.
    result is the same however:
    pyt: main.cpp:17: PyObject* pyNode_root(PyObject*, PyObject*):
    Assertion `co' failed.

    Program received signal SIGABRT, Aborted.
    [Switching to Thread -1218617216 (LWP 4807)]
    0x00a54eff in raise () from /lib/tls/libc.so.6

    with testimp.py:
    import node

    print "import test worked"

    def runtest():
    print "runtest()!!!"
    res = node.root()

    and the code
    #undef _POSIX_C_SOURCE
    #include <Python.h>

    #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
    #define PyMODINIT_FUNC void
    #endif

    PyObject *g_mainmod;
    PyObject *g_maindict;
    bool worked = false;
    static
    PyObject *
    pyNode_root(PyObject *self, PyObject *args)
    {
    PyObject *dict = PyEval_GetGlobals();
    PyObject *co = PyDict_GetItemString(dict, "interp");
    assert(co);
    assert(PyCObject_Check(co));
    void *interp = PyCObject_AsVoidPtr(co);
    assert(interp);
    // ...
    printf("root() worked\n");
    worked=true;
    return 0;
    }

    static PyMethodDef module_methods[] = {
    /* no need to create pyNode from python programs
    {"new", pyNode_new, METH_VARARGS,
    PyDoc_STR("new() -> new Node object")},
    */
    {"root", pyNode_root, METH_VARARGS,
    PyDoc_STR("root('dm') -> wrapper for the rootnode")},

    {NULL} /* Sentinel */
    };

    int
    main()
    {
    Py_Initialize();
    PyRun_SimpleString("import sys\n"
    "sys.path.append('.')\n"
    "print ': ',sys.path\n"
    );

    g_mainmod = PyImport_AddModule("__main__");
    assert(g_mainmod);
    g_maindict = PyModule_GetDict(g_mainmod);
    assert(g_maindict);
    Py_INCREF(g_maindict); // it was a borrowed reference

    PyObject* m = Py_InitModule("node", module_methods);
    if (m == NULL)
    return 1;

    PyObject *eglobal = PyDict_New();
    void *handle = (void*)0xdeadc0ed;
    PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
    int st= PyDict_SetItemString(eglobal, "interp", ih);
    assert(!st);
    PyObject *builtins = PyEval_GetBuiltins();
    st= PyDict_SetItemString(eglobal, "__builtins__", builtins);
    assert(!st);

    PyObject *import = PyImport_ImportModule("testimp");
    if(!import) {
    PyErr_Print();
    return 3;
    }
    Py_INCREF(import);
    PyModule_AddObject(g_mainmod, "testimp", import);

    PyObject *mScriptHandle=
    Py_CompileString("testimp.runtest()",
    "comp", Py_eval_input);
    if(!mScriptHandle) {
    if (PyErr_Occurred()) {
    PyErr_Print();
    }
    return 2;
    }

    PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
    eglobal, g_maindict);
    if (PyErr_Occurred()) {
    PyErr_Print();
    }

    Py_DECREF(res);
    assert(worked);
    }
    , Nov 14, 2007
    #9
  10. Re: private data stashed in local/global execution context ofPyEval_EvalCode disappears down the execution stack

    En Wed, 14 Nov 2007 15:08:42 -0300, <> escribió:

    > thank you.
    > result is the same however:
    > pyt: main.cpp:17: PyObject* pyNode_root(PyObject*, PyObject*):
    > Assertion `co' failed.


    Well, "is the same" in the sense that the code still doesn't do what you
    want... But the previous error is gone. (Now I regret having said the
    reason it didn't work before: you took my words too literally).

    I suggest you first try to write the program in pure Python, even with a
    dummy Node. From what I can understand of what you are doing, it appears
    you don't fully understand how import works, how modules work, and
    namespaces in Python. The C API exposes the same concepts so it's easier
    when one knows how to do things in pure Python first. Right now you have
    two problems: "what" to do, and "how" to write that using the API. First
    try to solve the "what", in Python, and only later move to the second
    stage.

    --
    Gabriel Genellina
    Gabriel Genellina, Nov 14, 2007
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. DaveLessnau
    Replies:
    3
    Views:
    413
    Howard
    May 16, 2005
  2. Daniel Trstenjak

    'locals' argument of PyEval_EvalCode

    Daniel Trstenjak, Nov 28, 2006, in forum: Python
    Replies:
    0
    Views:
    323
    Daniel Trstenjak
    Nov 28, 2006
  3. Replies:
    3
    Views:
    929
    Gabriel Genellina
    Mar 11, 2009
  4. pk
    Replies:
    7
    Views:
    158
  5. Leslaw Bieniasz

    local static data vs. global data

    Leslaw Bieniasz, Oct 23, 2012, in forum: C++
    Replies:
    3
    Views:
    378
    Werner
    Oct 24, 2012
Loading...

Share This Page