private data stashed in local/global execution context of PyEval_EvalCode disappears down the execut

S

sndive

i naively created execution context:
PyObject *execcontext = PyDict_New();
stuffed a handle in it:
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(res, "interp", ih);

and later on in a function for a module that i defined
expected to extract that handle:
PyObject *dict = PyEval_GetGlobals(); // or GetLocals
depending on where
// execcontext ended up in PyEval_EvalCode
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
but i get null co either way and the dict does not match execcontext
that is passed into PyEval_EvalCode.

any ideas how to fix this?
 
G

Gabriel Genellina

i naively created execution context:
PyObject *execcontext = PyDict_New();
stuffed a handle in it:
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(res, "interp", ih);

What's `res`?
One should make a lot of assumptions about your code because it's not
complete. Please post a minimal complete example showing your problem.
 
S

sndive

What's `res`?

duh!
it should be execcontext:
int st= PyDict_SetItemString(execcontext, "interp", ih);
then i'd
PyEval_EvalCode(somecallable, execcontext, g_maindict);
where g_maindict is a dictionary from __main__ module
....

later as part of PyEval_EvalCode
a method in a module extending python is executed by python runtime
and i want to get "interp" assuming it would be in the global context:
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
but apparently PyEval_GetGlobals(); returns the dictionary
that has nothing to do with the execcontext passed into
PyEval_EvalCode higher up on the call stack
and i do not know how to get to the execcontext dictionary that
i passed in PyEval_EvalCode.
If each module has its own globals what the point of passing
global and local namespace into PyEval_EvalCode?
One should make a lot of assumptions about your code because it's not
complete. Please post a minimal complete example showing your problem.
It's a rather large program. My assumption was that just posting the
snippet around the call site and the callee pathetic attempt
to extract interp would be sufficient :(
 
C

Chris Mellon

It's a rather large program. My assumption was that just posting the
snippet around the call site and the callee pathetic attempt
to extract interp would be sufficient :(

The creation of a minimal runnable sample is a fantastic way to find
any bugs in your code, and has the benefit of (if the bug is not in
your code) giving other people a simple way to recreate the bug. If I
were to check this (and I'm not, but I would if you'd posted runnable
code) I'll have to write the code myself from your textual
description. Then, if the code works, I'll have to post the code that
I wrote as well as my negative response, and go through several back
and forths trying to isolate any differences between what I wrote and
what you wrote but didn't show. That's way more work than I'm willing
to do to solve someone else's problem.

In my experience, creating a minimal sample that demonstrates the bug
will lead you to the actual bug more than half the time. That's a lot
of time (yours and other peoples) that can be saved if you do it.
 
S

sndive

The creation of a minimal runnable sample is a fantastic way to find
any bugs in your code, and has the benefit of (if the bug is not in
your code) giving other people a simple way to recreate the bug. If I
were to check this (and I'm not, but I would if you'd posted runnable
code) I'll have to write the code myself from your textual
description. Then, if the code works, I'll have to post the code that
I wrote as well as my negative response, and go through several back
and forths trying to isolate any differences between what I wrote and
what you wrote but didn't show. That's way more work than I'm willing
to do to solve someone else's problem.

In my experience, creating a minimal sample that demonstrates the bug
will lead you to the actual bug more than half the time. That's a lot
of time (yours and other peoples) that can be saved if you do it.

working on a smaller example. i could not get pyNode_root invoked yet
and
PyRun_String("import node\nprint node.root()\n",
Py_file_input,
exec, g_maindict);
triggers Py_CompileString
failure with
Traceback (most recent call last):
File "<string>", line 1, in ?
ImportError: __import__ not found

without RyRun_String call program runs but the pyNode_root is not
invoked

#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *exec = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(exec, "interp", ih);
assert(!st);


PyRun_String("import node\nprint node.root()\n",
Py_file_input,
exec, g_maindict);

PyObject *mScriptHandle=
Py_CompileString("import node\nprint node.root()\n",
"comp", Py_file_input);
if(!mScriptHandle) {
PyErr_Print();
return 2;
}

PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
exec, g_maindict);

assert(res=Py_None); // compiled as file_input
assert(worked);
}
 
S

sndive

What's `res`?
One should make a lot of assumptions about your code because it's not
complete. Please post a minimal complete example showing your problem.

for some reason i can't get compilestring/run_string to work in my
example.
compile the code below
and from the command line type

import node
print node.root()

you'd get:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1218588544 (LWP 13539)]
0x08048e37 in pyNode_root (self=0x0, args=0xb759d02c) at main.cpp:17
17 assert(PyCObject_Check(co));
(gdb) p co
$1 = (PyObject *) 0x0

because interp is not on globals

#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *exec = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
//Py_INCREF(ih);
int st= PyDict_SetItemString(exec, "interp", ih);
assert(!st);

PyRun_InteractiveLoop(stdin, "<stdin>");
}
 
S

sndive

The creation of a minimal runnable sample is a fantastic way to find
any bugs in your code, and has the benefit of (if the bug is not in
your code) giving other people a simple way to recreate the bug. If I
were to check this (and I'm not, but I would if you'd posted runnable
code) I'll have to write the code myself from your textual
description. Then, if the code works, I'll have to post the code that
I wrote as well as my negative response, and go through several back
and forths trying to isolate any differences between what I wrote and
what you wrote but didn't show. That's way more work than I'm willing
to do to solve someone else's problem.

In my experience, creating a minimal sample that demonstrates the bug
will lead you to the actual bug more than half the time. That's a lot
of time (yours and other peoples) that can be saved if you do it.


save as testimp.py:
import node

print "import test worked"

def runtest():
print "runtest()!!!"
res = node.root()

the program reproducing the problem:
#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();
PyRun_SimpleString("import sys\n"
"sys.path.append('.')\n"
"print ': ',sys.path\n"
);

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *eglobal = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(eglobal, "interp", ih);
assert(!st);

PyObject *import = PyImport_ImportModule("testimp");
if(!import) {
PyErr_Print();
return 3;
}
Py_INCREF(import);
PyModule_AddObject(g_mainmod, "testimp", import);

PyObject *mScriptHandle=
Py_CompileString("testimp.runtest()",
"comp", Py_eval_input);
if(!mScriptHandle) {
if (PyErr_Occurred()) {
PyErr_Print();
}
return 2;
}

PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
eglobal, g_maindict);
if (PyErr_Occurred()) {
PyErr_Print();
}

Py_DECREF(res);
assert(worked);
}
 
G

Gabriel Genellina

working on a smaller example. i could not get pyNode_root invoked yet
and
PyRun_String("import node\nprint node.root()\n",
Py_file_input,
exec, g_maindict);

The globals argument should contain (at least) a key "__builtins__"
pointing to the __builtin__ module; else, no builtin functions (like
__import__) will be found. PyEval_Globals (you used it somewhere) returns
a suitable globals argument.
 
S

sndive

The globals argument should contain (at least) a key "__builtins__"
pointing to the __builtin__ module; else, no builtin functions (like
__import__) will be found. PyEval_Globals (you used it somewhere) returns
a suitable globals argument.
thank you.
result is the same however:
pyt: main.cpp:17: PyObject* pyNode_root(PyObject*, PyObject*):
Assertion `co' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread -1218617216 (LWP 4807)]
0x00a54eff in raise () from /lib/tls/libc.so.6

with testimp.py:
import node

print "import test worked"

def runtest():
print "runtest()!!!"
res = node.root()

and the code
#undef _POSIX_C_SOURCE
#include <Python.h>

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif

PyObject *g_mainmod;
PyObject *g_maindict;
bool worked = false;
static
PyObject *
pyNode_root(PyObject *self, PyObject *args)
{
PyObject *dict = PyEval_GetGlobals();
PyObject *co = PyDict_GetItemString(dict, "interp");
assert(co);
assert(PyCObject_Check(co));
void *interp = PyCObject_AsVoidPtr(co);
assert(interp);
// ...
printf("root() worked\n");
worked=true;
return 0;
}

static PyMethodDef module_methods[] = {
/* no need to create pyNode from python programs
{"new", pyNode_new, METH_VARARGS,
PyDoc_STR("new() -> new Node object")},
*/
{"root", pyNode_root, METH_VARARGS,
PyDoc_STR("root('dm') -> wrapper for the rootnode")},

{NULL} /* Sentinel */
};

int
main()
{
Py_Initialize();
PyRun_SimpleString("import sys\n"
"sys.path.append('.')\n"
"print ': ',sys.path\n"
);

g_mainmod = PyImport_AddModule("__main__");
assert(g_mainmod);
g_maindict = PyModule_GetDict(g_mainmod);
assert(g_maindict);
Py_INCREF(g_maindict); // it was a borrowed reference

PyObject* m = Py_InitModule("node", module_methods);
if (m == NULL)
return 1;

PyObject *eglobal = PyDict_New();
void *handle = (void*)0xdeadc0ed;
PyObject *ih = PyCObject_FromVoidPtr(handle, NULL);
int st= PyDict_SetItemString(eglobal, "interp", ih);
assert(!st);
PyObject *builtins = PyEval_GetBuiltins();
st= PyDict_SetItemString(eglobal, "__builtins__", builtins);
assert(!st);

PyObject *import = PyImport_ImportModule("testimp");
if(!import) {
PyErr_Print();
return 3;
}
Py_INCREF(import);
PyModule_AddObject(g_mainmod, "testimp", import);

PyObject *mScriptHandle=
Py_CompileString("testimp.runtest()",
"comp", Py_eval_input);
if(!mScriptHandle) {
if (PyErr_Occurred()) {
PyErr_Print();
}
return 2;
}

PyObject *res = PyEval_EvalCode((PyCodeObject*)mScriptHandle,
eglobal, g_maindict);
if (PyErr_Occurred()) {
PyErr_Print();
}

Py_DECREF(res);
assert(worked);
}
 
G

Gabriel Genellina

thank you.
result is the same however:
pyt: main.cpp:17: PyObject* pyNode_root(PyObject*, PyObject*):
Assertion `co' failed.

Well, "is the same" in the sense that the code still doesn't do what you
want... But the previous error is gone. (Now I regret having said the
reason it didn't work before: you took my words too literally).

I suggest you first try to write the program in pure Python, even with a
dummy Node. From what I can understand of what you are doing, it appears
you don't fully understand how import works, how modules work, and
namespaces in Python. The C API exposes the same concepts so it's easier
when one knows how to do things in pure Python first. Right now you have
two problems: "what" to do, and "how" to write that using the API. First
try to solve the "what", in Python, and only later move to the second
stage.
 

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top