'Import sys' succeeds in C++ embedded code, but module is not fullyvisible

[Ben Sizer]

I have the following C++ code and am attempting to embed Python 2.5,
but although the "import sys" statement works, attempting to reference
"sys.path" from inside a function after that point fails. It's as if
it's not treating it as a normal module but as any other global
variable which I'd have to explicitly qualify.


Py_InitializeEx(0); // the zero skips registration of signal
handlers.
PyObject* ourNamespace_ = PyDict_New();
PyDict_SetItemString(ourNamespace_, "__builtins__",
PyEval_GetBuiltins());
PyObject* locals = PyDict_New();

const char* scriptStr =
"print '1'\n"
"import sys\n"
"print sys.path\n"
"def debug_path_info():\n"
" print 'These are the directories Python looks into for
modules and source files:'\n"
" print '2'\n"
" for folder in sys.path:\n"
" print folder\n"
" print '--------------'\n"
" print 'This would be your present working folder/
directory:'\n"
" print '3'\n"
" print sys.path[0]\n"
"debug_path_info()\n";

PyObject* scriptResult = PyRun_String(
scriptStr, // Python code to execute
Py_file_input,
ourNamespace_, // globals dictionary
locals); // locals dictionary

if (!scriptResult)
{
std::cerr << "Python error: " << "Unhandled Python exception from
script." << std::endl;
PyErr_Print();
}
else
{
Py_DECREF(scriptResult); // don't need result any more
}

Py_DECREF(locals);
Py_DECREF(ourNamespace_);
Py_Finalize();


And the output is like this:

1
['E:\\code\\Python25\\lib\\site-packages\\turbokid-1.0.4-py2.5.egg',
'E:\\code\\
Python25\\lib\\site-packages\\turbocheetah-1.0-py2.5.egg', 'E:\\code\
\Python25\\
lib\\site-packages\\simplejson-1.8.1-py2.5-win32.egg', 'E:\\code\
\Python25\\lib\
\site-packages\\ruledispatch-0.5a0.dev_r2306-py2.5-win32.egg', 'E:\
\code\\Python
25\\lib\\site-packages\\pastescript-1.6.2-py2.5.egg', 'E:\\code\
\Python25\\lib\\
site-packages\\formencode-1.0.1-py2.5.egg', 'E:\\code\\Python25\\lib\
\site-packa
ges\\decoratortools-1.7-py2.5.egg', 'E:\\code\\Python25\\lib\\site-
packages\\con
figobj-4.5.2-py2.5.egg', 'E:\\code\\Python25\\lib\\site-packages\
\cherrypy-2.3.0
-py2.5.egg', 'E:\\code\\Python25\\lib\\site-packages\\kid-0.9.6-
py2.5.egg', 'E:\
\code\\Python25\\lib\\site-packages\\cheetah-2.0.1-py2.5-win32.egg',
'E:\\code\\
Python25\\lib\\site-packages\\pyprotocols-1.0a0-py2.5-win32.egg', 'E:\
\code\\Pyt
hon25\\lib\\site-packages\\pastedeploy-1.3.1-py2.5.egg', 'E:\\code\
\Python25\\li
b\\site-packages\\paste-1.6-py2.5.egg', 'E:\\code\\Python25\\lib\\site-
packages\
\sqlobject-0.10.0-py2.5.egg', 'E:\\code\\Python25\\lib\\site-packages\
\tgfastdat
a-0.9a7-py2.5.egg', 'E:\\code\\Python25\\lib\\site-packages\
\webhelpers-0.6-py2.
5.egg', 'E:\\code\\Python25\\lib\\site-packages\\shove-0.1.3-
py2.5.egg', 'E:\\co
de\\Python25\\lib\\site-packages\\boto-1.3a-py2.5.egg', 'E:\\code\
\Python25\\lib
\\site-packages\\sqlalchemy-0.5.0beta3-py2.5.egg', 'E:\\code\\Python25\
\lib\\sit
e-packages\\turbojson-1.1.4-py2.5.egg', 'E:\\code\\Python25\\lib\\site-
packages\
\setuptools-0.6c9-py2.5.egg', 'E:\\code\\Python25\\lib\\site-packages\
\turbogear
s-1.0.8-py2.5.egg', 'C:\\WINDOWS\\system32\\python25_d.zip', 'E:\\code\
\Python25
\\Lib', 'E:\\code\\Python25\\DLLs', 'E:\\code\\Python25\\Lib\\lib-tk',
'e:\\Visu
al Studio 2008\\Projects\\StacklessEmbed\\StacklessEmbed', 'e:\\Visual
Studio 20
08\\Projects\\StacklessEmbed\\Debug', 'E:\\code\\Python25', 'E:\\code\
\Python25\
\lib\\site-packages', 'E:\\code\\Python25\\lib\\site-packages\\PIL',
'E:\\code\\
Python25\\lib\\site-packages\\wx-2.8-msw-unicode']
These are the directories Python looks into for modules and source
files:
2
Python error: Unhandled Python exception from script.
Traceback (most recent call last):
File "<string>", line 13, in <module>
File "<string>", line 7, in debug_path_info
NameError: global name 'sys' is not defined
[12532 refs]


(Incidentally, the Stackless references are because I was originally
trying to embed Stackless, but I reverted to vanilla 2.5 to see if it
was a Stackless specific issue, which it appears not.)

Another interesting thing is that sys.path[0] doesn't appear to be the
current working directory, despite several sources online suggesting
it should be.

What am I doing wrong?

 

[Ivan Illarionov]

What am I doing wrong?

What are you trying to achieve?
If you want to modify sys.path I suggest using Python/C API directly:
(boilerplate removed)
PyImport_ImportModule("sys")
PyObject_GetAttrString(sysmod_pointer, "path")
PyList_Insert(pathobj_pointer, 0, path_python_str)

 

[Ben Sizer]

What are you trying to achieve?
If you want to modify sys.path I suggest using Python/C API directly:

No, I don't want to do anything with sys.path apart from see it. I
just wanted my original question answered, not a guess at my intent
and a solution for something I'm not doing. Thanks though!

Again - why can I not reference sys from within the function?

 

[Ivan Illarionov]

No, I don't want to do anything with sys.path apart from see it. I
just wanted my original question answered, not a guess at my intent
and a solution for something I'm not doing.  Thanks though!

Again - why can I not reference sys from within the function?

Ah, sorry for wrong guess.

I would try to use ourNamespace_ dict for
both globals and locals in PyRun_String call.

 

[Ben Sizer]

I would try to use ourNamespace_ dict for
both globals and locals in PyRun_String call.

I will try it when I get home. However I would like to be able to
treat them as separate dictionaries, as I want to be able to import
some symbols and modules at a global level, but be able to clear out
objects introduced at the local level on a periodic basis, so that I
can have some degree of isolation between distinct 'scripts'. The docs
aren't terribly clear about what the globals and locals parameters to
PyRun_String actually do, though.

I also wonder if this is something specific to the sys module, since
it's already been shown that there are some specific C API functions
for it. I will try with other modules and see if they exhibit the same
symptoms.

And I'm still wondering about the sys.path[0] question.

 

[Ivan Illarionov]

I will try it when I get home. However I would like to be able to
treat them as separate dictionaries, as I want to be able to import
some symbols and modules at a global level, but be able to clear out
objects introduced at the local level on a periodic basis, so that I
can have some degree of isolation between distinct 'scripts'. The docs
aren't terribly clear about what the globals and locals parameters to
PyRun_String actually do, though.

I also wonder if this is something specific to the sys module, since
it's already been shown that there are some specific C API functions
for it. I will try with other modules and see if they exhibit the same
symptoms.

After quick testing it looks like '__builtins__' must be in locals
dictionary for your embed Python code to work, it may be a separate
dictionary from globals though.

From what I know 'sys' module is related to builtins. My knowledge of
Python internals is not so deep to explain the details of this
relationship and answer your question about sys.path[0] though.

 

[Ivan Illarionov]

I will try it when I get home. However I would like to be able to
treat them as separate dictionaries, as I want to be able to import
some symbols and modules at a global level, but be able to clear out
objects introduced at the local level on a periodic basis, so that I
can have some degree of isolation between distinct 'scripts'. The docs
aren't terribly clear about what the globals and locals parameters to
PyRun_String actually do, though.

I also wonder if this is something specific to the sys module, since
it's already been shown that there are some specific C API functions
for it. I will try with other modules and see if they exhibit the same
symptoms.

After quick testing it looks like '__builtins__' must be in locals
dictionary for your embed Python code to work, it may be a separate
dictionary from globals though.

From what I know 'sys' module is related to builtins. My knowledge of
Python internals is not so deep to explain the details of this
relationship and answer your question about sys.path[0] though.

Sorry, I probably was terribly wrong in my last post, it doesn't work
this way with separate dictionaries. I never had to use separate
dictionaries. Maybe someone with better knowledge of Python internals
may help?

 

[Ben Sizer]

Ah, sorry for wrong guess.

I would try to use ourNamespace_ dict for
both globals and locals in PyRun_String call.

Yes, this seems to fix it, thanks. But why? Can some Python guru
explain why these two dictionaries must be the same? (Or what steps we
must take if we want them to be separate?) The documentation is not
very clear. I had hoped to be able to clear out the locals dictionary
while leaving useful functions intact in the globals dictionary, but
it would appear that is not practical.

(On a separate note, while trying to debug this it seemed that Python
will look for debug versions of a library when you embed it in debug
mode, and will fail to find a module if you only have the release
versions there. The error message you get isn't too helpful about this
however, and I only worked it out by looking at the very long list of
filesystem calls Python made to try and find it. Anybody wishing to
speed up import times might want to ensure they don't have a long
Python path and as few eggs in site-packages as possible.)

 

[Aahz]

I have the following C++ code and am attempting to embed Python 2.5,
but although the "import sys" statement works, attempting to reference
"sys.path" from inside a function after that point fails. It's as if
it's not treating it as a normal module but as any other global
variable which I'd have to explicitly qualify.

After skimming the thread and seeing a lack of answer, I suggest you try
(e-mail address removed)

 

[greg]

Yes, this seems to fix it, thanks. But why? Can some Python guru
explain why these two dictionaries must be the same? (Or what steps we
must take if we want them to be separate?)

What's happening is that the import statement is binding
the name 'sys' in the locals, not the globals. You don't
notice this in the top-level code, since both are in scope
there. But inside the function you can only see the top-level
globals plus the function's own locals.

I had hoped to be able to clear out the locals dictionary
while leaving useful functions intact in the globals dictionary, but
it would appear that is not practical.

You can probably do that by using global statements in the
top-level code for the things you want to preserve, e.g.

global sys
import sys
my_local_var = 42

global f
def f():
print "This function is in globals and can see sys.path"
print sys.path

def g():
print "This function is in locals and can't see sys.path"

Now clearing the locals will remove g and my_local_var
while leaving the rest in globals.

The only restriction is that anything you want to refer
to from a function will have to be in the globals. This
includes other functions -- i.e. f() won't be able to
call g() in the example above.