Python/C and PYTHONPATH

  • Thread starter Tero Pihlajakoski
  • Start date
T

Tero Pihlajakoski

Hi,

I've been experimenting on embedding Python to a C software, and ran into
a little problem with PYTHONPATH (I'm running on linux). Here's the deal:

When trying to call PyImport_Import("xyz"), Python returns an error "No
module named xyz". The problem _seems_ to be that I had no PYTHONPATH
variable defined (though python-shell works ok regardless), since the
following in bash helps:

<try, no go>
PYTHONPATH=
export PYTHONPATH
<try, works ok>

I'm not defining anything as the PATH, and now it works. Why doesn't it
look from "." without a dummy PYTHONPATH? Or is this actually a bug fixed
in a newer release (running 2.3.?)?

What is the "official" way to solve this? PySetProgramName()? Shell
script?

Thanks,
- Tero

--
 
J

Jeff Epler

You'd need to tell us how you're invoking Python. When the Python
executable is invoked without arguments, the current directory is on the
sys.path:

$ python
Python 2.2.2 (#1, Feb 24 2003, 19:13:11)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
''

When you run a script, the directory of that script is in sys.path[0]:
$ echo "import sys; print sys.path[0]" > /tmp/script.py
$ python /tmp/script.py
/tmp

Some Python editors may put the directory where they reside on sys.path,
but not the current directory. They might do other startup that is
different from normal. The newest Python I have on this machine
is 2.3b1, and its idle puts these elements at the start of Pythonpath:

Python 2.3b1 (#16, May 19 2003, 10:22:28)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
Type "copyright", "credits" or "license" for more information.
IDLE 0.8 -- press F1 for help
['/home/jepler', '/usr/local/bin']

.... so instead of "the current directory" ('' or '.'), sys.path[0]
is "the directory idle was invoked from", and that's followed by "the
directory where the idle script lives".

Jeff
 
T

Tero Pihlajakoski

Jeff Epler said:
You'd need to tell us how you're invoking Python. When the Python

Take a look at:
http://www.python.org/doc/current/ext/pure-embedding.html. Basically,
it's about same (and I tested with it too, same results: failing)
executable is invoked without arguments, the current directory is on the
sys.path:
$ python
Python 2.2.2 (#1, Feb 24 2003, 19:13:11)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import sys
sys.path[0]
''

When you run a script, the directory of that script is in sys.path[0]:
$ echo "import sys; print sys.path[0]" > /tmp/script.py
$ python /tmp/script.py
/tmp

If I run "python" as a shell (python <scriptname>), everything works ok.
Says path is ok: [ '', '<thelib>', ... ]. However, from C, something works
differently: PyRun_SimpleString("import sys\nprint sys.path"); Shows no ''
(the rest is ok). The question is: Why is there no '', and what is the
"official/safe" way to set it? Just insert it to the list? (Setting
"PYTHONPATH= " helps, but I'd like it to run "off the box")

Tero
 
S

Samuel Walters

Fair warning: At this point, I couldn't even get the example you quoted
to link properly. Of course, I didn't dig too deep, as I'm in a hurry.

throw this c code into a .h file (my name for it is envdump.h)

---code---
#include <stdio.h>

extern char **environ;

void fdump_env(FILE *fd)
{
char **ptr;
int i = 0;
ptr = environ;
while(ptr != NULL)
{
fprintf(fd, "%i : %s\n", ptr);
i++;
}
}


void dump_env()
{
char **ptr;
int i = 0;
ptr = environ;
while(ptr != NULL)
{
printf("%i : %s\n", i, ptr);
i++;
}
}
---code---

Now, just inside the main function (or, perhaps, wherever you please, but
I'd try the line right before the first if statement) call dump_env() and
it will spit out all the environment variables that your C program
received. You might find something there that's of interest.

Or, call setenv ("man 3 setenv") to manually set a python-path.
you can use that in combo with getenv ("man 3 getenv") to append to an
existing PYTHONPATH, if you so choose.

Give it a go, please let me know how it turns out. If this doesn't shed
some light on the matter I might have another idea once I'm not rushing
around.

HTH

Sam Walters
 
T

Tero Pihlajakoski

Samuel Walters said:
Fair warning: At this point, I couldn't even get the example you quoted
to link properly. Of course, I didn't dig too deep, as I'm in a hurry.
---code---

.... snip ...
Or, call setenv ("man 3 setenv") to manually set a python-path.
you can use that in combo with getenv ("man 3 getenv") to append to an
existing PYTHONPATH, if you so choose.
Give it a go, please let me know how it turns out. If this doesn't shed
some light on the matter I might have another idea once I'm not rushing
around.

(Sorry for the delay, I was offline)

I'll see if it's actually the C-part that's causing problems, but I
worked it around by adding:

PyRun_SimpleString("import sys\nsys.path.insert(0,'')");

right after the Py_Initialize().

Works ok. Guess I'm "allowed" to do that(?)

Thanks,
- Tero

--
 
S

Samuel Walters

|Thus Spake Tero Pihlajakoski On the now historical date of Mon, 05 Jan
2004 22:39:19 +0000|
I'll see if it's actually the C-part that's causing problems, but I worked
it around by adding:

PyRun_SimpleString("import sys\nsys.path.insert(0,'')");

right after the Py_Initialize().

Works ok. Guess I'm "allowed" to do that(?)
IMLK (In My Limited Knowledge) that seems okay, but it also feels a bit
ham-handed. try this snippet:

---untested code---
#include <string.h>
#include <stdlib.h>
/* prototyping may not be neccessary... dunno...*/
extern char *getenv(const char *name);
extern char *setenv(const char *name, const char *value, int overwrite);
/* comment them out if gcc complains about redeclarations. */

/* append something to pythonpath */
/* returns 0 on failure, 1 on creation of the env variable and 2 on an append
oh, and -1 in cases of catastrophic miscompilation */
int myPyPathAppend(char *extrapath)
{
char *buffer = NULL;
/* size to taste... You should do this dynamically to avoid future buffer overrun attacks*/
char eventual_path[1024] = { 0 };
/* take note: after getenv, buffer points to an external constant character buffer.
do NOT try to modify it directly. use strcpy(char *dest, char *src)
(he says knowingly... is she into photography mate?)
*/
if( (buffer = getenv("PYTHONPATH")) == NULL )
{
/* we're here because PYTHONPATH is not already part of the environment. */

setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
/* did it go happen .. you should check this more rigorously*/
if( (buffer = getenv("PYTHONPATH")) == NULL)
{
/* we failed... abend. */
return 0;
}
else
{
/* success! cheers! */
return 1;
}
return -1; /* dead code... should never reach here */
}
else
{
/* PYTHONPATH already exists. append ';', then our new path and update it. */

/* find the "=" in the buffer...
from string.h
extern char *strstr (__const char *__haystack, __const char *__needle)
there's a better way to do this, but I can't recall the function off the top of my head
*/
buffer = strstr(buffer, "=") + 1; /* +1 because buffer points to the equals. we want the string starting after it. */

/* copy the old PYTHONPATH string */
strcpy(eventual_path, buffer);
strcat(eventual_path, ";");
strcat(eventual_path, extrapath);

setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
/* did it go happen .. you should check this more rigorously*/
if( (buffer = getenv("PYTHONPATH")) == NULL)
{
/* we failed... abend. */
return 0;
}
else
{
/* success! cheers! */
return 2;
}
return -1; /* dead code... should never reach here */
}
else
{
/* PYTHONPATH already exists. append ';', then our new path and update it. */

/* find the "=" in the buffer...
from string.h
extern char *strstr (__const char *__haystack, __const char *__needle)
there's a better way to do this, but I can't recall the function off the top of my head
*/
buffer = strstr(buffer, "=") + 1; /* +1 because buffer points to the equals. we want the string starting after it. */

/* copy the old PYTHONPATH string */
strcpy(eventual_path, buffer);
strcat(eventual_path, ";");
strcat(eventual_path, extrapath);

setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
/* did it go happen .. you should check this more rigorously*/
if( (buffer = getenv("PYTHONPATH")) == NULL)
{
/* we failed... abend. */
return 0;
}
else
{
/* success! cheers! */
return 2;
}
return -1; /* dead code... should never reach here */
}
return -1; /* deader code... should *really* never reach here */
}
---untested code---
I haven't tested, compiled or even read through this code.
I'm late for a party and still added comments
That means you get punctuation patrol :p
Check the semicolons, check the braces
Hey, I hear that in some companies they call this teamwork methodology
"extreme-programming" We're buzzword compliant!

HTH
(Danm... I'm such a code monkey)
 
T

Tero Pihlajakoski

Samuel Walters said:
|Thus Spake Tero Pihlajakoski On the now historical date of Mon, 05 Jan
2004 22:39:19 +0000|
IMLK (In My Limited Knowledge) that seems okay, but it also feels a bit
ham-handed. try this snippet:

Ok, there are comments here, somewhere:
---untested code---
#include <string.h>
#include <stdlib.h>
/* prototyping may not be neccessary... dunno...*/
extern char *getenv(const char *name);
extern char *setenv(const char *name, const char *value, int overwrite);
/* comment them out if gcc complains about redeclarations. */
/* append something to pythonpath */
/* returns 0 on failure, 1 on creation of the env variable and 2 on an append
oh, and -1 in cases of catastrophic miscompilation */
int myPyPathAppend(char *extrapath)
{
char *buffer = NULL;
/* size to taste... You should do this dynamically to avoid future buffer overrun attacks*/
char eventual_path[1024] = { 0 };
/* take note: after getenv, buffer points to an external constant character buffer.
do NOT try to modify it directly. use strcpy(char *dest, char *src)
(he says knowingly... is she into photography mate?)
*/
if( (buffer = getenv("PYTHONPATH")) == NULL )
{
/* we're here because PYTHONPATH is not already part of the environment. */

setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
/* did it go happen .. you should check this more rigorously*/
if( (buffer = getenv("PYTHONPATH")) == NULL)
{
/* we failed... abend. */
return 0;
}
else
{
/* success! cheers! */
return 1;
}
return -1; /* dead code... should never reach here */
}
else
{
/* PYTHONPATH already exists. append ';', then our new path and update it. */

Here. I might not want to add ';' or ':', depending on the OS (probably
not)? I can solve this with #ifdefs for WIN32 and Linux, but everytime I
want to run it on a new system, I'd have to find out the delimiter... It
also needs a buffer underrun check on that [1024]. I'll stick with the
PyRun_... for now, but I'll definitely save this code, so thanks. Again.

Also, found this piece from sys.path docs (now that my net is up and
running):
... "A program is free to modify this (sys.path) list for its own
purposes." ...

/* find the "=" in the buffer...
from string.h
extern char *strstr (__const char *__haystack, __const char *__needle)
there's a better way to do this, but I can't recall the function off the top of my head
*/
buffer = strstr(buffer, "=") + 1; /* +1 because buffer points to the equals. we want the string starting after it. */

/* copy the old PYTHONPATH string */
strcpy(eventual_path, buffer);
strcat(eventual_path, ";");
strcat(eventual_path, extrapath);
setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
/* did it go happen .. you should check this more rigorously*/
if( (buffer = getenv("PYTHONPATH")) == NULL)
{
/* we failed... abend. */
return 0;
}
else
{
/* success! cheers! */
return 2;
}
return -1; /* dead code... should never reach here */
}

One if and two elses, have you started "getting ready" for the party
already? ;) Or maybe it's fuzzy logic ;)
else
{
/* PYTHONPATH already exists. append ';', then our new path and update it. */
/* find the "=" in the buffer...

.... snip ...
}
return -1; /* deader code... should *really* never reach here */
}
---untested code---
I haven't tested, compiled or even read through this code.
I'm late for a party and still added comments
That means you get punctuation patrol :p
Check the semicolons, check the braces
Hey, I hear that in some companies they call this teamwork methodology
"extreme-programming" We're buzzword compliant!

- Tero

--
 
J

Jeff Epler

Take a look at:
http://www.python.org/doc/current/ext/pure-embedding.html. Basically,
it's about same (and I tested with it too, same results: failing)

Here's the program I came up with:

#include <Python.h>

int
main(int argc, char *argv[])
{
Py_SetProgramName(argv[0]);
Py_Initialize();
PyRun_SimpleString("import sys; print sys.path, sys.executable");
PySys_SetArgv(argc-1, argv+1);
PyRun_SimpleString("import sys; print sys.path, sys.executable");
Py_Finalize();
return 0;
}

This program demonstrates that the initial sys.path element
corresponding to the location of the script is created when
PySys_SetArgv() is called:

$ ./embed
['/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]
['', '/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]

That first element depends on the location of the script, as shown here:

$ ./embed /tmp/x
['/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]
['/tmp', '/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]

I don't know where this is documented---PySys_SetArgv is mentioned in
the "api" document, but this side-effect is not:
http://www.python.org/doc/current/api/embedding.html#l2h-35
http://www.python.org/doc/current/api/initialization.html#l3h-713
You might want to explore just what PySys_SetArgv and submit a
documentation patch on sourceforge, both for the api document and for
the ext document. Interestingly, calling PySys_SetArgv multiple times
inserts multiple items in sys.path.

Jeff
 
S

Samuel Walters

Ack... definitely not pep-7 compliant code

And to think... A co-worker once called me "obsessive" about indention.
(They say "obsessive," I say "consistent and legible.")

Well, It seems PAN likes to perform a stealth word-wrap when you edit from
the non-attached editor. (I used vim because I'm comfy hammering out code
there) Nope, on second look, I must have accidentally mashed some keys
while in vim. That extraneous "else" block is just a copy of the "else"
block.

If you run it through indent, and clean up a couple of oddball line-wraps
it will at least be legible. Remove the unneeded prototypes, embarrassing
extra else block, and it will compile. Remove the whole block about
searching for the = and fix the typo in the second setenv call (extrapath
should be eventual_path) and it will run without segfault. Then replace
the ";" with ":" and it will run properly on a linux system. For myself,
I made the snippet a little more general and added the option of choosing
which pathlist env variable and changed the function name to
envPathAppend. Then it's not a bad little snippet.


|Thus Spake Tero Pihlajakoski On the now historical date of Tue, 06 Jan
2004 01:04:31 +0000|
Here. I might not want to add ';' or ':', depending on the OS (probably
not)? I can solve this with #ifdefs for WIN32 and Linux, but everytime I
want to run it on a new system, I'd have to find out the delimiter... It
also needs a buffer underrun check on that [1024]. I'll stick with the
PyRun_... for now, but I'll definitely save this code, so thanks. Again.

That buffer overrun is not the only bit of braindead logic in that
snippet. The "success/failure" condition testing leaves a couple of loose
ends.
Also, found this piece from sys.path docs (now that my net is up and
running):
... "A program is free to modify this (sys.path) list for its own
purposes." ...

Hmmm... I suppose it's a matter of aesthetics and scope of the project
you're working on. If you're writing a big program for which the python
interpretor is only one small or medium part of the functionality and the
majority is c/c++, then you're opening up a whole big realm of
cross-platform issues. However, if the bulk of your code is python, then
perhaps you should have the main program be in python with your extra c
code as a module. That would be cleaner and probably more portable.

I suspect that at this point you're just trying to tinker with the
interface, then use what's most comfortable. Now at least, you have an
option. Besides, it seems that someone already posted a cleaner fix than
I could have given you.

HTH

Sam Walters.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,764
Messages
2,569,565
Members
45,041
Latest member
RomeoFarnh

Latest Threads

Top