Python/C and PYTHONPATH

Discussion in 'Python' started by Tero Pihlajakoski, Jan 5, 2004.

  1. Hi,

    I've been experimenting on embedding Python to a C software, and ran into
    a little problem with PYTHONPATH (I'm running on linux). Here's the deal:

    When trying to call PyImport_Import("xyz"), Python returns an error "No
    module named xyz". The problem _seems_ to be that I had no PYTHONPATH
    variable defined (though python-shell works ok regardless), since the
    following in bash helps:

    <try, no go>
    PYTHONPATH=
    export PYTHONPATH
    <try, works ok>

    I'm not defining anything as the PATH, and now it works. Why doesn't it
    look from "." without a dummy PYTHONPATH? Or is this actually a bug fixed
    in a newer release (running 2.3.?)?

    What is the "official" way to solve this? PySetProgramName()? Shell
    script?

    Thanks,
    - Tero

    --
     
    Tero Pihlajakoski, Jan 5, 2004
    #1
    1. Advertising

  2. Tero Pihlajakoski

    Jeff Epler Guest

    You'd need to tell us how you're invoking Python. When the Python
    executable is invoked without arguments, the current directory is on the
    sys.path:

    $ python
    Python 2.2.2 (#1, Feb 24 2003, 19:13:11)
    [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.path[0]

    ''

    When you run a script, the directory of that script is in sys.path[0]:
    $ echo "import sys; print sys.path[0]" > /tmp/script.py
    $ python /tmp/script.py
    /tmp

    Some Python editors may put the directory where they reside on sys.path,
    but not the current directory. They might do other startup that is
    different from normal. The newest Python I have on this machine
    is 2.3b1, and its idle puts these elements at the start of Pythonpath:

    Python 2.3b1 (#16, May 19 2003, 10:22:28)
    [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
    Type "copyright", "credits" or "license" for more information.
    IDLE 0.8 -- press F1 for help
    >>> import sys
    >>> sys.path[:2]

    ['/home/jepler', '/usr/local/bin']

    .... so instead of "the current directory" ('' or '.'), sys.path[0]
    is "the directory idle was invoked from", and that's followed by "the
    directory where the idle script lives".

    Jeff
     
    Jeff Epler, Jan 5, 2004
    #2
    1. Advertising

  3. Jeff Epler <> wrote:
    > You'd need to tell us how you're invoking Python. When the Python


    Take a look at:
    http://www.python.org/doc/current/ext/pure-embedding.html. Basically,
    it's about same (and I tested with it too, same results: failing)

    > executable is invoked without arguments, the current directory is on the
    > sys.path:


    > $ python
    > Python 2.2.2 (#1, Feb 24 2003, 19:13:11)
    > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2
    > Type "help", "copyright", "credits" or "license" for more information.
    >>>> import sys
    >>>> sys.path[0]

    > ''


    > When you run a script, the directory of that script is in sys.path[0]:
    > $ echo "import sys; print sys.path[0]" > /tmp/script.py
    > $ python /tmp/script.py
    > /tmp


    If I run "python" as a shell (python <scriptname>), everything works ok.
    Says path is ok: [ '', '<thelib>', ... ]. However, from C, something works
    differently: PyRun_SimpleString("import sys\nprint sys.path"); Shows no ''
    (the rest is ok). The question is: Why is there no '', and what is the
    "official/safe" way to set it? Just insert it to the list? (Setting
    "PYTHONPATH= " helps, but I'd like it to run "off the box")

    Tero
     
    Tero Pihlajakoski, Jan 5, 2004
    #3
  4. Fair warning: At this point, I couldn't even get the example you quoted
    to link properly. Of course, I didn't dig too deep, as I'm in a hurry.

    throw this c code into a .h file (my name for it is envdump.h)

    ---code---
    #include <stdio.h>

    extern char **environ;

    void fdump_env(FILE *fd)
    {
    char **ptr;
    int i = 0;
    ptr = environ;
    while(ptr != NULL)
    {
    fprintf(fd, "%i : %s\n", ptr);
    i++;
    }
    }


    void dump_env()
    {
    char **ptr;
    int i = 0;
    ptr = environ;
    while(ptr != NULL)
    {
    printf("%i : %s\n", i, ptr);
    i++;
    }
    }
    ---code---

    Now, just inside the main function (or, perhaps, wherever you please, but
    I'd try the line right before the first if statement) call dump_env() and
    it will spit out all the environment variables that your C program
    received. You might find something there that's of interest.

    Or, call setenv ("man 3 setenv") to manually set a python-path.
    you can use that in combo with getenv ("man 3 getenv") to append to an
    existing PYTHONPATH, if you so choose.

    Give it a go, please let me know how it turns out. If this doesn't shed
    some light on the matter I might have another idea once I'm not rushing
    around.

    HTH

    Sam Walters

    --
    Never forget the halloween documents.
    http://www.opensource.org/halloween/
    """ Where will Microsoft try to drag you today?
    Do you really want to go there?"""
     
    Samuel Walters, Jan 5, 2004
    #4
  5. Samuel Walters <> wrote:

    > Fair warning: At this point, I couldn't even get the example you quoted
    > to link properly. Of course, I didn't dig too deep, as I'm in a hurry.


    > ---code---


    .... snip ...

    > Or, call setenv ("man 3 setenv") to manually set a python-path.
    > you can use that in combo with getenv ("man 3 getenv") to append to an
    > existing PYTHONPATH, if you so choose.


    > Give it a go, please let me know how it turns out. If this doesn't shed
    > some light on the matter I might have another idea once I'm not rushing
    > around.


    (Sorry for the delay, I was offline)

    I'll see if it's actually the C-part that's causing problems, but I
    worked it around by adding:

    PyRun_SimpleString("import sys\nsys.path.insert(0,'')");

    right after the Py_Initialize().

    Works ok. Guess I'm "allowed" to do that(?)

    Thanks,
    - Tero

    --
     
    Tero Pihlajakoski, Jan 5, 2004
    #5
  6. |Thus Spake Tero Pihlajakoski On the now historical date of Mon, 05 Jan
    2004 22:39:19 +0000|
    > I'll see if it's actually the C-part that's causing problems, but I worked
    > it around by adding:
    >
    > PyRun_SimpleString("import sys\nsys.path.insert(0,'')");
    >
    > right after the Py_Initialize().
    >
    > Works ok. Guess I'm "allowed" to do that(?)

    IMLK (In My Limited Knowledge) that seems okay, but it also feels a bit
    ham-handed. try this snippet:

    ---untested code---
    #include <string.h>
    #include <stdlib.h>
    /* prototyping may not be neccessary... dunno...*/
    extern char *getenv(const char *name);
    extern char *setenv(const char *name, const char *value, int overwrite);
    /* comment them out if gcc complains about redeclarations. */

    /* append something to pythonpath */
    /* returns 0 on failure, 1 on creation of the env variable and 2 on an append
    oh, and -1 in cases of catastrophic miscompilation */
    int myPyPathAppend(char *extrapath)
    {
    char *buffer = NULL;
    /* size to taste... You should do this dynamically to avoid future buffer overrun attacks*/
    char eventual_path[1024] = { 0 };
    /* take note: after getenv, buffer points to an external constant character buffer.
    do NOT try to modify it directly. use strcpy(char *dest, char *src)
    (he says knowingly... is she into photography mate?)
    */
    if( (buffer = getenv("PYTHONPATH")) == NULL )
    {
    /* we're here because PYTHONPATH is not already part of the environment. */

    setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
    /* did it go happen .. you should check this more rigorously*/
    if( (buffer = getenv("PYTHONPATH")) == NULL)
    {
    /* we failed... abend. */
    return 0;
    }
    else
    {
    /* success! cheers! */
    return 1;
    }
    return -1; /* dead code... should never reach here */
    }
    else
    {
    /* PYTHONPATH already exists. append ';', then our new path and update it. */

    /* find the "=" in the buffer...
    from string.h
    extern char *strstr (__const char *__haystack, __const char *__needle)
    there's a better way to do this, but I can't recall the function off the top of my head
    */
    buffer = strstr(buffer, "=") + 1; /* +1 because buffer points to the equals. we want the string starting after it. */

    /* copy the old PYTHONPATH string */
    strcpy(eventual_path, buffer);
    strcat(eventual_path, ";");
    strcat(eventual_path, extrapath);

    setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
    /* did it go happen .. you should check this more rigorously*/
    if( (buffer = getenv("PYTHONPATH")) == NULL)
    {
    /* we failed... abend. */
    return 0;
    }
    else
    {
    /* success! cheers! */
    return 2;
    }
    return -1; /* dead code... should never reach here */
    }
    else
    {
    /* PYTHONPATH already exists. append ';', then our new path and update it. */

    /* find the "=" in the buffer...
    from string.h
    extern char *strstr (__const char *__haystack, __const char *__needle)
    there's a better way to do this, but I can't recall the function off the top of my head
    */
    buffer = strstr(buffer, "=") + 1; /* +1 because buffer points to the equals. we want the string starting after it. */

    /* copy the old PYTHONPATH string */
    strcpy(eventual_path, buffer);
    strcat(eventual_path, ";");
    strcat(eventual_path, extrapath);

    setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
    /* did it go happen .. you should check this more rigorously*/
    if( (buffer = getenv("PYTHONPATH")) == NULL)
    {
    /* we failed... abend. */
    return 0;
    }
    else
    {
    /* success! cheers! */
    return 2;
    }
    return -1; /* dead code... should never reach here */
    }
    return -1; /* deader code... should *really* never reach here */
    }
    ---untested code---
    I haven't tested, compiled or even read through this code.
    I'm late for a party and still added comments
    That means you get punctuation patrol :p
    Check the semicolons, check the braces
    Hey, I hear that in some companies they call this teamwork methodology
    "extreme-programming" We're buzzword compliant!

    HTH
    (Danm... I'm such a code monkey)

    --
    Never forget the halloween documents.
    http://www.opensource.org/halloween/
    """ Where will Microsoft try to drag you today?
    Do you really want to go there?"""
     
    Samuel Walters, Jan 6, 2004
    #6
  7. Samuel Walters <> wrote:
    > |Thus Spake Tero Pihlajakoski On the now historical date of Mon, 05 Jan
    > 2004 22:39:19 +0000|
    >> I'll see if it's actually the C-part that's causing problems, but I worked
    >> it around by adding:
    >>
    >> PyRun_SimpleString("import sys\nsys.path.insert(0,'')");
    >>
    >> right after the Py_Initialize().
    >>
    >> Works ok. Guess I'm "allowed" to do that(?)

    > IMLK (In My Limited Knowledge) that seems okay, but it also feels a bit
    > ham-handed. try this snippet:


    Ok, there are comments here, somewhere:

    > ---untested code---
    > #include <string.h>
    > #include <stdlib.h>
    > /* prototyping may not be neccessary... dunno...*/
    > extern char *getenv(const char *name);
    > extern char *setenv(const char *name, const char *value, int overwrite);
    > /* comment them out if gcc complains about redeclarations. */


    > /* append something to pythonpath */
    > /* returns 0 on failure, 1 on creation of the env variable and 2 on an append
    > oh, and -1 in cases of catastrophic miscompilation */
    > int myPyPathAppend(char *extrapath)
    > {
    > char *buffer = NULL;
    > /* size to taste... You should do this dynamically to avoid future buffer overrun attacks*/
    > char eventual_path[1024] = { 0 };
    > /* take note: after getenv, buffer points to an external constant character buffer.
    > do NOT try to modify it directly. use strcpy(char *dest, char *src)
    > (he says knowingly... is she into photography mate?)
    > */
    > if( (buffer = getenv("PYTHONPATH")) == NULL )
    > {
    > /* we're here because PYTHONPATH is not already part of the environment. */
    >
    > setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
    > /* did it go happen .. you should check this more rigorously*/
    > if( (buffer = getenv("PYTHONPATH")) == NULL)
    > {
    > /* we failed... abend. */
    > return 0;
    > }
    > else
    > {
    > /* success! cheers! */
    > return 1;
    > }
    > return -1; /* dead code... should never reach here */
    > }
    > else
    > {
    > /* PYTHONPATH already exists. append ';', then our new path and update it. */


    Here. I might not want to add ';' or ':', depending on the OS (probably
    not)? I can solve this with #ifdefs for WIN32 and Linux, but everytime I
    want to run it on a new system, I'd have to find out the delimiter... It
    also needs a buffer underrun check on that [1024]. I'll stick with the
    PyRun_... for now, but I'll definitely save this code, so thanks. Again.

    Also, found this piece from sys.path docs (now that my net is up and
    running):
    ... "A program is free to modify this (sys.path) list for its own
    purposes." ...


    >
    > /* find the "=" in the buffer...
    > from string.h
    > extern char *strstr (__const char *__haystack, __const char *__needle)
    > there's a better way to do this, but I can't recall the function off the top of my head
    > */
    > buffer = strstr(buffer, "=") + 1; /* +1 because buffer points to the equals. we want the string starting after it. */
    >
    > /* copy the old PYTHONPATH string */
    > strcpy(eventual_path, buffer);
    > strcat(eventual_path, ";");
    > strcat(eventual_path, extrapath);


    > setenv("PYTHONPATH", extrapath, 1); /* the last argument makes sure that we create the env var*/
    > /* did it go happen .. you should check this more rigorously*/
    > if( (buffer = getenv("PYTHONPATH")) == NULL)
    > {
    > /* we failed... abend. */
    > return 0;
    > }
    > else
    > {
    > /* success! cheers! */
    > return 2;
    > }
    > return -1; /* dead code... should never reach here */
    > }


    One if and two elses, have you started "getting ready" for the party
    already? ;) Or maybe it's fuzzy logic ;)

    > else
    > {
    > /* PYTHONPATH already exists. append ';', then our new path and update it. */
    > /* find the "=" in the buffer...


    .... snip ...

    > }
    > return -1; /* deader code... should *really* never reach here */
    > }
    > ---untested code---
    > I haven't tested, compiled or even read through this code.
    > I'm late for a party and still added comments
    > That means you get punctuation patrol :p
    > Check the semicolons, check the braces
    > Hey, I hear that in some companies they call this teamwork methodology
    > "extreme-programming" We're buzzword compliant!


    - Tero

    --
     
    Tero Pihlajakoski, Jan 6, 2004
    #7
  8. Tero Pihlajakoski

    Jeff Epler Guest

    On Mon, Jan 05, 2004 at 04:17:52PM +0000, Tero Pihlajakoski wrote:
    > Jeff Epler <> wrote:
    > > You'd need to tell us how you're invoking Python. When the Python

    >
    > Take a look at:
    > http://www.python.org/doc/current/ext/pure-embedding.html. Basically,
    > it's about same (and I tested with it too, same results: failing)


    Here's the program I came up with:

    #include <Python.h>

    int
    main(int argc, char *argv[])
    {
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    PyRun_SimpleString("import sys; print sys.path, sys.executable");
    PySys_SetArgv(argc-1, argv+1);
    PyRun_SimpleString("import sys; print sys.path, sys.executable");
    Py_Finalize();
    return 0;
    }

    This program demonstrates that the initial sys.path element
    corresponding to the location of the script is created when
    PySys_SetArgv() is called:

    $ ./embed
    ['/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]
    ['', '/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]

    That first element depends on the location of the script, as shown here:

    $ ./embed /tmp/x
    ['/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]
    ['/tmp', '/usr/lib/python2.2', '/usr/lib/python2.2/plat-linux2', ...]

    I don't know where this is documented---PySys_SetArgv is mentioned in
    the "api" document, but this side-effect is not:
    http://www.python.org/doc/current/api/embedding.html#l2h-35
    http://www.python.org/doc/current/api/initialization.html#l3h-713
    You might want to explore just what PySys_SetArgv and submit a
    documentation patch on sourceforge, both for the api document and for
    the ext document. Interestingly, calling PySys_SetArgv multiple times
    inserts multiple items in sys.path.

    Jeff
     
    Jeff Epler, Jan 6, 2004
    #8
  9. Ack... definitely not pep-7 compliant code

    And to think... A co-worker once called me "obsessive" about indention.
    (They say "obsessive," I say "consistent and legible.")

    Well, It seems PAN likes to perform a stealth word-wrap when you edit from
    the non-attached editor. (I used vim because I'm comfy hammering out code
    there) Nope, on second look, I must have accidentally mashed some keys
    while in vim. That extraneous "else" block is just a copy of the "else"
    block.

    If you run it through indent, and clean up a couple of oddball line-wraps
    it will at least be legible. Remove the unneeded prototypes, embarrassing
    extra else block, and it will compile. Remove the whole block about
    searching for the = and fix the typo in the second setenv call (extrapath
    should be eventual_path) and it will run without segfault. Then replace
    the ";" with ":" and it will run properly on a linux system. For myself,
    I made the snippet a little more general and added the option of choosing
    which pathlist env variable and changed the function name to
    envPathAppend. Then it's not a bad little snippet.


    |Thus Spake Tero Pihlajakoski On the now historical date of Tue, 06 Jan
    2004 01:04:31 +0000|

    > Here. I might not want to add ';' or ':', depending on the OS (probably
    > not)? I can solve this with #ifdefs for WIN32 and Linux, but everytime I
    > want to run it on a new system, I'd have to find out the delimiter... It
    > also needs a buffer underrun check on that [1024]. I'll stick with the
    > PyRun_... for now, but I'll definitely save this code, so thanks. Again.


    That buffer overrun is not the only bit of braindead logic in that
    snippet. The "success/failure" condition testing leaves a couple of loose
    ends.

    > Also, found this piece from sys.path docs (now that my net is up and
    > running):
    > ... "A program is free to modify this (sys.path) list for its own
    > purposes." ...


    Hmmm... I suppose it's a matter of aesthetics and scope of the project
    you're working on. If you're writing a big program for which the python
    interpretor is only one small or medium part of the functionality and the
    majority is c/c++, then you're opening up a whole big realm of
    cross-platform issues. However, if the bulk of your code is python, then
    perhaps you should have the main program be in python with your extra c
    code as a module. That would be cleaner and probably more portable.

    I suspect that at this point you're just trying to tinker with the
    interface, then use what's most comfortable. Now at least, you have an
    option. Besides, it seems that someone already posted a cleaner fix than
    I could have given you.

    HTH

    Sam Walters.

    --
    Never forget the halloween documents.
    http://www.opensource.org/halloween/
    """ Where will Microsoft try to drag you today?
    Do you really want to go there?"""
     
    Samuel Walters, Jan 6, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. klappnase

    Re: Python..Tkinter..PYTHONPATH..

    klappnase, Jul 30, 2003, in forum: Python
    Replies:
    0
    Views:
    404
    klappnase
    Jul 30, 2003
  2. Steven Taschuk

    Re: Python..Tkinter..PYTHONPATH..

    Steven Taschuk, Jul 30, 2003, in forum: Python
    Replies:
    0
    Views:
    417
    Steven Taschuk
    Jul 30, 2003
  3. rh0dium
    Replies:
    3
    Views:
    493
    rh0dium
    Jun 23, 2006
  4. David Cournapeau

    PYTHONPATH and multiple python versions

    David Cournapeau, Jun 5, 2009, in forum: Python
    Replies:
    0
    Views:
    411
    David Cournapeau
    Jun 5, 2009
  5. Replies:
    2
    Views:
    653
    Martin v. Loewis
    Apr 12, 2010
Loading...

Share This Page