Customizing the python search path depending on source directory

Discussion in 'Python' started by Peter Schwalm, Apr 7, 2004.

  1. I'd like to modify the python search path depending on the source
    directory of the script being started. The reason is: I use a version
    control system, and the python scripts and modules are migrated
    through several stages before they are used in production. A python
    script running in a development environment should use different
    modules - possibly in other directories then the script source
    directory - than the same script running in a production environment.

    Ideally this should work without any modifications to the python
    programs, and the search path should be set up before the script
    invoked by the user gains control.

    My idea was to use site.py / sitecustomize.py in lib/site-packages and
    to have sitecustomize.py invoke a directory-specific customizing
    module - say dirCustomize.py - located in the script source directory.

    But it seems that this is the wrong place, because at the moment
    sitecustomize is imported

    a) the source directory of the script is still not in sys.path
    b) sys.argv is still not set, so I can't find out the source
    directory via sys.argv[0] and temporarily add it to sys.path for the
    invocation of dirCustomize.py.
    c) also sys.modules["__main__"].__file__ is still not set

    The following "sitecustomize.py" shows these difficulties:
    #-------------------------------------------------------------------------
    print "start siteCustomize.py ..."
    import sys
    import os

    try:
    print "argv=", sys.argv
    except AttributeError:
    print ""
    print "sys.argv does still not exist!"

    try:
    print "sys.modules[\"__main__\"]=",
    sys.modules["__main__"].__file__
    except AttributeError:
    print ""
    print "sys.modules[\"__main__\"].__file__ does still not exist"

    # ... shows that source directory is still not in sys.path
    print "sys.path="
    for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1, p1)

    print "... end siteCustomize.py"
    #-------------------------------------------------------------------------

    I'd not like to use the current working directory instead of the
    script directory, because the scripts may be started from anywhere
    directories by specifying the path name in the invocation.

    Does anyone have an idea?

    Thank you in advance,
    Peter
    Peter Schwalm, Apr 7, 2004
    #1
    1. Advertising

  2. Peter Schwalm

    Peter Hansen Guest

    Peter Schwalm wrote:

    > My idea was to use site.py / sitecustomize.py in lib/site-packages and
    > to have sitecustomize.py invoke a directory-specific customizing
    > module - say dirCustomize.py - located in the script source directory.
    >
    > But it seems that this is the wrong place, because at the moment
    > sitecustomize is imported
    >
    > a) the source directory of the script is still not in sys.path


    Are you sure? Looking at site.py, it appears that the sitecustomize
    import is done next-to-last, just before sys.setdefaultencoding is
    deleted. If you concluded this by examining sys.path manually at
    the point just as sitecustomize is imported, maybe you don't know
    that the '' entry in sys.path actually represents the source
    directory of the main script...

    It would seem your suggested solution above should actually work...

    -Peter
    Peter Hansen, Apr 7, 2004
    #2
    1. Advertising

  3. Peter Hansen <> wrote in message news:> >
    .....
    > > a) the source directory of the script is still not in sys.path

    >
    > Are you sure? Looking at site.py, it appears that the sitecustomize
    > import is done next-to-last, just before sys.setdefaultencoding is
    > deleted. If you concluded this by examining sys.path manually at
    > the point just as sitecustomize is imported, maybe you don't know
    > that the '' entry in sys.path actually represents the source
    > directory of the main script...


    Thank you for your answer. But I'm afraid it's not the source
    directory but the current working directory. You can see the
    difference if you copy the passage

    "for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1,
    p1)"

    to a "normal script". If you start that "normal script" from a
    directory other than the source directory, you can see that both
    directories are included in sys.path. If you run this code inside
    sitecustomize.py this is not the case.

    Thank you
    Peter
    Peter Schwalm, Apr 7, 2004
    #3
  4. Peter Schwalm

    Peter Hansen Guest

    Peter Schwalm wrote:

    > Peter Hansen <> wrote in message news:> >
    > Thank you for your answer. But I'm afraid it's not the source
    > directory but the current working directory. You can see the
    > difference if you copy the passage
    >
    > "for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1,
    > p1)"
    >
    > to a "normal script". If you start that "normal script" from a
    > directory other than the source directory, you can see that both
    > directories are included in sys.path. If you run this code inside
    > sitecustomize.py this is not the case.


    On the contrary. When I start that from another directory which
    is *not* in the path, I do not see that directory in the sys.path
    at all, but merely the directory in which the script itself is
    stored. This is the defined behaviour of the Python sys.path
    stuff and you must be misinterpreting something.

    The current directory is *not* added to sys.path, unless it
    just happens that the main script is in the current directory.

    -Peter
    Peter Hansen, Apr 7, 2004
    #4
  5. Peter Hansen <> wrote in message news:<>...
    > Peter Schwalm wrote:
    >
    > > Peter Hansen <> wrote in message news:> >
    > > Thank you for your answer. But I'm afraid it's not the source
    > > directory but the current working directory. You can see the
    > > difference if you copy the passage
    > >
    > > "for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1,
    > > p1)"
    > >
    > > to a "normal script". If you start that "normal script" from a
    > > directory other than the source directory, you can see that both
    > > directories are included in sys.path. If you run this code inside
    > > sitecustomize.py this is not the case.

    >
    > On the contrary. When I start that from another directory which
    > is *not* in the path, I do not see that directory in the sys.path
    > at all, but merely the directory in which the script itself is
    > stored. This is the defined behaviour of the Python sys.path
    > stuff and you must be misinterpreting something.
    >
    > The current directory is *not* added to sys.path, unless it
    > just happens that the main script is in the current directory.
    >

    Again, thank you for your answer.
    This answer lets me suspect that you have a different version of
    python installed. Mine is 2.3.2 for win32 (activepython).

    To show you what I observe I have modified my "sitecustomize.py"
    slightly and have copied the unmodified text do script "test1.py"

    The source for both looks as follows:

    #-------------------------------------------------------------------------
    import sys
    import os
    print "--->", __file__, "..."
    print 64 * "-"
    print "python-version=", sys.version
    try:
    print "argv=", sys.argv
    except AttributeError:
    print ""
    print "sys.argv does still not exist!"

    try:
    print "sys.modules[\"__main__\"]=",
    sys.modules["__main__"].__file__
    except AttributeError:
    print ""
    print "sys.modules[\"__main__\"].__file__ does still not exist"

    # ... shows that source directory is still not in sys.path
    print "sys.path="
    for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1, p1)

    print 64 * "-"
    print "<---", __file__, "..."
    #-------------------------------------------------------------------------

    Here is the output from test1.py when run it's own source directory:

    E:\temp>test1.py
    ---> C:\Python23\lib\sitecustomize.pyc ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv=
    sys.argv does still not exist!
    sys.modules["__main__"]=
    sys.modules["__main__"].__file__ does still not exist
    sys.path=
    00: E:\temp
    01: d:\d\d8\AFP\r01\python
    02: C:\WINNT\System32\python23.zip
    03: C:\Python23\Lib\site-packages\Pythonwin
    04: C:\Python23\Lib\site-packages\win32
    05: C:\Python23\Lib\site-packages\win32\lib
    06: C:\Python23\Lib\site-packages
    07: C:\Python23\DLLs
    08: C:\Python23\lib
    09: C:\Python23\lib\plat-win
    10: C:\Python23\lib\lib-tk
    11: C:\Python23
    12: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- C:\Python23\lib\sitecustomize.pyc ...
    ---> E:\temp\test1.py ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv= ['E:\\temp\\test1.py']
    sys.modules["__main__"]= E:\temp\test1.py
    sys.path=
    00: E:\temp
    01: E:\temp
    02: d:\d\d8\AFP\r01\python
    03: C:\WINNT\System32\python23.zip
    04: C:\Python23\Lib\site-packages\Pythonwin
    05: C:\Python23\Lib\site-packages\win32
    06: C:\Python23\Lib\site-packages\win32\lib
    07: C:\Python23\Lib\site-packages
    08: C:\Python23\DLLs
    09: C:\Python23\lib
    10: C:\Python23\lib\plat-win
    11: C:\Python23\lib\lib-tk
    12: C:\Python23
    13: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- E:\temp\test1.py ...

    E:\temp>

    And here is the output from test1.py when run from a different
    directory (from subdirectory sub1):


    E:\temp\sub1>..\test1.py
    ---> C:\Python23\lib\sitecustomize.pyc ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv=
    sys.argv does still not exist!
    sys.modules["__main__"]=
    sys.modules["__main__"].__file__ does still not exist
    sys.path=
    00: E:\temp\sub1
    01: d:\d\d8\AFP\r01\python
    02: C:\WINNT\System32\python23.zip
    03: C:\Python23\Lib\site-packages\Pythonwin
    04: C:\Python23\Lib\site-packages\win32
    05: C:\Python23\Lib\site-packages\win32\lib
    06: C:\Python23\Lib\site-packages
    07: C:\Python23\DLLs
    08: C:\Python23\lib
    09: C:\Python23\lib\plat-win
    10: C:\Python23\lib\lib-tk
    11: C:\Python23
    12: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- C:\Python23\lib\sitecustomize.pyc ...
    ---> E:\temp\test1.py ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv= ['E:\\temp\\test1.py']
    sys.modules["__main__"]= E:\temp\test1.py
    sys.path=
    00: E:\temp
    01: E:\temp\sub1
    02: d:\d\d8\AFP\r01\python
    03: C:\WINNT\System32\python23.zip
    04: C:\Python23\Lib\site-packages\Pythonwin
    05: C:\Python23\Lib\site-packages\win32
    06: C:\Python23\Lib\site-packages\win32\lib
    07: C:\Python23\Lib\site-packages
    08: C:\Python23\DLLs
    09: C:\Python23\lib
    10: C:\Python23\lib\plat-win
    11: C:\Python23\lib\lib-tk
    12: C:\Python23
    13: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- E:\temp\test1.py ...

    E:\temp\sub1>

    As you can see in the second output, the source directory is *not*
    included in sys.path during the processing of sitecustomize.py, but
    only during the processing of the "normal script" test1.py.

    Does the output look different if you run these programs in your
    environment?

    Thank you
    Peter
    Peter Schwalm, Apr 8, 2004
    #5
  6. Peter Schwalm

    Peter Hansen Guest

    Peter Schwalm wrote:

    > Does the output look different if you run these programs in your
    > environment?


    It may take me a moment to get the precise same configuration
    on my machine, because I don't have it set up to execute
    ..py scripts automatically at the command line just by typing
    their names. I have to manually invoke "python" and pass it
    the name of the script.

    Therefore, before I do this (I need to learn how, first),
    can you confirm that you get identical results if in each
    place where you typed "test1.py" you instead typed the
    equivalent with, for example, "python test1.py"?

    It's possible Windows' technique for directly invoking
    scripts your way screws up the sys.path stuff...

    (I _am_ using "stock" Python 2.3.3 on XP, by the way. I
    doubt ActiveState's behaves any different in this respect.)

    -Peter
    Peter Hansen, Apr 8, 2004
    #6
  7. Peter Hansen <> wrote in message news:<>...
    > Peter Schwalm wrote:
    >
    > ... can you confirm that you get identical results if in each
    > place where you typed "test1.py" you instead typed the
    > equivalent with, for example, "python test1.py"?
    >


    I have run the programs with "python <scriptname>.py" now. For my
    purposes the results look identical - the source directory is still
    not accessible during the processing of sitecustomize.py.

    Here are the outputs of the explicit invocations:

    Invocation from source directory:


    E:\temp>python test1.py
    ---> C:\Python23\lib\sitecustomize.pyc ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv=
    sys.argv does still not exist!
    sys.modules["__main__"]=
    sys.modules["__main__"].__file__ does still not exist
    sys.path=
    00: C:\WINNT\System32\python23.zip
    01: C:\Python23\Lib\site-packages\Pythonwin
    02: C:\Python23\Lib\site-packages\win32
    03: C:\Python23\Lib\site-packages\win32\lib
    04: C:\Python23\Lib\site-packages
    05: C:\Python23\DLLs
    06: C:\Python23\lib
    07: C:\Python23\lib\plat-win
    08: C:\Python23\lib\lib-tk
    09: C:\Python23
    10: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- C:\Python23\lib\sitecustomize.pyc ...
    ---> test1.py ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv= ['test1.py']
    sys.modules["__main__"]= test1.py
    sys.path=
    00: E:\temp
    01: C:\WINNT\System32\python23.zip
    02: C:\Python23\Lib\site-packages\Pythonwin
    03: C:\Python23\Lib\site-packages\win32
    04: C:\Python23\Lib\site-packages\win32\lib
    05: C:\Python23\Lib\site-packages
    06: C:\Python23\DLLs
    07: C:\Python23\lib
    08: C:\Python23\lib\plat-win
    09: C:\Python23\lib\lib-tk
    10: C:\Python23
    11: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- test1.py ...

    E:\temp>


    Invocation from different directory:

    E:\temp\sub1>python ..\test1.py
    ---> C:\Python23\lib\sitecustomize.pyc ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv=
    sys.argv does still not exist!
    sys.modules["__main__"]=
    sys.modules["__main__"].__file__ does still not exist
    sys.path=
    00: C:\WINNT\System32\python23.zip
    01: C:\Python23\Lib\site-packages\Pythonwin
    02: C:\Python23\Lib\site-packages\win32
    03: C:\Python23\Lib\site-packages\win32\lib
    04: C:\Python23\Lib\site-packages
    05: C:\Python23\DLLs
    06: C:\Python23\lib
    07: C:\Python23\lib\plat-win
    08: C:\Python23\lib\lib-tk
    09: C:\Python23
    10: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- C:\Python23\lib\sitecustomize.pyc ...
    ---> ..\test1.py ...
    ----------------------------------------------------------------
    python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
    (Intel)]
    argv= ['..\\test1.py']
    sys.modules["__main__"]= ..\test1.py
    sys.path=
    00: E:\temp
    01: C:\WINNT\System32\python23.zip
    02: C:\Python23\Lib\site-packages\Pythonwin
    03: C:\Python23\Lib\site-packages\win32
    04: C:\Python23\Lib\site-packages\win32\lib
    05: C:\Python23\Lib\site-packages
    06: C:\Python23\DLLs
    07: C:\Python23\lib
    08: C:\Python23\lib\plat-win
    09: C:\Python23\lib\lib-tk
    10: C:\Python23
    11: C:\Python23\lib\site-packages\PIL
    ----------------------------------------------------------------
    <--- ..\test1.py ...

    E:\temp\sub1>

    By the way, *if I remember that correctly*: to be able to invoke
    python scripts simply by name ("test1" instead of "python test1.py")
    you have to do
    two things:

    1) set the environment variable PATHEXT such that it contains
    ".py"(see sample output from set command)

    E:\temp>set pathext
    PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PY;.py;.pyw;
    .pyc;.pys;.pyo
    E:\temp>

    2) associate the file extension ".py" with c:\python23\python.exe
    right click a .py-file in explorer, select "open with" (my translation
    of the german "oeffnen mit") select python.exe as application and
    check "open always with".

    Thank you in advance
    Peter Schwalm
    Peter Schwalm, Apr 8, 2004
    #7
  8. Peter Schwalm

    Peter Hansen Guest

    Peter Schwalm wrote:
    > Peter Hansen <> wrote in message news:<>...
    >
    > I have run the programs with "python <scriptname>.py" now. For my
    > purposes the results look identical - the source directory is still
    > not accessible during the processing of sitecustomize.py.


    Thanks for your persistence, Peter. I've confirmed what you have
    observed here: during site.py processing (including near the end
    when "import sitecustomize" is tried) the directory containing the
    main script is not included in sys.path. Immediately after site.py
    is imported, sys.path does contain the proper script at the start.

    My guess is that recent changes to the import mechanism have broken
    this behaviour.

    Note that this change would also break sitecustomize processing in
    some installations, since it should (I believe) be possible to
    have a sitecustomize.py in the directory where the main script
    resides, but in Python 2.3.3 it is not imported at all.

    If anyone who knows about such things more than we do can confirm
    that this should in fact be considered a bug, I will add it to
    SourceForge.

    -Peter
    Peter Hansen, Apr 9, 2004
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. sgharvey
    Replies:
    6
    Views:
    406
    sgharvey
    Mar 23, 2008
  2. liveline
    Replies:
    2
    Views:
    660
    Jorgen Grahn
    May 16, 2010
  3. Abby Lee
    Replies:
    5
    Views:
    380
    Abby Lee
    Aug 2, 2004
  4. bdawg
    Replies:
    1
    Views:
    70
    Erwin Moller
    Feb 12, 2004
  5. Matt Garrish
    Replies:
    9
    Views:
    156
    Matt Garrish
    Jan 17, 2006
Loading...

Share This Page