Customizing the python search path depending on source directory

P

Peter Schwalm

I'd like to modify the python search path depending on the source
directory of the script being started. The reason is: I use a version
control system, and the python scripts and modules are migrated
through several stages before they are used in production. A python
script running in a development environment should use different
modules - possibly in other directories then the script source
directory - than the same script running in a production environment.

Ideally this should work without any modifications to the python
programs, and the search path should be set up before the script
invoked by the user gains control.

My idea was to use site.py / sitecustomize.py in lib/site-packages and
to have sitecustomize.py invoke a directory-specific customizing
module - say dirCustomize.py - located in the script source directory.

But it seems that this is the wrong place, because at the moment
sitecustomize is imported

a) the source directory of the script is still not in sys.path
b) sys.argv is still not set, so I can't find out the source
directory via sys.argv[0] and temporarily add it to sys.path for the
invocation of dirCustomize.py.
c) also sys.modules["__main__"].__file__ is still not set

The following "sitecustomize.py" shows these difficulties:
#-------------------------------------------------------------------------
print "start siteCustomize.py ..."
import sys
import os

try:
print "argv=", sys.argv
except AttributeError:
print ""
print "sys.argv does still not exist!"

try:
print "sys.modules[\"__main__\"]=",
sys.modules["__main__"].__file__
except AttributeError:
print ""
print "sys.modules[\"__main__\"].__file__ does still not exist"

# ... shows that source directory is still not in sys.path
print "sys.path="
for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1, p1)

print "... end siteCustomize.py"
#-------------------------------------------------------------------------

I'd not like to use the current working directory instead of the
script directory, because the scripts may be started from anywhere
directories by specifying the path name in the invocation.

Does anyone have an idea?

Thank you in advance,
Peter
 
P

Peter Hansen

Peter said:
My idea was to use site.py / sitecustomize.py in lib/site-packages and
to have sitecustomize.py invoke a directory-specific customizing
module - say dirCustomize.py - located in the script source directory.

But it seems that this is the wrong place, because at the moment
sitecustomize is imported

a) the source directory of the script is still not in sys.path
Click to expand...

Are you sure? Looking at site.py, it appears that the sitecustomize
import is done next-to-last, just before sys.setdefaultencoding is
deleted. If you concluded this by examining sys.path manually at
the point just as sitecustomize is imported, maybe you don't know
that the '' entry in sys.path actually represents the source
directory of the main script...

It would seem your suggested solution above should actually work...

-Peter
 
P

Peter Schwalm

Peter Hansen said:
Are you sure? Looking at site.py, it appears that the sitecustomize
import is done next-to-last, just before sys.setdefaultencoding is
deleted. If you concluded this by examining sys.path manually at
the point just as sitecustomize is imported, maybe you don't know
that the '' entry in sys.path actually represents the source
directory of the main script...
Click to expand...

Thank you for your answer. But I'm afraid it's not the source
directory but the current working directory. You can see the
difference if you copy the passage

"for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1,
p1)"

to a "normal script". If you start that "normal script" from a
directory other than the source directory, you can see that both
directories are included in sys.path. If you run this code inside
sitecustomize.py this is not the case.

Thank you
Peter
 
P

Peter Hansen

Peter said:
Thank you for your answer. But I'm afraid it's not the source
directory but the current working directory. You can see the
difference if you copy the passage

"for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1,
p1)"

to a "normal script". If you start that "normal script" from a
directory other than the source directory, you can see that both
directories are included in sys.path. If you run this code inside
sitecustomize.py this is not the case.
Click to expand...

On the contrary. When I start that from another directory which
is *not* in the path, I do not see that directory in the sys.path
at all, but merely the directory in which the script itself is
stored. This is the defined behaviour of the Python sys.path
stuff and you must be misinterpreting something.

The current directory is *not* added to sys.path, unless it
just happens that the main script is in the current directory.

-Peter
 
P

Peter Schwalm

Peter Hansen said:
On the contrary. When I start that from another directory which
is *not* in the path, I do not see that directory in the sys.path
at all, but merely the directory in which the script itself is
stored. This is the defined behaviour of the Python sys.path
stuff and you must be misinterpreting something.

The current directory is *not* added to sys.path, unless it
just happens that the main script is in the current directory.
Click to expand...
Again, thank you for your answer.
This answer lets me suspect that you have a different version of
python installed. Mine is 2.3.2 for win32 (activepython).

To show you what I observe I have modified my "sitecustomize.py"
slightly and have copied the unmodified text do script "test1.py"

The source for both looks as follows:

#-------------------------------------------------------------------------
import sys
import os
print "--->", __file__, "..."
print 64 * "-"
print "python-version=", sys.version
try:
print "argv=", sys.argv
except AttributeError:
print ""
print "sys.argv does still not exist!"

try:
print "sys.modules[\"__main__\"]=",
sys.modules["__main__"].__file__
except AttributeError:
print ""
print "sys.modules[\"__main__\"].__file__ does still not exist"

# ... shows that source directory is still not in sys.path
print "sys.path="
for ix1, p1 in enumerate(sys.path): print "%02.02d: %s" % (ix1, p1)

print 64 * "-"
print "<---", __file__, "..."
#-------------------------------------------------------------------------

Here is the output from test1.py when run it's own source directory:

E:\temp>test1.py
---> C:\Python23\lib\sitecustomize.pyc ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv=
sys.argv does still not exist!
sys.modules["__main__"]=
sys.modules["__main__"].__file__ does still not exist
sys.path=
00: E:\temp
01: d:\d\d8\AFP\r01\python
02: C:\WINNT\System32\python23.zip
03: C:\Python23\Lib\site-packages\Pythonwin
04: C:\Python23\Lib\site-packages\win32
05: C:\Python23\Lib\site-packages\win32\lib
06: C:\Python23\Lib\site-packages
07: C:\Python23\DLLs
08: C:\Python23\lib
09: C:\Python23\lib\plat-win
10: C:\Python23\lib\lib-tk
11: C:\Python23
12: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- C:\Python23\lib\sitecustomize.pyc ...
---> E:\temp\test1.py ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv= ['E:\\temp\\test1.py']
sys.modules["__main__"]= E:\temp\test1.py
sys.path=
00: E:\temp
01: E:\temp
02: d:\d\d8\AFP\r01\python
03: C:\WINNT\System32\python23.zip
04: C:\Python23\Lib\site-packages\Pythonwin
05: C:\Python23\Lib\site-packages\win32
06: C:\Python23\Lib\site-packages\win32\lib
07: C:\Python23\Lib\site-packages
08: C:\Python23\DLLs
09: C:\Python23\lib
10: C:\Python23\lib\plat-win
11: C:\Python23\lib\lib-tk
12: C:\Python23
13: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- E:\temp\test1.py ...

E:\temp>

And here is the output from test1.py when run from a different
directory (from subdirectory sub1):


E:\temp\sub1>..\test1.py
---> C:\Python23\lib\sitecustomize.pyc ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv=
sys.argv does still not exist!
sys.modules["__main__"]=
sys.modules["__main__"].__file__ does still not exist
sys.path=
00: E:\temp\sub1
01: d:\d\d8\AFP\r01\python
02: C:\WINNT\System32\python23.zip
03: C:\Python23\Lib\site-packages\Pythonwin
04: C:\Python23\Lib\site-packages\win32
05: C:\Python23\Lib\site-packages\win32\lib
06: C:\Python23\Lib\site-packages
07: C:\Python23\DLLs
08: C:\Python23\lib
09: C:\Python23\lib\plat-win
10: C:\Python23\lib\lib-tk
11: C:\Python23
12: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- C:\Python23\lib\sitecustomize.pyc ...
---> E:\temp\test1.py ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv= ['E:\\temp\\test1.py']
sys.modules["__main__"]= E:\temp\test1.py
sys.path=
00: E:\temp
01: E:\temp\sub1
02: d:\d\d8\AFP\r01\python
03: C:\WINNT\System32\python23.zip
04: C:\Python23\Lib\site-packages\Pythonwin
05: C:\Python23\Lib\site-packages\win32
06: C:\Python23\Lib\site-packages\win32\lib
07: C:\Python23\Lib\site-packages
08: C:\Python23\DLLs
09: C:\Python23\lib
10: C:\Python23\lib\plat-win
11: C:\Python23\lib\lib-tk
12: C:\Python23
13: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- E:\temp\test1.py ...

E:\temp\sub1>

As you can see in the second output, the source directory is *not*
included in sys.path during the processing of sitecustomize.py, but
only during the processing of the "normal script" test1.py.

Does the output look different if you run these programs in your
environment?

Thank you
Peter
 
P

Peter Hansen

Peter said:
Does the output look different if you run these programs in your
environment?
Click to expand...

It may take me a moment to get the precise same configuration
on my machine, because I don't have it set up to execute
..py scripts automatically at the command line just by typing
their names. I have to manually invoke "python" and pass it
the name of the script.

Therefore, before I do this (I need to learn how, first),
can you confirm that you get identical results if in each
place where you typed "test1.py" you instead typed the
equivalent with, for example, "python test1.py"?

It's possible Windows' technique for directly invoking
scripts your way screws up the sys.path stuff...

(I _am_ using "stock" Python 2.3.3 on XP, by the way. I
doubt ActiveState's behaves any different in this respect.)

-Peter
 
P

Peter Schwalm

Peter Hansen said:
Peter Schwalm wrote:

... can you confirm that you get identical results if in each
place where you typed "test1.py" you instead typed the
equivalent with, for example, "python test1.py"?
Click to expand...

I have run the programs with "python <scriptname>.py" now. For my
purposes the results look identical - the source directory is still
not accessible during the processing of sitecustomize.py.

Here are the outputs of the explicit invocations:

Invocation from source directory:


E:\temp>python test1.py
---> C:\Python23\lib\sitecustomize.pyc ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv=
sys.argv does still not exist!
sys.modules["__main__"]=
sys.modules["__main__"].__file__ does still not exist
sys.path=
00: C:\WINNT\System32\python23.zip
01: C:\Python23\Lib\site-packages\Pythonwin
02: C:\Python23\Lib\site-packages\win32
03: C:\Python23\Lib\site-packages\win32\lib
04: C:\Python23\Lib\site-packages
05: C:\Python23\DLLs
06: C:\Python23\lib
07: C:\Python23\lib\plat-win
08: C:\Python23\lib\lib-tk
09: C:\Python23
10: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- C:\Python23\lib\sitecustomize.pyc ...
---> test1.py ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv= ['test1.py']
sys.modules["__main__"]= test1.py
sys.path=
00: E:\temp
01: C:\WINNT\System32\python23.zip
02: C:\Python23\Lib\site-packages\Pythonwin
03: C:\Python23\Lib\site-packages\win32
04: C:\Python23\Lib\site-packages\win32\lib
05: C:\Python23\Lib\site-packages
06: C:\Python23\DLLs
07: C:\Python23\lib
08: C:\Python23\lib\plat-win
09: C:\Python23\lib\lib-tk
10: C:\Python23
11: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- test1.py ...

E:\temp>


Invocation from different directory:

E:\temp\sub1>python ..\test1.py
---> C:\Python23\lib\sitecustomize.pyc ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv=
sys.argv does still not exist!
sys.modules["__main__"]=
sys.modules["__main__"].__file__ does still not exist
sys.path=
00: C:\WINNT\System32\python23.zip
01: C:\Python23\Lib\site-packages\Pythonwin
02: C:\Python23\Lib\site-packages\win32
03: C:\Python23\Lib\site-packages\win32\lib
04: C:\Python23\Lib\site-packages
05: C:\Python23\DLLs
06: C:\Python23\lib
07: C:\Python23\lib\plat-win
08: C:\Python23\lib\lib-tk
09: C:\Python23
10: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- C:\Python23\lib\sitecustomize.pyc ...
---> ..\test1.py ...
----------------------------------------------------------------
python-version= 2.3.2 (#49, Oct 24 2003, 13:37:57) [MSC v.1200 32 bit
(Intel)]
argv= ['..\\test1.py']
sys.modules["__main__"]= ..\test1.py
sys.path=
00: E:\temp
01: C:\WINNT\System32\python23.zip
02: C:\Python23\Lib\site-packages\Pythonwin
03: C:\Python23\Lib\site-packages\win32
04: C:\Python23\Lib\site-packages\win32\lib
05: C:\Python23\Lib\site-packages
06: C:\Python23\DLLs
07: C:\Python23\lib
08: C:\Python23\lib\plat-win
09: C:\Python23\lib\lib-tk
10: C:\Python23
11: C:\Python23\lib\site-packages\PIL
----------------------------------------------------------------
<--- ..\test1.py ...

E:\temp\sub1>

By the way, *if I remember that correctly*: to be able to invoke
python scripts simply by name ("test1" instead of "python test1.py")
you have to do
two things:

1) set the environment variable PATHEXT such that it contains
".py"(see sample output from set command)

E:\temp>set pathext
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PY;.py;.pyw;
.pyc;.pys;.pyo
E:\temp>

2) associate the file extension ".py" with c:\python23\python.exe
right click a .py-file in explorer, select "open with" (my translation
of the german "oeffnen mit") select python.exe as application and
check "open always with".

Thank you in advance
Peter Schwalm
 
P

Peter Hansen

Peter said:
I have run the programs with "python <scriptname>.py" now. For my
purposes the results look identical - the source directory is still
not accessible during the processing of sitecustomize.py.
Click to expand...

Thanks for your persistence, Peter. I've confirmed what you have
observed here: during site.py processing (including near the end
when "import sitecustomize" is tried) the directory containing the
main script is not included in sys.path. Immediately after site.py
is imported, sys.path does contain the proper script at the start.

My guess is that recent changes to the import mechanism have broken
this behaviour.

Note that this change would also break sitecustomize processing in
some installations, since it should (I believe) be possible to
have a sitecustomize.py in the directory where the main script
resides, but in Python 2.3.3 it is not imported at all.

If anyone who knows about such things more than we do can confirm
that this should in fact be considered a bug, I will add it to
SourceForge.

-Peter
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Testing python command line apps -- Running from within the projectsw/o installing 0
IDLE not setting current directory in its path 6
module import search path strangeness 8
Finding the local directory of a file in the python path 1
How to add the current dir to sys.path when calling a python file? 3
Python default search paths 2
Is there any advantage to using a main() in python scripts? 8
Python horks on WinXP path names 2

Members online

No members online now.
Total: 42 (members: 2, guests: 40)
Robots: 413

Forum statistics

Threads
473,769
Messages
2,569,577
Members
45,054
Latest member
LucyCarper

Latest Threads

Top