How to insert my own module in front of site eggs?

[Roy Smith]

I'm trying to use a custom version of mongoengine. I cloned the git
repo and put the directory on my PYTHONPATH, but python is still
importing the system's installed version. Looking at sys.path, it's
obvious why:

$ echo $PYTHONPATH
/home/roy/songza:/home/roy/lib/mongoengine
['',
'/usr/local/lib/python2.6/dist-packages/selenium-2.0a5-py2.6.egg',
'/usr/local/lib/python2.6/dist-packages/unittest2-0.5.1-py2.6.egg',

'/usr/local/lib/python2.6/dist-packages/pymongo-1.9-py2.6-linux-x86_64.eg
g',
'/usr/local/lib/python2.6/dist-packages/virtualenv-1.5.2-py2.6.egg',
'/usr/local/lib/python2.6/dist-packages/mongoengine-0.5.2-py2.6.egg',
'/home/roy/songza',
'/home/roy/lib/mongoengine',
'/usr/lib/python2.6',
'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/usr/lib/python2.6/dist-packages',
'/usr/lib/python2.6/dist-packages/PIL',
'/usr/lib/python2.6/dist-packages/gst-0.10',
'/usr/lib/pymodules/python2.6',
'/usr/lib/python2.6/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.6/gtk-2.0',
'/usr/local/lib/python2.6/dist-packages']

The system eggs come before my path. I found
http://mail.python.org/pipermail/distutils-sig/2006-July/006520.html in
the archives; it explains that eggs come before PYTHONPATH, but doesn't
explain how to get around this problem. I emphatically agree with
Michael Bayer who said:

I cant think of a possible scenario where a path would explicitly
exist in PYTHONPATH, non-egg or egg, where the user would still like the
system-wide installation to take precedence

So, is there any way to get my local copy of mongoengine loaded instead
of the system egg? I could probably import sys, and do an egg-ectomy on
sys.path before importing mongoengine, but that's too gross to
contemplate.

 

[Roy Smith]

So, is there any way to get my local copy of mongoengine loaded instead
of the system egg? I could probably import sys, and do an egg-ectomy on
sys.path before importing mongoengine, but that's too gross to
contemplate.

The only way I found is to edit the easy_install.pth file and comment
the two lines starting with "import sys". You'll have to do that every
time you install/upgrade an egg via easy_install (and maybe setuptools).
In your case the right file should be
/usr/local/lib/python2.6/dist-packages/easy_install.pth

BTW: You could try pip (http://www.pip-installer.org/) instead of
easy_install, it doesn't mess with sys.path.[/QUOTE]

But, you're talking about installers. I'm talking about if I've already
got something installed, how do I force one particular python process to
pull in a local copy of a module in preference to the installed one?

In some cases, I own the machine and can make changes to /usr/local/lib
if I want to. But what about on a shared machine? I don't want to (or
perhaps can't) play with what's in /usr/local/lib just to make my stuff
load first.

 

[Paul Rudin]

But, you're talking about installers. I'm talking about if I've already
got something installed, how do I force one particular python process to
pull in a local copy of a module in preference to the installed one?

In some cases, I own the machine and can make changes to /usr/local/lib
if I want to. But what about on a shared machine? I don't want to (or
perhaps can't) play with what's in /usr/local/lib just to make my stuff
load first.

Maybe I'm missing something - but if I want to do this I just mess about
with sys.path at the top of my python script/program. Always seems to
work... is there a situation in which it doesn't?

 

[Roy Smith]

Maybe I'm missing something - but if I want to do this I just mess about
with sys.path at the top of my python script/program. Always seems to
work... is there a situation in which it doesn't?

What if the first import of a module is happening inside some code you
don't have access to?

It just seems mind-boggling to me that PYTHONPATH doesn't preempt
everything else.

 

[Roy Smith]

Have you considered running your code in a virtualenv?
http://pypi.python.org/pypi/virtualenv

I do that for some projects. In fact, I suspect I will do that for all
furture projects that I start from scratch. For this particular one,
there's a lot of stuff already installed in the system that I need.

 

[alex23]

What if the first import of a module is happening inside some code you
don't have access to?

No import will happen until you import something. As long as you
change sys.path before you do, all subsequent imports will use that
path.

 

[Paul Rudin]

What if the first import of a module is happening inside some code you
don't have access to?

If you change sys.path first - before you do any imports - then any
other imports will surely come from the first thing on sys.path (unless
something else you import also changes sys.path)?

 

[Hans Mulder]

No import will happen until you import something.

That would be the case if you use the '-S' command line option.

Otherwise, site.py is imported before you get a chance to alter
sys.path, and by default site.py imports other modules.

-- HansM

 

[spartan.the]

It just seems mind-boggling to me that PYTHONPATH doesn't preempt
everything else.

I was surprised too, but this issue is widespread. Best explained
here:
https://bugs.launchpad.net/tahoe-lafs/+bug/821000/comments/0
Quote:

"
The crux of the problem is that the Python documentation says:
"The PYTHONPATH variable can be set to a list of paths that will be
added to the beginning of sys.path."
http://docs.python.org/install/index.html#modifying-python-s-search-path
This is true with standard distutils, false with setuptools, false
with distribute...
"

Phillip J. Eby (setuptools guy) was religious about "easy installed"
packages over anything else to comment on this issue 3 years ago:

"
This behavior change is unacceptable, as it makes it impossible to
ensure that
an easy_install-ed package takes precedence over an existing,
distutils-installed version of the same package in the same PYTHONPATH
directory.

Please note that that is the intended default behavior of
easy_install: if you
wish to override it, you should use --multi-version, and explicitly
select the
egg(s) to be added to sys.path instead.
"
(http://bugs.python.org/setuptools/issue53)