Virtualenv loses context

[rhythmicdevil]

I am somewhat new to Python and virtualenv. I have setup a virtualenv with no site packages under the assumption that all 3rd party packages will be installed into the site-packages directory for the virtualenv. Another assumption I have about virtualenv is that it's completely self contained.

When I first setup the virtualenv, install my packages and then install my application both of those assumptions seem to be correct. However, at some point I start getting ImportErrors. The be clear, I tested the application and it was all working correctly, the ImportErrors seem to show up at some random point that I cannot reproduce.

I have a requirements file that I made with 'pip freeze'. I try to use thisto reinstall the packages but I get errors saying that I cannot install tothe location. The location is the global site-packages directory on my system. I have the virtualenv activated while installing so I dont know why this is happening.

Due to something about Centos6 I cant install M2Crypto using pip and have to do it manually. This gives me the same types of errors:

(botnet_etl)[swright@localhost M2Crypto]$ ./fedora_setup.sh install
running install
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 13] Permission denied: '/usr/lib64/python2.6/site-packages/test-easy-inst
all-29133.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

/usr/lib64/python2.6/site-packages/

Perhaps your account does not have write access to this directory? If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account. If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

http://peak.telecommunity.com/EasyInstall.html

Please make the appropriate changes for your system and try again.




The only fix I have managed for this problem so far is to reinstall the entire virtualenv from scratch. This works fine for a while and I have to repeat the process. I would think that I was doing something wrong if the ImportErrors showed up every time I attempt to run the app but the fact that they show up later leads me to believe something funky is going on.




Os = Centos6
Python = 2.6.6


Thanks
Steve

 

[rhythmicdevil]

I wrote this little test script to prove my virtualenv is reading from global site packages:

Script Content
from distutils.sysconfig import get_python_lib
print (get_python_lib())

Script Output
(botnet_etl)[swright@localhost app]$ python test.py
/usr/lib/python2.6/site-packages
(botnet_etl)[swright@localhost app]$


Please note that (botnet_etl) is the name of the virtualenv

 

[rhythmicdevil]

This may have something to do with it. I create the virtualenv in:
/home/swright/workspace/botnet_etl/

After I install my app the directory structure looks like this:
(botnet_etl)[swright@localhost botnet_etl]$ ll
total 32
drwxrwxr-x 4 swright swright 4096 Dec 19 09:21 app
drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 bin
drwxrwxr-x 9 swright swright 4096 Dec 18 10:24 build
drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 include
drwxrwxr-x 4 swright swright 4096 Dec 18 10:34 lib
-rw-rw-r-- 1 swright swright 1973 Dec 17 15:35 README
-rw-rw-r-- 1 swright swright 109 Dec 17 15:35 stable-req.txt
drwxrwxr-x 3 swright swright 4096 Dec 17 15:35 utils


When I run the following command the path is not showing up which is probably what is causing the problem. I have to reinstall from scratch to see if this is different at install time.

But, am I supposed to create virtualenv in specific location in the file system? That seems a little odd.

 

[rhythmicdevil]

This may have something to do with it. I create the virtualenv in:
/home/swright/workspace/botnet_etl/

After I install my app the directory structure looks like this:
(botnet_etl)[swright@localhost botnet_etl]$ ll
total 32
drwxrwxr-x 4 swright swright 4096 Dec 19 09:21 app
drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 bin
drwxrwxr-x 9 swright swright 4096 Dec 18 10:24 build
drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 include
drwxrwxr-x 4 swright swright 4096 Dec 18 10:34 lib
-rw-rw-r-- 1 swright swright 1973 Dec 17 15:35 README
-rw-rw-r-- 1 swright swright 109 Dec 17 15:35 stable-req.txt
drwxrwxr-x 3 swright swright 4096 Dec 17 15:35 utils


When I run the following command the path is not showing up which is probably what is causing the problem. Clearly the site packages for my virtualenv are not listed. I have to reinstall from scratch to see if this is different at install time.

(botnet_etl)[swright@localhost app]$ python -m site
sys.path = [
'/home/swright/workspace/botnet_etl/app',
'/usr/lib/python2.6/site-packages/pymongo-2.3-py2.6-linux-x86_64.egg',
'/usr/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg',
'/usr/lib64/python26.zip',
'/usr/lib64/python2.6',
'/usr/lib64/python2.6/plat-linux2',
'/usr/lib64/python2.6/lib-tk',
'/usr/lib64/python2.6/lib-old',
'/usr/lib64/python2.6/lib-dynload',
'/usr/lib64/python2.6/site-packages',
'/usr/lib64/python2.6/site-packages/gst-0.10',
'/usr/lib64/python2.6/site-packages/gtk-2.0',
'/usr/lib64/python2.6/site-packages/webkit-1.0',
'/usr/lib/python2.6/site-packages',
'/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info',
]
USER_BASE: '/home/swright/.local' (exists)
USER_SITE: '/home/swright/.local/lib/python2.6/site-packages' (doesn't exist)
ENABLE_USER_SITE: True





But, am I supposed to create virtualenv in specific location in the file system? That seems a little odd

 

[rhythmicdevil]

Just installed a brand new virtualenv along with two packages. Ran this and I got nothing:

(venvtest)[swright@localhost venvtest]$ python -m site
(venvtest)[swright@localhost venvtest]$

I expected to have at least one path in sys.path

 

[rhythmicdevil]

Just installed a brand new virtualenv along with two packages. Ran this and I got nothing:

(venvtest)[swright@localhost venvtest]$ python -m site
(venvtest)[swright@localhost venvtest]$

I expected to have at least one path in sys.path



[swright@localhost workspace]$ virtualenv --no-site-packages venvtest
New python executable in venvtest/bin/python
Installing setuptools............done.
Installing pip...............done.


[swright@localhost venvtest]$ source bin/activate
(venvtest)[swright@localhost venvtest]$ python -m site
(venvtest)[swright@localhost venvtest]$


(venvtest)[swright@localhost venvtest]$ pip install -r stable-req.txt
Downloading/unpacking pycrypto==2.6 (from -r stable-req.txt (line 1))
Downloading pycrypto-2.6.tar.gz (443kB): 443kB downloaded
Running setup.py egg_info for package pycrypto
Downloading/unpacking pymongo==2.4 (from -r stable-req.txt (line 2))
Downloading pymongo-2.4.tar.gz (273kB): 273kB downloaded
Running setup.py egg_info for package pymongo
Downloading/unpacking yolk==0.4.3 (from -r stable-req.txt (line 3))
Downloading yolk-0.4.3.tar.gz (86kB): 86kB downloaded
Running setup.py egg_info for package yolk
warning: no files found matching '*.txt' under directory 'tests'
warning: no files found matching '*.conf' under directory 'docs'
warning: no files found matching '*.css_t' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
warning: no files found matching 'tests/test_cli.py'
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python
2.6/site-packages/setuptools-0.6c11-py2.6.egg (from yolk==0.4.3->-r stable-req.txt (
line 3))
.... lots more text...




(venvtest)[swright@localhost venvtest]$ pwd
/home/swright/workspace/venvtest
(venvtest)[swright@localhost venvtest]$ ll lib/python2.6/site-packages/
total 372
drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 bson
drwxrwxr-x 10 swright swright 4096 Dec 19 09:34 Crypto
-rw-rw-r-- 1 swright swright 237 Dec 19 09:32 easy-install.pth
drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 gridfs
drwxrwxr-x 4 swright swright 4096 Dec 19 09:32 pip-1.2.1-py2.6.egg
drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pycrypto-2.6-py2.6.egg-info
drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pymongo
drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pymongo-2.4-py2.6.egg-info
-rw-r--r-- 1 swright swright 333447 Dec 5 09:13 setuptools-0.6c11-py2.6.egg
-rw-rw-r-- 1 swright swright 30 Dec 19 09:32 setuptools.pth
drwxrwxr-x 3 swright swright 4096 Dec 19 09:34 yolk
drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 yolk-0.4.3-py2.6.egg-info
(venvtest)[swright@localhost venvtest]$

 

[rhythmicdevil]

So I reinstalled the virtualenv for my project from scratch. Application runs as expected. Here are the notes that I took while installing. Interestingly the command 'python -m site' produces no output now. Notice that beforeI reinstalled the virtualenv I got a bunch of paths from that command. I am going to reboot my machine to see if that does something. (must be used to windows)


[swright@localhost workspace]$ virtualenv botnet_etl
New python executable in botnet_etl/bin/python
Installing setuptools............done.
Installing pip...............done.


[swright@localhost workspace]$ cd botnet_etl/
[swright@localhost botnet_etl]$ source bin/activate
(botnet_etl)[swright@localhost botnet_etl]$


(botnet_etl)[swright@localhost botnet_etl]$ python -m site
(botnet_etl)[swright@localhost botnet_etl]$

(botnet_etl)[swright@localhost botnet_etl]$ pip install -r stable-req.txt
--- lots of text follows this for compiling and installing

Install M2Crypto manually:
Installed /home/swright/workspace/botnet_etl/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg
Processing dependencies for M2Crypto==0.21.1
Finished processing dependencies for M2Crypto==0.21.1

Install PySVN manually:
(botnet_etl)[swright@localhost M2Crypto]$ cp -r ~/.virtualenvs/myvirtualenv/lib/python2.6/site-packages/pysvn ~/workspace/botnet_etl/lib/python2.6/site-packages/.



(botnet_etl)[swright@localhost botnet_etl]$ python -m site
(botnet_etl)[swright@localhost botnet_etl]$

 

[Hans Mulder]

Just installed a brand new virtualenv along with two packages. Ran this and I got nothing:

(venvtest)[swright@localhost venvtest]$ python -m site
(venvtest)[swright@localhost venvtest]$

I expected to have at least one path in sys.path

For some reason, that doesn't work with python2.7.
I guess that's a bug in 2.7.

One variant that does work, is:

python -c 'import site; site._script();'

Another would be:

python /usr/lib/python2.7/site.py

If you think you may have accidentally deactivated your
virtualenv, you can do:

python -c 'import sys; print sys.executable;'

Your prompt suggests that "venvtest" is active.
However, it's possible to deactivate a virtualenv without
resetting the prompt; you may have unintentionally done that.
If that's the problem, you can solve it by sourcing the
"activate" script again.


Hope this helps,

-- HansM

 

[rhythmicdevil]

Thanks for the information Hans, I will double check that stuff. I am positive however that the environment was active. In addition that fact that I get different responses from python -m site from the botnet_etl virtual environment before I blew it away and after I rebuilt it really concerns me.

 

[rhythmicdevil]

Brought my laptop out of hibernation to do some work this morning. I attempted to run one of my ETLs and got the following error. I made no changes since it was running yesterday.



[swright@localhost app]$ python etl_botnet_meta.py --mode dev -f
Traceback (most recent call last):
File "etl_botnet_meta.py", line 9, in <module>
from make_zip import EtlMakeZip
File "/home/swright/workspace/botnet_etl/app/make_zip.py", line 1, in <module>
import M2Crypto, os, time, datetime, json, hashlib, base64, zipfile
ImportError: No module named M2Crypto
[swright@localhost app]$ python -m site
sys.path = [
'/home/swright/workspace/botnet_etl/app',
'/usr/lib/python2.6/site-packages/pymongo-2.3-py2.6-linux-x86_64.egg',
'/usr/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg',
'/usr/lib64/python26.zip',
'/usr/lib64/python2.6',
'/usr/lib64/python2.6/plat-linux2',
'/usr/lib64/python2.6/lib-tk',
'/usr/lib64/python2.6/lib-old',
'/usr/lib64/python2.6/lib-dynload',
'/usr/lib64/python2.6/site-packages',
'/usr/lib64/python2.6/site-packages/gst-0.10',
'/usr/lib64/python2.6/site-packages/gtk-2.0',
'/usr/lib64/python2.6/site-packages/webkit-1.0',
'/usr/lib/python2.6/site-packages',
'/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info',
]
USER_BASE: '/home/swright/.local' (exists)
USER_SITE: '/home/swright/.local/lib/python2.6/site-packages' (doesn't exist)
ENABLE_USER_SITE: True
[swright@localhost app]$

 

[Ian Kelly]

Brought my laptop out of hibernation to do some work this morning. I attempted to run one of my ETLs and got the following error. I made no changes since it was running yesterday.



[swright@localhost app]$ python etl_botnet_meta.py --mode dev -f

Are you sure you activated the virtual env? I see no indication of it
in your prompt.