Virtualenv loses context

Discussion in 'Python' started by rhythmicdevil@gmail.com, Dec 19, 2012.

  1. Guest

    I am somewhat new to Python and virtualenv. I have setup a virtualenv with no site packages under the assumption that all 3rd party packages will be installed into the site-packages directory for the virtualenv. Another assumption I have about virtualenv is that it's completely self contained.

    When I first setup the virtualenv, install my packages and then install my application both of those assumptions seem to be correct. However, at some point I start getting ImportErrors. The be clear, I tested the application and it was all working correctly, the ImportErrors seem to show up at some random point that I cannot reproduce.

    I have a requirements file that I made with 'pip freeze'. I try to use thisto reinstall the packages but I get errors saying that I cannot install tothe location. The location is the global site-packages directory on my system. I have the virtualenv activated while installing so I dont know why this is happening.

    Due to something about Centos6 I cant install M2Crypto using pip and have to do it manually. This gives me the same types of errors:

    (botnet_etl)[swright@localhost M2Crypto]$ ./fedora_setup.sh install
    running install
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

    [Errno 13] Permission denied: '/usr/lib64/python2.6/site-packages/test-easy-inst
    all-29133.write-test'

    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

    /usr/lib64/python2.6/site-packages/

    Perhaps your account does not have write access to this directory? If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account. If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

    http://peak.telecommunity.com/EasyInstall.html

    Please make the appropriate changes for your system and try again.




    The only fix I have managed for this problem so far is to reinstall the entire virtualenv from scratch. This works fine for a while and I have to repeat the process. I would think that I was doing something wrong if the ImportErrors showed up every time I attempt to run the app but the fact that they show up later leads me to believe something funky is going on.




    Os = Centos6
    Python = 2.6.6


    Thanks
    Steve
     
    , Dec 19, 2012
    #1
    1. Advertising

  2. Guest

    I wrote this little test script to prove my virtualenv is reading from global site packages:

    Script Content
    from distutils.sysconfig import get_python_lib
    print (get_python_lib())

    Script Output
    (botnet_etl)[swright@localhost app]$ python test.py
    /usr/lib/python2.6/site-packages
    (botnet_etl)[swright@localhost app]$


    Please note that (botnet_etl) is the name of the virtualenv
     
    , Dec 19, 2012
    #2
    1. Advertising

  3. Guest

    This may have something to do with it. I create the virtualenv in:
    /home/swright/workspace/botnet_etl/

    After I install my app the directory structure looks like this:
    (botnet_etl)[swright@localhost botnet_etl]$ ll
    total 32
    drwxrwxr-x 4 swright swright 4096 Dec 19 09:21 app
    drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 bin
    drwxrwxr-x 9 swright swright 4096 Dec 18 10:24 build
    drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 include
    drwxrwxr-x 4 swright swright 4096 Dec 18 10:34 lib
    -rw-rw-r-- 1 swright swright 1973 Dec 17 15:35 README
    -rw-rw-r-- 1 swright swright 109 Dec 17 15:35 stable-req.txt
    drwxrwxr-x 3 swright swright 4096 Dec 17 15:35 utils


    When I run the following command the path is not showing up which is probably what is causing the problem. I have to reinstall from scratch to see if this is different at install time.

    But, am I supposed to create virtualenv in specific location in the file system? That seems a little odd.
     
    , Dec 19, 2012
    #3
  4. Guest

    This may have something to do with it. I create the virtualenv in:
    /home/swright/workspace/botnet_etl/

    After I install my app the directory structure looks like this:
    (botnet_etl)[swright@localhost botnet_etl]$ ll
    total 32
    drwxrwxr-x 4 swright swright 4096 Dec 19 09:21 app
    drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 bin
    drwxrwxr-x 9 swright swright 4096 Dec 18 10:24 build
    drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 include
    drwxrwxr-x 4 swright swright 4096 Dec 18 10:34 lib
    -rw-rw-r-- 1 swright swright 1973 Dec 17 15:35 README
    -rw-rw-r-- 1 swright swright 109 Dec 17 15:35 stable-req.txt
    drwxrwxr-x 3 swright swright 4096 Dec 17 15:35 utils


    When I run the following command the path is not showing up which is probably what is causing the problem. Clearly the site packages for my virtualenv are not listed. I have to reinstall from scratch to see if this is different at install time.

    (botnet_etl)[swright@localhost app]$ python -m site
    sys.path = [
    '/home/swright/workspace/botnet_etl/app',
    '/usr/lib/python2.6/site-packages/pymongo-2.3-py2.6-linux-x86_64.egg',
    '/usr/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg',
    '/usr/lib64/python26.zip',
    '/usr/lib64/python2.6',
    '/usr/lib64/python2.6/plat-linux2',
    '/usr/lib64/python2.6/lib-tk',
    '/usr/lib64/python2.6/lib-old',
    '/usr/lib64/python2.6/lib-dynload',
    '/usr/lib64/python2.6/site-packages',
    '/usr/lib64/python2.6/site-packages/gst-0.10',
    '/usr/lib64/python2.6/site-packages/gtk-2.0',
    '/usr/lib64/python2.6/site-packages/webkit-1.0',
    '/usr/lib/python2.6/site-packages',
    '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info',
    ]
    USER_BASE: '/home/swright/.local' (exists)
    USER_SITE: '/home/swright/.local/lib/python2.6/site-packages' (doesn't exist)
    ENABLE_USER_SITE: True





    But, am I supposed to create virtualenv in specific location in the file system? That seems a little odd
     
    , Dec 19, 2012
    #4
  5. Guest

    Just installed a brand new virtualenv along with two packages. Ran this and I got nothing:

    (venvtest)[swright@localhost venvtest]$ python -m site
    (venvtest)[swright@localhost venvtest]$

    I expected to have at least one path in sys.path
     
    , Dec 19, 2012
    #5
  6. Guest

    Just installed a brand new virtualenv along with two packages. Ran this and I got nothing:

    (venvtest)[swright@localhost venvtest]$ python -m site
    (venvtest)[swright@localhost venvtest]$

    I expected to have at least one path in sys.path



    [swright@localhost workspace]$ virtualenv --no-site-packages venvtest
    New python executable in venvtest/bin/python
    Installing setuptools............done.
    Installing pip...............done.


    [swright@localhost venvtest]$ source bin/activate
    (venvtest)[swright@localhost venvtest]$ python -m site
    (venvtest)[swright@localhost venvtest]$


    (venvtest)[swright@localhost venvtest]$ pip install -r stable-req.txt
    Downloading/unpacking pycrypto==2.6 (from -r stable-req.txt (line 1))
    Downloading pycrypto-2.6.tar.gz (443kB): 443kB downloaded
    Running setup.py egg_info for package pycrypto
    Downloading/unpacking pymongo==2.4 (from -r stable-req.txt (line 2))
    Downloading pymongo-2.4.tar.gz (273kB): 273kB downloaded
    Running setup.py egg_info for package pymongo
    Downloading/unpacking yolk==0.4.3 (from -r stable-req.txt (line 3))
    Downloading yolk-0.4.3.tar.gz (86kB): 86kB downloaded
    Running setup.py egg_info for package yolk
    warning: no files found matching '*.txt' under directory 'tests'
    warning: no files found matching '*.conf' under directory 'docs'
    warning: no files found matching '*.css_t' under directory 'docs'
    warning: no files found matching 'indexsidebar.html' under directory 'docs'
    warning: no files found matching 'tests/test_cli.py'
    Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python
    2.6/site-packages/setuptools-0.6c11-py2.6.egg (from yolk==0.4.3->-r stable-req.txt (
    line 3))
    .... lots more text...




    (venvtest)[swright@localhost venvtest]$ pwd
    /home/swright/workspace/venvtest
    (venvtest)[swright@localhost venvtest]$ ll lib/python2.6/site-packages/
    total 372
    drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 bson
    drwxrwxr-x 10 swright swright 4096 Dec 19 09:34 Crypto
    -rw-rw-r-- 1 swright swright 237 Dec 19 09:32 easy-install.pth
    drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 gridfs
    drwxrwxr-x 4 swright swright 4096 Dec 19 09:32 pip-1.2.1-py2.6.egg
    drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pycrypto-2.6-py2.6.egg-info
    drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pymongo
    drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pymongo-2.4-py2.6.egg-info
    -rw-r--r-- 1 swright swright 333447 Dec 5 09:13 setuptools-0.6c11-py2.6.egg
    -rw-rw-r-- 1 swright swright 30 Dec 19 09:32 setuptools.pth
    drwxrwxr-x 3 swright swright 4096 Dec 19 09:34 yolk
    drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 yolk-0.4.3-py2.6.egg-info
    (venvtest)[swright@localhost venvtest]$
     
    , Dec 19, 2012
    #6
  7. Guest

    So I reinstalled the virtualenv for my project from scratch. Application runs as expected. Here are the notes that I took while installing. Interestingly the command 'python -m site' produces no output now. Notice that beforeI reinstalled the virtualenv I got a bunch of paths from that command. I am going to reboot my machine to see if that does something. (must be used to windows)


    [swright@localhost workspace]$ virtualenv botnet_etl
    New python executable in botnet_etl/bin/python
    Installing setuptools............done.
    Installing pip...............done.


    [swright@localhost workspace]$ cd botnet_etl/
    [swright@localhost botnet_etl]$ source bin/activate
    (botnet_etl)[swright@localhost botnet_etl]$


    (botnet_etl)[swright@localhost botnet_etl]$ python -m site
    (botnet_etl)[swright@localhost botnet_etl]$

    (botnet_etl)[swright@localhost botnet_etl]$ pip install -r stable-req.txt
    --- lots of text follows this for compiling and installing

    Install M2Crypto manually:
    Installed /home/swright/workspace/botnet_etl/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg
    Processing dependencies for M2Crypto==0.21.1
    Finished processing dependencies for M2Crypto==0.21.1

    Install PySVN manually:
    (botnet_etl)[swright@localhost M2Crypto]$ cp -r ~/.virtualenvs/myvirtualenv/lib/python2.6/site-packages/pysvn ~/workspace/botnet_etl/lib/python2.6/site-packages/.



    (botnet_etl)[swright@localhost botnet_etl]$ python -m site
    (botnet_etl)[swright@localhost botnet_etl]$
     
    , Dec 19, 2012
    #7
  8. Hans Mulder Guest

    On 19/12/12 15:38:01, wrote:
    > Just installed a brand new virtualenv along with two packages. Ran this and I got nothing:
    >
    > (venvtest)[swright@localhost venvtest]$ python -m site
    > (venvtest)[swright@localhost venvtest]$
    >
    > I expected to have at least one path in sys.path


    For some reason, that doesn't work with python2.7.
    I guess that's a bug in 2.7.

    One variant that does work, is:

    python -c 'import site; site._script();'

    Another would be:

    python /usr/lib/python2.7/site.py

    If you think you may have accidentally deactivated your
    virtualenv, you can do:

    python -c 'import sys; print sys.executable;'

    Your prompt suggests that "venvtest" is active.
    However, it's possible to deactivate a virtualenv without
    resetting the prompt; you may have unintentionally done that.
    If that's the problem, you can solve it by sourcing the
    "activate" script again.


    Hope this helps,

    -- HansM
     
    Hans Mulder, Dec 19, 2012
    #8
  9. Guest

    Thanks for the information Hans, I will double check that stuff. I am positive however that the environment was active. In addition that fact that I get different responses from python -m site from the botnet_etl virtual environment before I blew it away and after I rebuilt it really concerns me.
     
    , Dec 19, 2012
    #9
  10. Guest

    Brought my laptop out of hibernation to do some work this morning. I attempted to run one of my ETLs and got the following error. I made no changes since it was running yesterday.



    [swright@localhost app]$ python etl_botnet_meta.py --mode dev -f
    Traceback (most recent call last):
    File "etl_botnet_meta.py", line 9, in <module>
    from make_zip import EtlMakeZip
    File "/home/swright/workspace/botnet_etl/app/make_zip.py", line 1, in <module>
    import M2Crypto, os, time, datetime, json, hashlib, base64, zipfile
    ImportError: No module named M2Crypto
    [swright@localhost app]$ python -m site
    sys.path = [
    '/home/swright/workspace/botnet_etl/app',
    '/usr/lib/python2.6/site-packages/pymongo-2.3-py2.6-linux-x86_64.egg',
    '/usr/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg',
    '/usr/lib64/python26.zip',
    '/usr/lib64/python2.6',
    '/usr/lib64/python2.6/plat-linux2',
    '/usr/lib64/python2.6/lib-tk',
    '/usr/lib64/python2.6/lib-old',
    '/usr/lib64/python2.6/lib-dynload',
    '/usr/lib64/python2.6/site-packages',
    '/usr/lib64/python2.6/site-packages/gst-0.10',
    '/usr/lib64/python2.6/site-packages/gtk-2.0',
    '/usr/lib64/python2.6/site-packages/webkit-1.0',
    '/usr/lib/python2.6/site-packages',
    '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info',
    ]
    USER_BASE: '/home/swright/.local' (exists)
    USER_SITE: '/home/swright/.local/lib/python2.6/site-packages' (doesn't exist)
    ENABLE_USER_SITE: True
    [swright@localhost app]$
     
    , Dec 20, 2012
    #10
  11. Ian Kelly Guest

    On Thu, Dec 20, 2012 at 5:50 AM, <> wrote:
    > Brought my laptop out of hibernation to do some work this morning. I attempted to run one of my ETLs and got the following error. I made no changes since it was running yesterday.
    >
    >
    >
    > [swright@localhost app]$ python etl_botnet_meta.py --mode dev -f


    Are you sure you activated the virtual env? I see no indication of it
    in your prompt.
     
    Ian Kelly, Dec 20, 2012
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. VS_NET_DEV
    Replies:
    2
    Views:
    3,802
    jenny
    May 25, 2004
  2. =?Utf-8?B?U3VuU21pbGU=?=
    Replies:
    0
    Views:
    707
    =?Utf-8?B?U3VuU21pbGU=?=
    Jan 10, 2006
  3. Replies:
    0
    Views:
    282
  4. Gelonida N
    Replies:
    4
    Views:
    981
    Gelonida N
    Aug 11, 2011
  5. -Lost
    Replies:
    7
    Views:
    183
    -Lost
    Mar 24, 2007
Loading...

Share This Page