"non central" package management

Discussion in 'Python' started by Miki Tebeka, Nov 28, 2012.

  1. Miki Tebeka

    Miki Tebeka Guest

    Greetings,

    The usual package mangers (easy_install, pip ...) install packages in one central location.

    I'm looking for a solution that will allow every project (which run on the same machine) use it's own packages and versions of packages. (Context - we're running several applications on the same machine, and can't test *everything* when we update a package. Also these applications might ship to other machines to run - via hadoop streaming).

    I can see two options:
    1. Use a different virtual env for every package (and will need activate the right one for every package).
    2. Have a "lib" directory in each application with versioned packages (we use something like that with svn:externals for version management).
    3. Write my own package manager.

    Anyone had the same problem? Any known solutions?

    Thanks,
    --
    Miki
    Miki Tebeka, Nov 28, 2012
    #1
    1. Advertising

  2. Miki Tebeka

    Roy Smith Guest

    In article <>,
    Miki Tebeka <> wrote:

    > Greetings,
    >
    > The usual package mangers (easy_install, pip ...) install packages in one
    > central location.
    >
    > I'm looking for a solution that will allow every project (which run on the
    > same machine) use it's own packages and versions of packages. (Context -
    > we're running several applications on the same machine, and can't test
    > *everything* when we update a package. Also these applications might ship to
    > other machines to run - via hadoop streaming).


    You want virtualenv. Use pip to install packages, and they'll go into
    your (per-project) virtualenv.

    > 2. Have a "lib" directory in each application with versioned packages (we use
    > something like that with svn:externals for version management).


    What we do is run "pip freeze > requirements.txt" and check that into
    version control. When we deploy, we create a new virtualenv, then do
    "pip install -r requirements.txt".

    For hadoop jobs that we run on EMR under mrjob, we have the virtualenv
    setup in the bootstrap script.
    Roy Smith, Nov 28, 2012
    #2
    1. Advertising

  3. Miki Tebeka

    Miki Tebeka Guest

    On Tuesday, November 27, 2012 8:21:48 PM UTC-8, Roy Smith wrote:
    > What we do is run "pip freeze > requirements.txt" and check that into
    > version control.

    That's the idea I was toying with, thanks for confirming.

    > When we deploy, we create a new virtualenv, then do
    > "pip install -r requirements.txt".

    1. Do you do that for every run?
    2. Our ops team don't like to depend on pypi, they want internal repo - but I guess I can do that with "pip install -f".

    Thanks,
    --
    Miki
    Miki Tebeka, Nov 28, 2012
    #3
  4. Miki Tebeka

    Roy Smith Guest

    In article <>,
    Miki Tebeka <> wrote:

    > > When we deploy, we create a new virtualenv, then do
    > > "pip install -r requirements.txt".

    > 1. Do you do that for every run?


    Well, sort of.

    We are currently using a single virtualenv per deployment host. Each
    time we deploy new code, we checkout all the code into a fresh
    directory, but each of those shares a common virtualenv. As part of the
    deploy process, we do indeed execute "pip install -r requirements.txt",
    which picks up any new required packages.

    Unfortunately, that process doesn't give us a good way to back out a bad
    update. It's easy for us to revert to an previous version of our code,
    but we don't have a good way to revert the virtualenv to its previous
    state. Fortunately, that's been mostly a theoretical issue for us so
    far.

    In the future, the plan is to build a complete fresh virtualenv for
    every deployment. But we're not there yet.

    > 2. Our ops team don't like to depend on pypi, they want internal repo - but I
    > guess I can do that with "pip install -f".


    Listen to your ops team. Right now, we install out of pypi, but that's
    slow, and on occasion, fails. This is part of why we still use a shared
    virtualenv, and what your ops guys are trying to avoid :)

    Eventually, we'll mirror all the packages we need locally, and pip
    install out of that. Once we've got that all working, we'll move to a
    new virtualenv per deployment (which sometimes is several times a day).
    Roy Smith, Nov 28, 2012
    #4
  5. Miki Tebeka

    Miki Tebeka Guest

    On Tuesday, November 27, 2012 8:45:56 PM UTC-8, Roy Smith wrote:
    > In the future, the plan is to build a complete fresh virtualenv for
    > every deployment. But we're not there yet.

    Maybe a repository of virtualenvs, then when deploying you can see if there's one the matches what you need and use it. Otherwise create a new one.
    Miki Tebeka, Nov 28, 2012
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Floris van Haaster

    Project management / bug management

    Floris van Haaster, Sep 23, 2005, in forum: ASP .Net
    Replies:
    3
    Views:
    1,224
    Jon Paal
    Sep 23, 2005
  2. pouet
    Replies:
    2
    Views:
    730
    Will Hartung
    Jul 30, 2004
  3. Parvinder
    Replies:
    6
    Views:
    727
    Thomas G. Marshall
    Feb 27, 2005
  4. Christian Schlichtherle

    Looking for Java License Management Package

    Christian Schlichtherle, Jan 22, 2005, in forum: Java
    Replies:
    3
    Views:
    637
    Thomas Weidenfeller
    Jan 24, 2005
  5. Richard Harter

    RFC on a storage management utility package

    Richard Harter, Nov 8, 2006, in forum: C Programming
    Replies:
    14
    Views:
    525
    Mark McIntyre
    Nov 9, 2006
Loading...

Share This Page