"non central" package management

[Miki Tebeka]

Greetings,

The usual package mangers (easy_install, pip ...) install packages in one central location.

I'm looking for a solution that will allow every project (which run on the same machine) use it's own packages and versions of packages. (Context - we're running several applications on the same machine, and can't test *everything* when we update a package. Also these applications might ship to other machines to run - via hadoop streaming).

I can see two options:
1. Use a different virtual env for every package (and will need activate the right one for every package).
2. Have a "lib" directory in each application with versioned packages (we use something like that with svn:externals for version management).
3. Write my own package manager.

Anyone had the same problem? Any known solutions?

Thanks,

 

[Roy Smith]

Greetings,

The usual package mangers (easy_install, pip ...) install packages in one
central location.

I'm looking for a solution that will allow every project (which run on the
same machine) use it's own packages and versions of packages. (Context -
we're running several applications on the same machine, and can't test
*everything* when we update a package. Also these applications might ship to
other machines to run - via hadoop streaming).

You want virtualenv. Use pip to install packages, and they'll go into
your (per-project) virtualenv.

2. Have a "lib" directory in each application with versioned packages (we use
something like that with svn:externals for version management).

What we do is run "pip freeze > requirements.txt" and check that into
version control. When we deploy, we create a new virtualenv, then do
"pip install -r requirements.txt".

For hadoop jobs that we run on EMR under mrjob, we have the virtualenv
setup in the bootstrap script.

 

[Miki Tebeka]

What we do is run "pip freeze > requirements.txt" and check that into
version control.

That's the idea I was toying with, thanks for confirming.

When we deploy, we create a new virtualenv, then do
"pip install -r requirements.txt".

1. Do you do that for every run?
2. Our ops team don't like to depend on pypi, they want internal repo - but I guess I can do that with "pip install -f".

Thanks,

 

[Roy Smith]

When we deploy, we create a new virtualenv, then do
"pip install -r requirements.txt".

1. Do you do that for every run?[/QUOTE]

Well, sort of.

We are currently using a single virtualenv per deployment host. Each
time we deploy new code, we checkout all the code into a fresh
directory, but each of those shares a common virtualenv. As part of the
deploy process, we do indeed execute "pip install -r requirements.txt",
which picks up any new required packages.

Unfortunately, that process doesn't give us a good way to back out a bad
update. It's easy for us to revert to an previous version of our code,
but we don't have a good way to revert the virtualenv to its previous
state. Fortunately, that's been mostly a theoretical issue for us so
far.

In the future, the plan is to build a complete fresh virtualenv for
every deployment. But we're not there yet.

2. Our ops team don't like to depend on pypi, they want internal repo - but I
guess I can do that with "pip install -f".

Listen to your ops team. Right now, we install out of pypi, but that's
slow, and on occasion, fails. This is part of why we still use a shared
virtualenv, and what your ops guys are trying to avoid

Eventually, we'll mirror all the packages we need locally, and pip
install out of that. Once we've got that all working, we'll move to a
new virtualenv per deployment (which sometimes is several times a day).

 

[Miki Tebeka]

In the future, the plan is to build a complete fresh virtualenv for
every deployment. But we're not there yet.

Maybe a repository of virtualenvs, then when deploying you can see if there's one the matches what you need and use it. Otherwise create a new one.