How to turn a package into something pip can install

[Roy Smith]

I want to use (https://github.com/timetric/python-metar). Our
deployment process more or less requires that it be installed via pip.
We maintain our own cache of packages and install using:

pip install --no-index --quiet --find-links packages --requirement
requirements.txt

What I can't figure out is what I need to do to go from a clone of the
github repo to a tarball I can drop into our packages directory. Is
there some tutorial somewhere that explains this?

 

[Ryan Gonzalez]

python setup.py sdist

I want to use (https://github.com/timetric/python-metar). Our
deployment process more or less requires that it be installed via pip.
We maintain our own cache of packages and install using:

pip install --no-index --quiet --find-links packages --requirement
requirements.txt

What I can't figure out is what I need to do to go from a clone of the
github repo to a tarball I can drop into our packages directory. Is
there some tutorial somewhere that explains this?

 

[Roy Smith]

python setup.py sdist

OK, I run that and I get a metar-1.4.0.tar.gz under dist. If I move
that tarfile to my packages directory, and run pip, I get:

$ pip install --no-index --quiet --find-links packages metar==1.4.0
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File
"/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.
1-py2.7.egg/pip/index.py", line 245, in _get_queued_page
page = self._get_page(location, req)
File
"/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.
1-py2.7.egg/pip/index.py", line 335, in _get_page
return HTMLPage.get_page(link, req, cache=self.cache)
File
"/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.
1-py2.7.egg/pip/index.py", line 452, in get_page
resp = urlopen(url)
File
"/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.
1-py2.7.egg/pip/download.py", line 85, in __call__
response = urllib2.urlopen(self.get_request(url))
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 392, in open
protocol = req.get_type()
File "/usr/lib/python2.7/urllib2.py", line 254, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: packages

Could not find any downloads that satisfy the requirement metar==1.4.0
No distributions at all found for metar==1.4.0
Storing complete log in /home/roy/.pip/pip.log

 

[Chris “Kwpolska†Warrick]

python setup.py sdist

OK, I run that and I get a metar-1.4.0.tar.gz under dist. If I move
that tarfile to my packages directory, and run pip, I get:

$ pip install --no-index --quiet --find-links packages metar==1.4.0 [snip]
ValueError: unknown url type: packages

The path to your cache directory is incorrect. I suggest using
absolute paths (eg. /home/user/packages) instead of relative paths,
which is likely what caused this issue.

What I can't figure out is what I need to do to go from a clone of the
github repo to a tarball I can drop into our packages directory. Is
there some tutorial somewhere that explains this?

Actually, you could even tar up that entire repo (or even get a nice
ready tarball from GItHub) and you will get something usable with pip.
For example, we in the Nikola project
(https://github.com/getnikola/nikola) upload the GitHub tarballs to
PyPI because we ship 99.9% of our tree anyways and hiring `setup.py
sdist` would be a waste of time (and would produce two
almost-identical-but-not-quite tarballs).

 

[Roy Smith]

python setup.py sdist

OK, I run that and I get a metar-1.4.0.tar.gz under dist. If I move
that tarfile to my packages directory, and run pip, I get:

$ pip install --no-index --quiet --find-links packages metar==1.4.0 [snip]
ValueError: unknown url type: packages

The path to your cache directory is incorrect. I suggest using
absolute paths (eg. /home/user/packages) instead of relative paths,
which is likely what caused this issue.

No, that's not it. It doesn't work with an absolute path either.

-----------------------------
$ pip install -v --no-index --find-links /home/roy/deploy/current/code/deploy/python/packages metar==1.4.0
Ignoring indexes: http://pypi.python.org/simple/
Downloading/unpacking metar==1.4.0
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/index.py", line 245, in _get_queued_page
page = self._get_page(location, req)
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/index.py", line 335, in _get_page
return HTMLPage.get_page(link, req, cache=self.cache)
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/index.py", line 452, in get_page
resp = urlopen(url)
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/download.py", line 85, in __call__
response = urllib2.urlopen(self.get_request(url))
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 392, in open
protocol = req.get_type()
File "/usr/lib/python2.7/urllib2.py", line 254, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: /home/roy/deploy/current/code/deploy/python/packages

Could not find any downloads that satisfy the requirement metar==1.4.0
No distributions at all found for metar==1.4.0
Exception information:
Traceback (most recent call last):
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 104, in main
status = self.run(options, args)
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/commands/install.py", line 245, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 978, in prepare_files
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/index.py", line 157, in find_requirement
raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for metar==1.4.0

Storing complete log in /home/roy/.pip/pip.log
-----------------------------



The tar file is there:

$ ls -l /home/roy/deploy/current/code/deploy/python/packages/metar-1.4.0.tar.gz
-rw-rw-r-- 1 roy roy 28542 Feb 15 16:55 /home/roy/deploy/current/code/deploy/python/packages/metar-1.4.0.tar.gz

 

[Roy Smith]

$ pip install --no-index --quiet --find-links packages metar==1.4.0 [snip]
ValueError: unknown url type: packages

The path to your cache directory is incorrect. I suggest using
absolute paths (eg. /home/user/packages) instead of relative paths,
which is likely what caused this issue.

No, that's not it. It doesn't work with an absolute path either.[/QUOTE]

OK, I figured this out. The on-line docs
(http://www.pip-installer.org/en/latest/reference/pip_wheel.html) say
you can give --find-links a path, but it looks like it insists on it
being a valid URL. If I prepend "file:" to the absolute path, it works.

Maybe this is something which has changed in newer versions of pip?
I've got 1.1 (and python 2.7.3). I'm pretty sure both of these are what
came with Ubuntu Precise.

 

[Chris Angelico]

It’s heavily outdated, and that IS the cause of your problem. pip 1.5
accepts such paths just fine. Please upgrade your pip.

http://packages.ubuntu.com/precise/python-pip
says it's shipping 1.0.1, even older. You get 1.1 with Quantal:
http://packages.ubuntu.com/quantal/python-pip
and (unsurprisingly) newer versions with newer Ubuntus.

Debian's just as bad, incidentally. On Wheezy (current stable), Debian
ships 1.1, though Jessie (current testing) has 1.4.1. But neither
Ubuntu Trusty nor Debian Sid (unreleased versions of each) ships 1.5.

ChrisA

 

[Roy Smith]

http://packages.ubuntu.com/precise/python-pip
says it's shipping 1.0.1, even older. You get 1.1 with Quantal:
http://packages.ubuntu.com/quantal/python-pip
and (unsurprisingly) newer versions with newer Ubuntus.

Debian's just as bad, incidentally. On Wheezy (current stable), Debian
ships 1.1, though Jessie (current testing) has 1.4.1. But neither
Ubuntu Trusty nor Debian Sid (unreleased versions of each) ships 1.5.

ChrisA

Yup. I just checked around. My dev machine (which is Ubuntu Precise,
plus some random upgrade history) has 1.1. Our production boxes (which
are much cleaner Precise installs) have 1.0 in /usr/bin; that's only
used for bootstrapping deployments. We have 1.4.1 in the virtualenv we
run out of.

Oh, yeah, we've still got a few Lucid boxes floating around on some
back-end machines. They're running:

pip 0.3.1 from /usr/lib/python2.6/dist-packages (python 2.6)

We tend not to upgrade stuff unless there's a good reason to. You never
know what will break (looking furtively in the direction of the Python
3.x mafiosi).

 

[Mark Lawrence]

We tend not to upgrade stuff unless there's a good reason to. You never
know what will break (looking furtively in the direction of the Python
3.x mafiosi).

Yeah, those really unpleasant, nasty, horrible mafiosi who have the
audacity to point out that people have only been given seven (ish) years
so far to plan and implement their upgrades. Then the mafiosi further
complain when people ask if they can have a Python 2.8 to help plan and
implement their upgrades. Yep, this mafiosi mob really do have a lot to
answer for. Not.

 

[MRAB]

Yeah, those really unpleasant, nasty, horrible mafiosi who have the
audacity to point out that people have only been given seven (ish) years
so far to plan and implement their upgrades. Then the mafiosi further
complain when people ask if they can have a Python 2.8 to help plan and
implement their upgrades. Yep, this mafiosi mob really do have a lot to
answer for. Not.

And, what's more, this mafiosi mob cruelly continues to provide the
previous releases on its website free of charge, including all the
source code.

 

[Mark Lawrence]

And, what's more, this mafiosi mob cruelly continues to provide the
previous releases on its website free of charge, including all the
source code.

The obligatory "And apart from that, what have the mafiosi ..."