ubuntu dist-packages

[Robin Becker]

I was surprised a couple of days ago when trying to assist a colleage with his
python setup on a ubuntu 9.04 system.

We built our c-extensions and manually copied them into place, but site-packages
wasn't there. It seems that ubuntu now wants stuff to go into
lib/python2.6/dist-packages.

What is the relation between dist-packages/site-packages if any? Is this just a
name change or is there some other problem being addressed?

For developers is it best just to create one's own private installations from
the original tarballs?

 

[Diez B. Roggisch]

I was surprised a couple of days ago when trying to assist a colleage with
his python setup on a ubuntu 9.04 system.

We built our c-extensions and manually copied them into place, but
site-packages wasn't there. It seems that ubuntu now wants stuff to go
into lib/python2.6/dist-packages.

What is the relation between dist-packages/site-packages if any? Is this
just a name change or is there some other problem being addressed?

For developers is it best just to create one's own private installations
from the original tarballs?

I don't know much about the atrocities distributions commit to
python-installations (ripping out distutils "because it's a developer-only
thing", trying to save a few bytes here and there by unifying
install-locations between interpreter versions and whatnot), but I think
the main problem is that you don't use distutils - or setuptools (that gets
often a bad rap even if for most cases it works flawless)

If you'd use that to build & install your extension, it would figure out
where to place the resulting package (or egg) by itself.

Diez

 

[Jorgen Grahn]

....

I don't know much about the atrocities distributions commit to
python-installations (ripping out distutils "because it's a developer-only
thing",

Well, if you are thinking about Debian Linux, it's not as much
"ripping out" as "splitting into a separate package with a non-obvious
name". Annoying at times, but hardly an atrocity.

/Jorgen

 

[Robin Becker]

On Wed, 26 Aug 2009 12:46:13 +0200, Diez B. Roggisch .........
Well, if you are thinking about Debian Linux, it's not as much
"ripping out" as "splitting into a separate package with a non-obvious
name". Annoying at times, but hardly an atrocity.

so where is the official place for user installed stuff on ubuntu/debian ie will
there be dist-packages and site-packages.

 

[Robin Becker]

On Wed, 26 Aug 2009 12:46:13 +0200, Diez B. Roggisch .........
Well, if you are thinking about Debian Linux, it's not as much
"ripping out" as "splitting into a separate package with a non-obvious
name". Annoying at times, but hardly an atrocity.

so where is the official place for user installed stuff on ubuntu/debian ie will
there be dist-packages and site-packages.

 

[Florian Diesch]

I was surprised a couple of days ago when trying to assist a colleage
with his python setup on a ubuntu 9.04 system.

We built our c-extensions and manually copied them into place, but
site-packages wasn't there. It seems that ubuntu now wants stuff to go
into lib/python2.6/dist-packages.

What is the relation between dist-packages/site-packages if any? Is
this just a name change or is there some other problem being
addressed?

For developers is it best just to create one's own private
installations from the original tarballs?

From /usr/lib/python2.6/site.py:

,----
| For Debian and derivatives, this sys.path is augmented with directories
| for packages distributed within the distribution. Local addons go
| into /usr/local/lib/python<version>/dist-packages, Debian addons
| install into /usr/{lib,share}/python<version>/dist-packages.
| /usr/lib/python<version>/site-packages is not used.
`----




Florian

 

[Jorgen Grahn]

so where is the official place for user installed stuff on
ubuntu/debian ie will
there be dist-packages and site-packages.

I don't know, but I know Debian has a group of people working out how
to package Python and software written in Python. Those guys have a
home page somewhere at debian.org -- there should be information
there, and/or under /usr/share/doc/python* on your system.

Another answer is "let distutils do its job and do not worry", but
sometimes you need to know the rules and the reasons ...

/Jorgen

 

[Paul Boddie]

Well, if you are thinking about Debian Linux, it's not as much
"ripping out" as "splitting into a separate package with a non-obvious
name". Annoying at times, but hardly an atrocity.

Indeed. Having seen two packages today which insisted on setuptools,
neither really needing it, and with one actively trying to download
stuff from the Internet (fifteen seconds warning - how generous!) when
running setup.py, it seems to me that it isn't the distribution
packagers who need to be re-thinking how they install Python software.

Generally, distributions have to manage huge amounts of software and
uphold reasonable policies without creating unnecessary maintenance.
Sadly, until very recently (and I'm still not entirely sure if there's
really been an attitude change) the Pythonic packaging brigade has
refused to even consider the needs of one of the biggest groups of
consumers of the upstream code. Consequently, distributions will
always devise different ways of storing installed Python software,
documentation and resources, mostly because the Pythonic tools have
been deficient, particularly in the management of the latter
categories.

Paul

 

[Robin Becker]

Florian Diesch wrote:
..........

,----
| For Debian and derivatives, this sys.path is augmented with directories
| for packages distributed within the distribution. Local addons go
| into /usr/local/lib/python<version>/dist-packages, Debian addons
| install into /usr/{lib,share}/python<version>/dist-packages.
| /usr/lib/python<version>/site-packages is not used.
`----

the above is not present in my windows documentation (or indeed site.py) at all
so it seems they just decided to change the name. Anyone trying to debug why
their distutils or setuptools or whichever python packager is going wrong will
have yet another detail to remember. In addition, as any one who has done such
trivial changes will already know, they forgot to do it globally eg my 0.4.1.0
version of the "Debian Python Policy" document explicitly mentions site-packages.

 

[Robin Becker]

Florian Diesch wrote:
..........

,----
| For Debian and derivatives, this sys.path is augmented with directories
| for packages distributed within the distribution. Local addons go
| into /usr/local/lib/python<version>/dist-packages, Debian addons
| install into /usr/{lib,share}/python<version>/dist-packages.
| /usr/lib/python<version>/site-packages is not used.
`----

the above is not present in my windows documentation (or indeed site.py) at all
so it seems they just decided to change the name. Anyone trying to debug why
their distutils or setuptools or whichever python packager is going wrong will
have yet another detail to remember. In addition, as any one who has done such
trivial changes will already know, they forgot to do it globally eg my 0.4.1.0
version of the "Debian Python Policy" document explicitly mentions site-packages.

 

[Diez B. Roggisch]

Indeed. Having seen two packages today which insisted on setuptools,
neither really needing it, and with one actively trying to download
stuff from the Internet (fifteen seconds warning - how generous!) when
running setup.py, it seems to me that it isn't the distribution
packagers who need to be re-thinking how they install Python software.

Generally, distributions have to manage huge amounts of software and
uphold reasonable policies without creating unnecessary maintenance.
Sadly, until very recently (and I'm still not entirely sure if there's
really been an attitude change) the Pythonic packaging brigade has
refused to even consider the needs of one of the biggest groups of
consumers of the upstream code. Consequently, distributions will
always devise different ways of storing installed Python software,
documentation and resources, mostly because the Pythonic tools have
been deficient, particularly in the management of the latter
categories.

You mean it's the problem of the python packaging that it can't deal with
RPMs, debs, tgzs, OSX bundles, MSIs and
<put-in-the-next-big-packaging-thing-here>?

Multiplied by the various packaging philosophies the respective distros
build based on these have?

I'm a Python-developer. I develop libraries and tools for Python, and want
others to be able to install these - as I want to install things *other*
python developers created.

Setuptools let's me do that (most of the time. And I mean most).

If somebody thinks he wants to include these in whatever form he prefers -
fine with me. But it's hardly *my* problem, or that of the Python world in
general, to fulfill the requirements some other people come up with.


Diez

 

[Paul Boddie]

You mean it's the problem of the python packaging that it can't deal with
RPMs, debs, tgzs, OSX bundles, MSIs and
<put-in-the-next-big-packaging-thing-here>?

No, it's the problem of the Pythonic packaging brigade that package
retrieval, building and installing is combined into one unsatisfactory
whole. Certainly, it's annoying to discover when building some
extension that the Python headers are missing, but the rhetorical
vehicle used in the Python community is to frame the distributions as
people who like to move stuff around unnecessarily and to corrupt
other people's work. In fact, the distributions have proven themselves
to be quite competent at managing huge numbers of software packages in
a semi-automated fashion, so it's baffling that people doing work on
Pythonic packaging tools wouldn't want to learn as much as they can
about how those people manage it.

For me, when making Debian packages, it has become easier to use the
debhelper stuff to install things like documentation and resources
than it is to figure out which special options have to be passed to
the particular distutils incarnation of the setup function in order to
get such stuff put in the right place, especially since distutils
probably still employs an ad-hoc 1990s proprietary UNIX "oh just dump
that stuff in some directory or other and forget about it" mentality.
Really, distutils should be all about *dist*ribution and even then get
out of the way as much as possible - the hard stuff is extracting the
appropriate knowledge about the built Python interpreter in order to
build extensions. Installation should be left to something which has
an opinion about where things should be placed on a particular system,
and which has the capability to know how to uninstall stuff - a
capability which never seems to get any closer in the distutils scene.

Paul

 

[David Cournapeau]

You mean it's the problem of the python packaging that it can't deal with
RPMs, debs, tgzs, OSX bundles, MSIs and
<put-in-the-next-big-packaging-thing-here>?

Of course it is - not because distutils should know about them but on
the contrary because it should be possible to tweak the installation
parameters to accomodate the various packaging solutions. Autotools,
cmake, etc... do not need anything about rpm, debian, msi, and yet,
they can be used to that purpose.

cheers,

David

 

[Florian Diesch]

Florian Diesch wrote:
.........

the above is not present in my windows documentation (or indeed
site.py) at all so it seems they just decided to change the
name. Anyone trying to debug why their distutils or setuptools or
whichever python packager is going wrong will have yet another detail
to remember.

setuptools works fine for me in Ubuntu 9.04; eggs go to
/usr/local/lib/python2.6/dist-packages/ like they should according to site.py

In addition, as any one who has done such trivial changes
will already know, they forgot to do it globally eg my 0.4.1.0 version
of the "Debian Python Policy" document explicitly mentions
site-packages.

It's a change for the Python 2.6 package, see
<http://packages.debian.org/changelogs/pool/main/p/python2.6/python2.6_2.6.2-2/changelog>
and /usr/lib/python2.5/site.py

Python 2.6 is only in Debian experimental, I guess there is a new
version of the Debian Python Policy coming.



Florian

 

[David Lyon]

No, it's the problem of the Pythonic packaging brigade that package
retrieval, building and installing is combined into one unsatisfactory
whole.

Brigade? That implies a disciplined and systematic approach..

I would suggest a better term is just guerrilla fighters.

There's no central planning.. no looking out for the greater good..

No sense of future.. forgive my pessimism..

In fact, the distributions have proven themselves
to be quite competent at managing huge numbers of software packages in
a semi-automated fashion, so it's baffling that people doing work on
Pythonic packaging tools wouldn't want to learn as much as they can
about how those people manage it.

That's a serious and true allegation. I think the answer is that the
people that are running it aren't interested in cleaning things up
for the linux or windows platform.

Consensus seems to be just "remove" anything that relates to windows
or linux as an approach to harmonising it.

I believe that it should be harmonised.

For me, when making Debian packages, it has become easier to use the
debhelper stuff to install things like documentation and resources
than it is to figure out which special options have to be passed to
the particular distutils incarnation of the setup function in order to
get such stuff put in the right place, especially since distutils
probably still employs an ad-hoc 1990s proprietary UNIX "oh just dump
that stuff in some directory or other and forget about it" mentality.

It doesn't. Actually, it can't even do that very well.

90's.... which '90s ? 1890's ?

Really, distutils should be all about *dist*ribution and even then get
out of the way as much as possible - the hard stuff is extracting the
appropriate knowledge about the built Python interpreter in order to
build extensions.
Exactly.

Installation should be left to something which has
an opinion about where things should be placed on a particular system,
and which has the capability to know how to uninstall stuff - a
capability which never seems to get any closer in the distutils scene.

(sigh) - it is true...

David