Features for a Python package manager?

[Georg Brandl]

Hello c.l.py,

what features would you expect of a Python package manager, similar to
CPAN or rubygems?

I am currently planning to write such a thing, at first privately for
myself, and if it's proving useful, I think about releasing it.

I plan to model it after gentoo's portage, with less features of course.
Would this be acceptable?

feel-free-to-comment-ly yours,
Georg

 

[Nick Coghlan]

Hello c.l.py,

what features would you expect of a Python package manager, similar to
CPAN or rubygems?

I am currently planning to write such a thing, at first privately for
myself, and if it's proving useful, I think about releasing it.

I've contemplated such a beast myself, but have never got further than a cursory
look at "what work has already been done that I should take into account?"

Howevever, it sounds like that info would be useful to you, so here's what I've got:

PEP 314 (Metadata for Python Packages v 1.1) details the properties of the
metadata that the Python repository relies on.

Don't worry about PEP 241 - it has been superceded by PEP 314.

PEP 301 describes PyPI (Python Package Index) - the central index of available
packages. A running version of it can be found at http://www.python.org/pypi

PEP 243 is probably worth looking at, although I believe it is basically
superceded by PEP 301.

PEP 262 describes a proposed (local system) installation database. An
implementation already exists in the nondist section of CPython's CVS
repository, but further progress is currently stalled due to lack of a champion.

It would also be good to be aware of how distutils currently works (even if
that's just by reading its documentation). Doing your own trawl through PEP 0
wouldn't hurt, either (it's possible I missed something relevant). The archives
of python-dev and the distutils SIG may also provide good info.

If you're serious about doing a good job (i.e. to a "possibly included with the
standard Python distribution some day" standard), it may be worth posting a
question on python-dev about it. Just say that you're contemplating writing such
a tool for your own use, and are interested in any ideas which aren't currently
documented in the places I mentioned above. You'll find several of the authors
of the above material frequent python-dev, and will hopefully have some good
pointers regarding what needs to be done.

> I plan to model it after gentoo's portage, with less features of course.
> Would this be acceptable?

I don't know enough about Portage to answer that question. I do know any package
manager which made it into the standard distribution would need to work for at
least the big three platforms (Windows/Mac/*nix)

Cheers,
Nick.

 

[Mike Meyer]

I don't know enough about Portage to answer that question. I do know
any package manager which made it into the standard distribution would
need to work for at least the big three platforms (Windows/Mac/*nix)

Being written in python - and hopefully integrated into Distutils -
why shouldn't it work on any platform that Python worked on?

<mike

 

[Robert Kern]

Being written in python - and hopefully integrated into Distutils -
why shouldn't it work on any platform that Python worked on?

Assumptions about directory structures, and the like. IIRC, Portage was
written for the Gentoo project, so it could assume that it was
installing stuff to a Gentoo system. Package management systems have a
distressing habit of infecting their *architecture* with these
assumptions. It makes adapting them difficult.

Also, Portage needs to execute subprocesses, something which is
notoriously platform dependent. 2.4's subprocess module should probably
be used here, but I don't think Portage does, yet. OTOH, it's Gentoo, so
it wouldn't surprise me, either.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Georg Brandl]

Assumptions about directory structures, and the like.

That is already taken care by the distutils. The various PEPs already
describe a simple method how to store package metadata, and I will try
to follow these standards as close as possible. Of course, the PyPI
would have to be adapted for that.

IIRC, Portage was
written for the Gentoo project, so it could assume that it was
installing stuff to a Gentoo system. Package management systems have a
distressing habit of infecting their *architecture* with these
assumptions. It makes adapting them difficult.

Also, Portage needs to execute subprocesses, something which is
notoriously platform dependent. 2.4's subprocess module should probably
be used here, but I don't think Portage does, yet. OTOH, it's Gentoo, so
it wouldn't surprise me, either.

That's right, but I would just draw the line at Python's standard
library. What is possible with it, is done, what not, is left out.

regards,
Georg

 

[Georg Brandl]

That is already taken care by the distutils.

More or less. I'm in the middle of packaging up ~40 Python packages for
the Mac[1]. For a "standard" packaging mechanism, distutils allows for
some bloody idiosyncratic ways to say "put these files there". This is a
hard problem, and it's not solved entirely by distutils.

I don't think anyone has satisfactorily solved the problem of
distributing data with libraries. Well, okay, the *distribution* isn't
the problem. Having the library be able to locate that data on all
platforms is difficult. Some packages will more-or-less hardcode
*nix-type paths which may be inappropriate even on some *nix-type
platforms (yes, PyX, I'm looking at you ). A general package system
like Portage has the freedom of being able to dictate these things. A
Python package manager does not. You can establish a standard for each
of the Big Three platforms, but it may not do you much good if the
libraries don't know about it.

CPAN is a closer analogue in this regard and would probably be a better
tool to study and copy from than Portage. I don't know much about it,
but how it responds to these issues will probably more instructive than
how Portage does.

CPAN, as I understand it, is a mirror system for packages, which
downloads packages and manages those. Correct me if I'm wrong, but the
actual installation is done by each package's Makefile.PL script and not
by CPAN itself.

This is also Portage's philosopy. It only issues those commands needed
to build and install the package (ideally "tar xzf package; cd package;
make install"). Of course one can do additional steps such as applying
patches, but where the files go etc. is the author's responsibility, not
the package system's.

[Problems with dependencies]

I am aware of these problems - however I feel like the package author
has to be the one to address them. There must be a cooperation between
author and packager to clear these issues.

About non-Python-dependencies: The plan is to add a "Non-Py-Requires:"
(or so) field to the metadata that is displayed to the user, and it will
be his responsibility to install these libraries/headers/files first.

Fair enough. What parts of Portage do you intend to steal, then?

Right, that's my weak point. I do not overly like the PEPs' way, but
integrating with distutils is the way to go.

regards,
Georg

 

[Robert Kern]

That is already taken care by the distutils.

More or less. I'm in the middle of packaging up ~40 Python packages for
the Mac[1]. For a "standard" packaging mechanism, distutils allows for
some bloody idiosyncratic ways to say "put these files there". This is a
hard problem, and it's not solved entirely by distutils.

I don't think anyone has satisfactorily solved the problem of
distributing data with libraries. Well, okay, the *distribution* isn't
the problem. Having the library be able to locate that data on all
platforms is difficult. Some packages will more-or-less hardcode
*nix-type paths which may be inappropriate even on some *nix-type
platforms (yes, PyX, I'm looking at you ). A general package system
like Portage has the freedom of being able to dictate these things. A
Python package manager does not. You can establish a standard for each
of the Big Three platforms, but it may not do you much good if the
libraries don't know about it.

CPAN is a closer analogue in this regard and would probably be a better
tool to study and copy from than Portage. I don't know much about it,
but how it responds to these issues will probably more instructive than
how Portage does.

You also have problems with distributing non-Python dependencies like
libraries. You can sometimes punt and require the user to have already
installed said libraries and headers to compile against. This will
probably only work for source distributions on *nixes. You can probably
get away with appropriately placed DLLs on Windows. Macs may be trickier
(we Mac users are fussy ).

Exercise: VTK with its Python and TCL libraries (why TCL? It's necessary
to use VTK with Tkinter, which the premier VTK-Python application,
MayaVi, uses) should package up cleanly for all 3 main platforms.
External libraries *and* data! It's a joy!

Some other "problem packages"[2] to practice on: PyX, ReportLab, PIL,
Scipy, matplotlib.

[1] http://www.scipy.org/wikis/featurerequests/MacEnthon
[2] In the sense that they are, to some extent, not painless to package
up in a very neat way on multiple platforms for one reason or another.
These are all very fine packages; packaging is a hard problem; and I'm
grateful for how well they *do* package up so far.

The various PEPs already
describe a simple method how to store package metadata, and I will try
to follow these standards as close as possible. Of course, the PyPI
would have to be adapted for that.

Fair enough. What parts of Portage do you intend to steal, then?

Please don't read any of this as discouragement. Python package
management desperately needs a champion, and I'm very glad to see
someone tackling these issues. I for one fully intend to use as much of
what you produce as I can to make MacEnthon fully upgradable.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Bulba!]

what features would you expect of a Python package manager, similar to
CPAN or rubygems?

IMVHO it would be nice if it had a feature for "upload package/module
I have just developed" - maybe PyPi would fill up faster if it would.

 

[Mike Meyer]

IMVHO it would be nice if it had a feature for "upload package/module
I have just developed" - maybe PyPi would fill up faster if it would.

Distutils already has a feature to do that for you.

<mike