Features for a Python package manager?

Discussion in 'Python' started by Georg Brandl, Dec 25, 2004.

  1. Georg Brandl

    Georg Brandl Guest

    Hello c.l.py,

    what features would you expect of a Python package manager, similar to
    CPAN or rubygems?

    I am currently planning to write such a thing, at first privately for
    myself, and if it's proving useful, I think about releasing it.

    I plan to model it after gentoo's portage, with less features of course.
    Would this be acceptable?

    feel-free-to-comment-ly yours,
    Georg
     
    Georg Brandl, Dec 25, 2004
    #1
    1. Advertising

  2. Georg Brandl

    Nick Coghlan Guest

    Georg Brandl wrote:
    > Hello c.l.py,
    >
    > what features would you expect of a Python package manager, similar to
    > CPAN or rubygems?
    >
    > I am currently planning to write such a thing, at first privately for
    > myself, and if it's proving useful, I think about releasing it.


    I've contemplated such a beast myself, but have never got further than a cursory
    look at "what work has already been done that I should take into account?"

    Howevever, it sounds like that info would be useful to you, so here's what I've got:

    PEP 314 (Metadata for Python Packages v 1.1) details the properties of the
    metadata that the Python repository relies on.

    Don't worry about PEP 241 - it has been superceded by PEP 314.

    PEP 301 describes PyPI (Python Package Index) - the central index of available
    packages. A running version of it can be found at http://www.python.org/pypi

    PEP 243 is probably worth looking at, although I believe it is basically
    superceded by PEP 301.

    PEP 262 describes a proposed (local system) installation database. An
    implementation already exists in the nondist section of CPython's CVS
    repository, but further progress is currently stalled due to lack of a champion.

    It would also be good to be aware of how distutils currently works (even if
    that's just by reading its documentation). Doing your own trawl through PEP 0
    wouldn't hurt, either (it's possible I missed something relevant). The archives
    of python-dev and the distutils SIG may also provide good info.

    If you're serious about doing a good job (i.e. to a "possibly included with the
    standard Python distribution some day" standard), it may be worth posting a
    question on python-dev about it. Just say that you're contemplating writing such
    a tool for your own use, and are interested in any ideas which aren't currently
    documented in the places I mentioned above. You'll find several of the authors
    of the above material frequent python-dev, and will hopefully have some good
    pointers regarding what needs to be done.

    > I plan to model it after gentoo's portage, with less features of course.
    > Would this be acceptable?


    I don't know enough about Portage to answer that question. I do know any package
    manager which made it into the standard distribution would need to work for at
    least the big three platforms (Windows/Mac/*nix) :)

    Cheers,
    Nick.

    --
    Nick Coghlan | | Brisbane, Australia
    ---------------------------------------------------------------
    http://boredomandlaziness.skystorm.net
     
    Nick Coghlan, Dec 25, 2004
    #2
    1. Advertising

  3. Georg Brandl

    Mike Meyer Guest

    Nick Coghlan <> writes:

    > I don't know enough about Portage to answer that question. I do know
    > any package manager which made it into the standard distribution would
    > need to work for at least the big three platforms (Windows/Mac/*nix) :)


    Being written in python - and hopefully integrated into Distutils -
    why shouldn't it work on any platform that Python worked on?

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, Dec 26, 2004
    #3
  4. Georg Brandl

    Robert Kern Guest

    Mike Meyer wrote:
    > Nick Coghlan <> writes:
    >
    >
    >>I don't know enough about Portage to answer that question. I do know
    >>any package manager which made it into the standard distribution would
    >>need to work for at least the big three platforms (Windows/Mac/*nix) :)

    >
    >
    > Being written in python - and hopefully integrated into Distutils -
    > why shouldn't it work on any platform that Python worked on?


    Assumptions about directory structures, and the like. IIRC, Portage was
    written for the Gentoo project, so it could assume that it was
    installing stuff to a Gentoo system. Package management systems have a
    distressing habit of infecting their *architecture* with these
    assumptions. It makes adapting them difficult.

    Also, Portage needs to execute subprocesses, something which is
    notoriously platform dependent. 2.4's subprocess module should probably
    be used here, but I don't think Portage does, yet. OTOH, it's Gentoo, so
    it wouldn't surprise me, either. :)

    --
    Robert Kern


    "In the fields of hell where the grass grows high
    Are the graves of dreams allowed to die."
    -- Richard Harter
     
    Robert Kern, Dec 26, 2004
    #4
  5. Georg Brandl

    Georg Brandl Guest

    Robert Kern wrote:
    > Mike Meyer wrote:
    >> Nick Coghlan <> writes:
    >>
    >>
    >>>I don't know enough about Portage to answer that question. I do know
    >>>any package manager which made it into the standard distribution would
    >>>need to work for at least the big three platforms (Windows/Mac/*nix) :)

    >>
    >>
    >> Being written in python - and hopefully integrated into Distutils -
    >> why shouldn't it work on any platform that Python worked on?

    >
    > Assumptions about directory structures, and the like.


    That is already taken care by the distutils. The various PEPs already
    describe a simple method how to store package metadata, and I will try
    to follow these standards as close as possible. Of course, the PyPI
    would have to be adapted for that.

    > IIRC, Portage was
    > written for the Gentoo project, so it could assume that it was
    > installing stuff to a Gentoo system. Package management systems have a
    > distressing habit of infecting their *architecture* with these
    > assumptions. It makes adapting them difficult.


    > Also, Portage needs to execute subprocesses, something which is
    > notoriously platform dependent. 2.4's subprocess module should probably
    > be used here, but I don't think Portage does, yet. OTOH, it's Gentoo, so
    > it wouldn't surprise me, either. :)


    That's right, but I would just draw the line at Python's standard
    library. What is possible with it, is done, what not, is left out.

    regards,
    Georg
     
    Georg Brandl, Dec 26, 2004
    #5
  6. Georg Brandl

    Georg Brandl Guest

    Robert Kern wrote:
    > Georg Brandl wrote:
    >> Robert Kern wrote:
    >>
    >>>Mike Meyer wrote:
    >>>
    >>>>Nick Coghlan <> writes:
    >>>>
    >>>>
    >>>>
    >>>>>I don't know enough about Portage to answer that question. I do know
    >>>>>any package manager which made it into the standard distribution would
    >>>>>need to work for at least the big three platforms (Windows/Mac/*nix) :)
    >>>>
    >>>>
    >>>>Being written in python - and hopefully integrated into Distutils -
    >>>>why shouldn't it work on any platform that Python worked on?
    >>>
    >>>Assumptions about directory structures, and the like.

    >>
    >>
    >> That is already taken care by the distutils.

    >
    > More or less. I'm in the middle of packaging up ~40 Python packages for
    > the Mac[1]. For a "standard" packaging mechanism, distutils allows for
    > some bloody idiosyncratic ways to say "put these files there". This is a
    > hard problem, and it's not solved entirely by distutils.
    >
    > I don't think anyone has satisfactorily solved the problem of
    > distributing data with libraries. Well, okay, the *distribution* isn't
    > the problem. Having the library be able to locate that data on all
    > platforms is difficult. Some packages will more-or-less hardcode
    > *nix-type paths which may be inappropriate even on some *nix-type
    > platforms (yes, PyX, I'm looking at you :)). A general package system
    > like Portage has the freedom of being able to dictate these things. A
    > Python package manager does not. You can establish a standard for each
    > of the Big Three platforms, but it may not do you much good if the
    > libraries don't know about it.
    >
    > CPAN is a closer analogue in this regard and would probably be a better
    > tool to study and copy from than Portage. I don't know much about it,
    > but how it responds to these issues will probably more instructive than
    > how Portage does.


    CPAN, as I understand it, is a mirror system for packages, which
    downloads packages and manages those. Correct me if I'm wrong, but the
    actual installation is done by each package's Makefile.PL script and not
    by CPAN itself.

    This is also Portage's philosopy. It only issues those commands needed
    to build and install the package (ideally "tar xzf package; cd package;
    make install"). Of course one can do additional steps such as applying
    patches, but where the files go etc. is the author's responsibility, not
    the package system's.

    [Problems with dependencies]

    I am aware of these problems - however I feel like the package author
    has to be the one to address them. There must be a cooperation between
    author and packager to clear these issues.

    About non-Python-dependencies: The plan is to add a "Non-Py-Requires:"
    (or so) field to the metadata that is displayed to the user, and it will
    be his responsibility to install these libraries/headers/files first.

    >> The various PEPs already
    >> describe a simple method how to store package metadata, and I will try
    >> to follow these standards as close as possible. Of course, the PyPI
    >> would have to be adapted for that.

    >
    > Fair enough. What parts of Portage do you intend to steal, then?


    Right, that's my weak point. I do not overly like the PEPs' way, but
    integrating with distutils is the way to go.

    regards,
    Georg
     
    Georg Brandl, Dec 26, 2004
    #6
  7. Georg Brandl

    Robert Kern Guest

    Georg Brandl wrote:
    > Robert Kern wrote:
    >
    >>Mike Meyer wrote:
    >>
    >>>Nick Coghlan <> writes:
    >>>
    >>>
    >>>
    >>>>I don't know enough about Portage to answer that question. I do know
    >>>>any package manager which made it into the standard distribution would
    >>>>need to work for at least the big three platforms (Windows/Mac/*nix) :)
    >>>
    >>>
    >>>Being written in python - and hopefully integrated into Distutils -
    >>>why shouldn't it work on any platform that Python worked on?

    >>
    >>Assumptions about directory structures, and the like.

    >
    >
    > That is already taken care by the distutils.


    More or less. I'm in the middle of packaging up ~40 Python packages for
    the Mac[1]. For a "standard" packaging mechanism, distutils allows for
    some bloody idiosyncratic ways to say "put these files there". This is a
    hard problem, and it's not solved entirely by distutils.

    I don't think anyone has satisfactorily solved the problem of
    distributing data with libraries. Well, okay, the *distribution* isn't
    the problem. Having the library be able to locate that data on all
    platforms is difficult. Some packages will more-or-less hardcode
    *nix-type paths which may be inappropriate even on some *nix-type
    platforms (yes, PyX, I'm looking at you :)). A general package system
    like Portage has the freedom of being able to dictate these things. A
    Python package manager does not. You can establish a standard for each
    of the Big Three platforms, but it may not do you much good if the
    libraries don't know about it.

    CPAN is a closer analogue in this regard and would probably be a better
    tool to study and copy from than Portage. I don't know much about it,
    but how it responds to these issues will probably more instructive than
    how Portage does.

    You also have problems with distributing non-Python dependencies like
    libraries. You can sometimes punt and require the user to have already
    installed said libraries and headers to compile against. This will
    probably only work for source distributions on *nixes. You can probably
    get away with appropriately placed DLLs on Windows. Macs may be trickier
    (we Mac users are fussy :)).

    Exercise: VTK with its Python and TCL libraries (why TCL? It's necessary
    to use VTK with Tkinter, which the premier VTK-Python application,
    MayaVi, uses) should package up cleanly for all 3 main platforms.
    External libraries *and* data! It's a joy!

    Some other "problem packages"[2] to practice on: PyX, ReportLab, PIL,
    Scipy, matplotlib.

    [1] http://www.scipy.org/wikis/featurerequests/MacEnthon
    [2] In the sense that they are, to some extent, not painless to package
    up in a very neat way on multiple platforms for one reason or another.
    These are all very fine packages; packaging is a hard problem; and I'm
    grateful for how well they *do* package up so far.

    > The various PEPs already
    > describe a simple method how to store package metadata, and I will try
    > to follow these standards as close as possible. Of course, the PyPI
    > would have to be adapted for that.


    Fair enough. What parts of Portage do you intend to steal, then?

    Please don't read any of this as discouragement. Python package
    management desperately needs a champion, and I'm very glad to see
    someone tackling these issues. I for one fully intend to use as much of
    what you produce as I can to make MacEnthon fully upgradable.

    --
    Robert Kern


    "In the fields of hell where the grass grows high
    Are the graves of dreams allowed to die."
    -- Richard Harter
     
    Robert Kern, Dec 26, 2004
    #7
  8. Georg Brandl

    Bulba! Guest

    On Sat, 25 Dec 2004 11:37:42 +0100, Georg Brandl <>
    wrote:

    >what features would you expect of a Python package manager, similar to
    >CPAN or rubygems?


    IMVHO it would be nice if it had a feature for "upload package/module
    I have just developed" - maybe PyPi would fill up faster if it would.





    --
    It's a man's life in a Python Programming Association.
     
    Bulba!, Dec 29, 2004
    #8
  9. Georg Brandl

    Mike Meyer Guest

    Bulba! <> writes:

    > On Sat, 25 Dec 2004 11:37:42 +0100, Georg Brandl <>
    > wrote:
    >
    >>what features would you expect of a Python package manager, similar to
    >>CPAN or rubygems?

    >
    > IMVHO it would be nice if it had a feature for "upload package/module
    > I have just developed" - maybe PyPi would fill up faster if it would.


    Distutils already has a feature to do that for you.

    <mike
    --
    Mike Meyer <> http://www.mired.org/home/mwm/
    Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
     
    Mike Meyer, Dec 30, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jeffrey A. Zelt
    Replies:
    0
    Views:
    398
    Jeffrey A. Zelt
    Feb 28, 2005
  2. Jonathan Mcdougall
    Replies:
    2
    Views:
    525
    Kaz Kylheku
    Nov 3, 2005
  3. BillJosephson
    Replies:
    148
    Views:
    2,773
    peter koch
    Jan 27, 2007
  4. padma
    Replies:
    0
    Views:
    378
    padma
    Oct 3, 2007
  5. Metalone
    Replies:
    0
    Views:
    366
    Metalone
    Jan 6, 2010
Loading...

Share This Page