Python needs a CPyAN

[John Roth]

[Stephen Ferg]
[ ... wants a CPyAN ...]

[John Roth]

How does the Python Package Index not meet this need?

I can think of one very serious difference, which has
kept me from placing a module in the PPI.

Ummm... Care to let us in on it? While your subtle hint may make sense
to those who have submitted packages for inclusion, I suspect I speak
for the many more of us who haven't. What's your objection?

1. Where do I store the module? I don't have a web site, and
it isn't on sourceforge.

2. For what I've done, the standard installer script required
for PPI is stoopid. All that's really needed to install is unpack
to the right directory. That's it. However, to use distutils I
had to write a packager program to create all the entries
I needed. This took me quite a few hours and left a very
sour taste since a lot of things that I thought should have
been automatic weren't.

That's why the Python port of FIT isn't on PPI. And
it won't be on PPI until I get around to moving it to
Sourceforge, which will happen sometime in the very
misty future.

 

[John Roth]

typo, he probably meant

"I can't think of a ..."

You may think you're funny.
You're not.

John Roth

 

[=?ISO-8859-1?Q?Michael_Str=F6der?=]

<aol>
We had some program (can't remember if it was spamassassin or an
antivir, I'm not the sysadmin !-) gone mad and all our customers mails
blocked for a full day because of such a wonderfull update of a perl
module...
</aol>

Well, could have happened with a Debian [*] or *BSD, etc. update as well,
right?

Yes.

Ciao, Michael.

 

[Mike Meyer]

Gerhard> Well, could have happened with a Debian [*] or *BSD,
Gerhard> etc. update as well, right?

Yes, userss of systems with proper package management shouldn't want
to install anything from CPAN. On the other hand, there are many
people who are stuck with a braindead distribution or even OS that
doesn't have proper package management, or whose packages are severely
limited.

Actually, what people on systems with proper package management want
is packages that pull stuff from CPAN and install them as
packages. Doing this right is hard, because CPAN makes assumptions
about where things are liable to be installed. For Python, what we
want are packages that use distutils to install them as
packages. Doing this right is trivial, because distutils gets all the
relevant information from the environment.

<mike

 

[Mike Meyer]

1. Where do I store the module? I don't have a web site, and
it isn't on sourceforge.

Ok, that's a definite need. Possibly python.org could host a mechanism
to allow people to post modules. I'd do it myself, but I'm not sure
how long I'm going to have my current net connection for.

2. For what I've done, the standard installer script required
for PPI is stoopid. All that's really needed to install is unpack
to the right directory. That's it. However, to use distutils I
had to write a packager program to create all the entries
I needed. This took me quite a few hours and left a very
sour taste since a lot of things that I thought should have
been automatic weren't.

I found the standard installer script trivial to set up. If it took
hours, then you're doing something wrong. It even made it possible for
me to distribute C modules to Windows, whereas those trying to build
the module on Windows prior to me putting it into distutils never
succeeded.

<mike

 

[Nick Craig-Wood]

Actually, what people on systems with proper package management want
is packages that pull stuff from CPAN and install them as
packages.

Absolutely!

For instance with Debian and Perl, dh-make-perl will turn a perl
module (tar.gz or straight from CPAN) into a .deb which you then
managage as normal. When using Debian I never use CPAN and always
dh-make-perl because I'd much rather have the package managed via
dpkg/apt-get.

That said I haven't found many perl modules that I wanted that weren't
already packaged for Debian. The same with python modules too.

Doing this right is hard, because CPAN makes assumptions
about where things are liable to be installed.

No idea how dh-make-perl works. It does though!

For Python, what we want are packages that use distutils to install
them as packages. Doing this right is trivial, because distutils
gets all the relevant information from the environment.

distutils already knows how to make .rpm files.

Eg choosing a package at the top of PyPI :-

# Download
wget 'http://ftp.livinglogic.de/xist/ll-xist-2.6.1.tar.gz'
# Unpack
tar zxvf ll-xist-2.6.1.tar.gz
cd ll-xist-2.6.1
# Make rpm
python setup.py bdist_rpm
# Make deb
cd dist/
fakeroot alien ll-xist-2.6.1-1.i386.rpm ll-xist-2.6.1-1.i386.deb
# Install with package manager
sudo dpkg -i ll-xist_2.6.1-2_i386.deb

This hasn't worked with every package I've tried it with though!

....

To make PyPI into a CPAN replacement it needs to
1) make all the submitters use distutils, and automatically test the packaging
2) host the source tar balls
3) have a system of mirrors and provide this list for download
4) provide dependency information (maybe this should be in the distutils
information in the package)
5) Provide the list of packages, descriptions, and dependencies for
download
6) Some sort of review system (maintainers, peer review, rating etc)
for each package for quality assurance

Given all that the an automatically download package x + dependencies
program would be easy.

A cross platform software repository like this would be a goal well
worth aiming for IMHO! PyPI is an excellent start though.

 

[Eric S. Johansson]

Absolutely!

For instance with Debian and Perl, dh-make-perl will turn a perl
module (tar.gz or straight from CPAN) into a .deb which you then
managage as normal. When using Debian I never use CPAN and always
dh-make-perl because I'd much rather have the package managed via
dpkg/apt-get.

That said I haven't found many perl modules that I wanted that weren't
already packaged for Debian. The same with python modules too.




No idea how dh-make-perl works. It does though!




distutils already knows how to make .rpm files.

Eg choosing a package at the top of PyPI :-

# Download
wget 'http://ftp.livinglogic.de/xist/ll-xist-2.6.1.tar.gz'
# Unpack
tar zxvf ll-xist-2.6.1.tar.gz
cd ll-xist-2.6.1
# Make rpm
python setup.py bdist_rpm
# Make deb
cd dist/
fakeroot alien ll-xist-2.6.1-1.i386.rpm ll-xist-2.6.1-1.i386.deb
# Install with package manager
sudo dpkg -i ll-xist_2.6.1-2_i386.deb

This hasn't worked with every package I've tried it with though!

...

To make PyPI into a CPAN replacement it needs to
1) make all the submitters use distutils, and automatically test the packaging
2) host the source tar balls
3) have a system of mirrors and provide this list for download
4) provide dependency information (maybe this should be in the distutils
information in the package)
5) Provide the list of packages, descriptions, and dependencies for
download
6) Some sort of review system (maintainers, peer review, rating etc)
for each package for quality assurance

Given all that the an automatically download package x + dependencies
program would be easy.

I have provided a some of this functionality in raging dormouse.
http://harvee.org/stamper/raging_dormouse/

can download from the sourceforge "click a link" maze or a regular web
site.
Verifies checksum of package.
Can handle list of download sites to try (could be expanded to mirror
list)

the build script in raging_dormouse is fairly application-specific. It
wouldn't be hard to redraft it as a configuration file driven process
while rewriting it in Python vs. its current Bash incarnation.

you're welcome to it if you find useful.

---eric

 

[John Roth]

Ok, that's a definite need. Possibly python.org could host a mechanism
to allow people to post modules. I'd do it myself, but I'm not sure
how long I'm going to have my current net connection for.


I found the standard installer script trivial to set up. If it took
hours, then you're doing something wrong. It even made it possible for
me to distribute C modules to Windows, whereas those trying to build
the module on Windows prior to me putting it into distutils never
succeeded.

Oh, the standard installer would have been fine - if all I
was distributing was Python modules. However, I have
a fairly large number of examples and quite a large amount
of HTML documentation. For some reason, the standard
installer decided that it would copy them to the install file
just fine, and then not install them. Nothing I could do would
make them install other than listing each and every file in
the install script, individually and by hand.

That's absolutely stoopid when they go into the exact
same directory they came out of. The time was spent

1) asking for help on the newsgroup. Nobody had any
helpful suggestions.

2) Hammering on the thing trying to make it work.
No joy.

3) Writing a small script to build the distutils script.
That worked, although there still seems to be an
intermittant problem on the install.

John Roth

 

[Stephen Ferg]

Lots of people *want* a PyPAN; the number of people willing to work on it is about zero.

Admitted. Or perhaps those with the skills needed for the work aren't
able to undertake such a big job without financial support. That's why
I proposed that PSF put out a special RFP for grant money to seed the
work. My whole post was an argument for that proposal. Here's the
argument in a nutshell.

**********************************************

PSF's mission is (among other things) to support Python's long-term
sucess.

Popularity is part of what it means to be successful. Popularity is
important to the Python community because it means jobs for Python
developers, and usage of Python by organizations that need it.

Python will never catch up to Perl in popularity without CPAN-like
support.

Therefore, Support for CPAN-like capabilities for Python is critical
to Python's long-term success.

Therefore, in keeping with its mission, PSF should offer grants
specifically targeted at producing a CPyAN.

***********************************************

This issue should be of critical interest to all Python developers,
because it means the difference between the existence and the
non-existence of opportunities for developers to use Python on the
job.

 

[Ville Vainio]

Steve> This issue should be of critical interest to all Python
Steve> developers, because it means the difference between the
Steve> existence and the non-existence of opportunities for
Steve> developers to use Python on the job.

If it was that important, I suppose the catalog-sig would have, say,
one non-spam message in the recent months. Perhaps people have assumed
that PyPI and distros' own package systems have solved the problem
already.

It should be noted that many significant Python packages have
non-python dependencies (wxPython <- wxWidgets). That kind of
dependency needs to be handled too, and it's the kind that is most
problematic. So in the end it basically boils down to not having
apt/yum for Windows.

 

[Stephen Ferg]

Hell, people rarely even mention Perl these days

-----------------------------------------
location number of hits for
Python Perl
-----------------------------------------

Monster.com 18 552

Washington Post
classifieds 25 353

InfoWorld 81 954
job openings

TotalJobs (British) 32 200

 

[Peter Hickman]

Hell, people rarely even mention Perl these days anyway.

People hardly speak Mandarin Chinese these days, I can go months without hearing
any being spoken. Same goes for Urdu and Spanish.

Today we learn the difference between an 'opinion' and an 'informed opinion'.

 

[A.M. Kuchling]

Admitted. Or perhaps those with the skills needed for the work aren't
able to undertake such a big job without financial support. That's why

I'm doubtful that waving money in the air will magically produce volunteers,
but who knows? I suggest you present your proposal to the PSF; I suppose
the Board would be the group that would consider it.

--amk

 

[Ville Vainio]

Steve> -----------------------------------------
Steve> location number of hits for
Steve> Python Perl
Steve> -----------------------------------------

Steve> Monster.com 18 552

.... etc.

This does not take away the fact that a few years ago, it was felt
that Python is in a bloody fight over popularity with Perl. It was
felt that Perl still had this or that over Python; this doesn't seem
to be the case anymore. Python is no longer the obscure up-and-coming
scripting language. It is used in several high-profile places, while
Perl usage seems to be mostly going down in popularity. In the Open
Source world, Python aleady seems to have a higher profile than
Perl. The popularity of Python is already big enough to not be an
impediment for the company to switch over - it may be that there is
still a need to switch over in the first place, but that's just a
refreshing challenge ;-). Hence, I wouldn't be too worried about
competing with Perl.

Decision to use a library requires some thinking already, it's not a
big additional hurdle to download it from SF/whatever and add it to
the company version control system. I wouldn't mind a CPANish system,
but the lack of it isn't the bottleneck in Python adoption.

 

[Thomas Heller]

Great point. Collectively, apt-get, yum et. al. and the underlying
package models will get far more development time, more polishing and
more real-world use than a PyPAN ever would.

Perhaps what Python needs is for Win32 to grow a working apt-get...;-)

The distutils' created bdist_wininst installers in principle have all
that is needed to build something like that. Except command line flags
for silent installation, maybe. But since a silent installer has no
need to display a user interface, it would be simple to write a
bdist_wininst_silent in pure Python.

Thomas

 

[Alex Martelli]

Python is no longer the obscure up-and-coming
scripting language.

Yep, that's what Ruby is for.


Alex

 

[Stephen Ferg]

I suggest you present your proposal to the PSF; I suppose

the Board would be the group that would consider it.

I agree.

I attempted to post it to the grants-discuss list, but was turned
down.

"Interesting discussion, but I don't think the grants-discuss list is
the right place for it."