Python needs a CPyAN

[Stephen Ferg]

I am a very satisfied user of Python and have been for number of
years. I would never willing use another language. I wish all good
things for Python, and that moves me to express some thoughts about
Python's future prospects.

I submit that the future expansion of Python usage is constrained by
Python's lack of a CPAN-like facility, and I submit that without a
CPyAN Python will never even get close to achieving the degree of
widespread usage that Perl currently enjoys.

In saying this, I am not faulting Python's "batteries included"
philosophy or the standard library. The standard library is one of
Python's greatest strengths. The standard library is Python's "jewel
in the crown" and I whole-heartedly endorse Andrew Kuchling's recent
proposal that we focus our energies on improving it.

But there are limits to a standard library. You can't put
_everything_ into a standard library, nor would you want to. There
will always be many specialized modules that shouldn't be put into a
standard library, with new modules being developed all the time.

These not-in-the-standard-library modules -- let's call them external
modules -- are the keys to a language's growth, its popularity, and
ultimately its long-term survival. The better the support for
external modules is, the more actively that they will be used and
(more importantly) re-used. If external module support is good, that
makes it easy for developers to create external modules built on top
of existing external modules, and then to create even higher-level
external modules built on those, until a very large archive of very
powerful external modules is created. With such an archive of
external modules, it is possible to do very complex, very specialized
tasks with only a few lines of code.

Note that the key to success is not the size of the archive. CPAN is
not a success because it is large. Rather, it is large because it is
a success. CPAN is only secondarily a collection of modules.
Primarily, CPAN is a set of *capabilities*: capabilities for storing,
documenting, checking for updates, searching, downloading, testing,
and installing external modules. It includes support for creating
module tests and documentation. And it is a social organization --
the CPAN testers group -- that guarantees at least a minimal level of
quality assurance for (and therefore trust in) modules in CPAN. CPAN
(the library) is large because CPAN (the external module support
mechanism) is powerful and easy to use.

It is no good saying that Python doesn't need a CPyAN because we've
got Google, or we've got SourceForge, or we've got PyPI or distutils
or the Vaults of Parnassus. Even used together, all of these tools
still fall short of the capabilities of CPAN. Only a full CPyAN will
provide the quality and ease-of-use of external modules that will
enable Python to flourish in the coming decade.

This is _not_ to say that Python will die without a CPyAN. It will
certainly survive and thrive. But it will remain a niche language.
Without a CPyAN, Python usage and the Python community can never and
will never grow to a size that even comes close to rivalling the size
of Perl.

Some might object that Python is not in a race with Perl, or that
popularity and size shouldn't be goals: that smaller and better should
be our goal. But I submit that popularity _is_ a goal. Ask any
Python programmer what his (or her) first wish is, and you will get
the reply: "I wish I could have a job in which I could spend all of my
time (or even, _some_ of my time) programming in Python." The only
answer to that wish is popularity; such jobs won't exist until Python
becomes more popular. (The second wish is an altruistic, professional
one: "I wish I could convince my organization to use Python, because
Python really is a better technology, and my organization really does
need it." And the answer to that wish, too, lies in making Python
more popular.)

To those who treasure the standard library, and who switched to Python
to escape the need to visit CPAN for even trivial things, we can say:
that won't change. The standard library will remain as strong as
ever; CPyAN will supplement but not replace it.

To those who shrink in revulsion from everything Perlish, I say: CPAN
is a great system, regardless of who invented it. It is the best
thing about Perl. In the great tradition of Python eclecticism, let's
steal it!

Because a CPyAN is key to the long-term growth of Python, creating a
CPyAN should be one of the highest -- perhaps THE highest -- of the
Python Software Foundation's priorities. Therefore, I would like to
suggest that the Python Software Foundation issue an RFP (request for
proposal) specifically for proposals to start building a CPyAN.

Building a CPyAN will be a big job, no question. But I think that for
the Python community and for the Python Software Foundation, it should
be job number one.

-- Stephen Ferg ([email protected])

REFERENCES:

a very informative post on CPAN by Sean Reifschneider
http://groups.google.com/[email protected]&rnum=6

a post by Hans Nowak
http://groups.google.com/groups?hl=...2coff=1&selm=39D0B743.A60%40hvision.nl&rnum=8

google c.l.p for "something like CPAN"
http://groups.google.com/groups?q="...=&group=comp.lang.python.*&c2coff=1&scoring=d

requirements for the catalog SIG
http://www.python.org/sigs/catalog-sig/requirements.html

 

[Michael Hoffman]

I submit that the future expansion of Python usage is constrained by
Python's lack of a CPAN-like facility, and I submit that without a
CPyAN Python will never even get close to achieving the degree of
widespread usage that Perl currently enjoys.

An excellent idea. I look forward to seeing what you accomplish in this
regard.

 

[Gerhard Haering]

[...] I submit that the future expansion of Python usage is constrained by
Python's lack of a CPAN-like facility, and I submit that without a CPyAN
Python will never even get close to achieving the degree of widespread usage
that Perl currently enjoys. [...]

I'd suggest that those who are interested join the Catalog SIG. It
could need active participants. http://python.org/sigs/catalog-sig

-- Gerhard
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBh5vxdIO4ozGCH14RAtVnAJsGtRugXsvq/odHnVB5ILOYdzoaVACdG35H
+3QMRLCqA9bS1eWcpreGznw=
=v+Jk
-----END PGP SIGNATURE-----

 

[John Roth]

How does the Python Package Index not meet this need?

I can think of one very serious difference, which has
kept me from placing a module in the PPI.

John Roth

I am a very satisfied user of Python and have been for number of
years. I would never willing use another language. I wish all good
things for Python, and that moves me to express some thoughts about
Python's future prospects.

[rest of post snipped]

 

[Ville Vainio]

Steve> Python's lack of a CPAN-like facility, and I submit that
Steve> without a CPyAN Python will never even get close to
Steve> achieving the degree of widespread usage that Perl
Steve> currently enjoys.

....

Steve> It is no good saying that Python doesn't need a CPyAN
Steve> because we've got Google, or we've got SourceForge, or
Steve> we've got PyPI or distutils or the Vaults of Parnassus.
Steve> Even used together, all of these tools still fall short of
Steve> the capabilities of CPAN. Only a full CPyAN will provide
Steve> the quality and ease-of-use of external modules that will
Steve> enable Python to flourish in the coming decade.

I think you are overselling CPAN a little bit here. It is not an
absolute requirement, and I think Python can easily surpass Perl in
popularity even without CPAN functionality. Perl popularity in general
seems to be going down, and I don't think Perl is something to worry
about anymore. Hell, people rarely even mention Perl these days
anyway.

Steve> one: "I wish I could convince my organization to use
Steve> Python, because Python really is a better technology, and
Steve> my organization really does need it." And the answer to
Steve> that wish, too, lies in making Python more popular.)

I don't think Python is too unpopular to sell to companies
anymore. All you need to do is to demonstrate technical superiority,
and that should be trivial if the other contender is Perl. Provided
that your company is not already too stuck with Perl, which is rarely
the case because with Perl you usually dealing with simple scripts.

Steve> Building a CPyAN will be a big job, no question. But I
Steve> think that for the Python community and for the Python
Steve> Software Foundation, it should be job number one.

Perhaps the priority should be to build a generic Open Source
catalogued repository retrieval system. I don't see why the Python
community should do something like this alone, because there really
isn't too many Python specific aspects in the problem anyway.

Perhaps Arch might be a good starting point...

 

[Peter Hickman]

I think you are overselling CPAN a little bit here. It is not an
absolute requirement, and I think Python can easily surpass Perl in
popularity even without CPAN functionality. Perl popularity in general
seems to be going down, and I don't think Perl is something to worry
about anymore. Hell, people rarely even mention Perl these days
anyway.

Just today we required a module that was not installed on our system.

sudo perl -MCPAN -e 'install Data:ager'

And it was installed with all it's dependencies. Having a central repository for
all your modules is a real time saver. All you need to know is the name of the
module and away you go. No searching for website addresses (I don't even know
where the module was downloaded from) and seeing which one has the correct
version and then going through the whole thing again for the dependencies.

CPAN has made it easier for Perl programmers to get work done, it would be of
benefit to Python programmers too.

Talk to the people at CPAN, they may help you to set one up. Over in Ruby land a
similar system is being set up called Gems. Once you have persuaded the module
creators to place their wares on CPyAN then it will be the 'one stop shop' for
Python module and then anyone who writes a new one would automatically assume
that they need to put theirs there too.

I don't think Python is too unpopular to sell to companies
anymore. All you need to do is to demonstrate technical superiority,
and that should be trivial if the other contender is Perl. Provided
that your company is not already too stuck with Perl, which is rarely
the case because with Perl you usually dealing with simple scripts.

You are talking out of your arse.

Perhaps the priority should be to build a generic Open Source
catalogued repository retrieval system. I don't see why the Python
community should do something like this alone, because there really
isn't too many Python specific aspects in the problem anyway.

The real trick is going to be getting the existing module authors to commit to
putting their code in your repository in the first place and then making sure
your CPyAN tools are part of the standard Python distribution.

Well worth the effort.

 

[A.M. Kuchling]

The real trick is going to be getting the existing module authors to commit to
putting their code in your repository in the first place and then making sure
your CPyAN tools are part of the standard Python distribution.

No, the trick is going to be implementing something. Remember, there were
three or four catalog strawmen before Richard Jones wrote PyPI; each of
those was announced on catalog-sig, feedback was requested, and absolutely
no comments were made publicly. Richard just went ahead and volunteered to
set things up on python.org anyway.

I expect the same results for a PyPAN; someone will post a prototype that
does 30% of what's needed, ask for reactions, and get nothing. Lots of
people *want* a PyPAN; the number of people willing to work on it is about
zero.

And it should be noted I can already install Python packages with
dependencies by running a command such as "apt-get install
python2.3-gnome2", for example.

--amk

 

[=?ISO-8859-1?Q?Michael_Str=F6der?=]

Just today we required a module that was not installed on our system.

sudo perl -MCPAN -e 'install Data:ager'

And it was installed with all it's dependencies.

I'd rather call that a security and maintenance nightmare...

Ciao, Michael.

 

[Mike Meyer]

Just today we required a module that was not installed on our system.

sudo perl -MCPAN -e 'install Data:ager'

With PyPI and distutils, we've got 90% of that already. PyPI needs to
grow a link to the tarball, and a way to turn a package name into that
link. Then you need a tool bundled with Python that gets the link from
PyPI, downloads the package, and runs "python setup.py install".

If you wanted to be brave about it, you could try scraping the
download link out of the PyPI page, then look for the link to the
tarball on that page.

<mike

 

[John Hazen]

[Stephen Ferg]
[ ... wants a CPyAN ...]

[John Roth]

How does the Python Package Index not meet this need?

I can think of one very serious difference, which has
kept me from placing a module in the PPI.

Ummm... Care to let us in on it? While your subtle hint may make sense
to those who have submitted packages for inclusion, I suspect I speak
for the many more of us who haven't. What's your objection?

-John Hazen

 

[Istvan Albert]

Ummm... Care to let us in on it?

typo, he probably meant

"I can't think of a ..."

 

[Istvan Albert]

Peter Hickman wrote:

The real trick is going to be getting the existing module authors to

The real trick is that the need for CPyAN is not an acute one.
A whole lot of tasks can be done just with the standard library.
This makes CPAN like functionality useful but not essential.
I can't think of a single instance where I would have had to
hunt down package dependencies whereas when I was using perl
that was a very significant problem where automated
installations really saved a lot of time.

Programming environments evolve like real biological systems.
They develop useful features in response to much needed
functionality and not vice versa.

Istvan.

 

[Graham Fawcett]

And it should be noted I can already install Python packages with
dependencies by running a command such as "apt-get install
python2.3-gnome2", for example.

Great point. Collectively, apt-get, yum et. al. and the underlying
package models will get far more development time, more polishing and
more real-world use than a PyPAN ever would.

Perhaps what Python needs is for Win32 to grow a working apt-get...;-)
-- G

 

[Ian Bicking]

With PyPI and distutils, we've got 90% of that already. PyPI needs to
grow a link to the tarball, and a way to turn a package name into that
link. Then you need a tool bundled with Python that gets the link from
PyPI, downloads the package, and runs "python setup.py install".

I might imagine:

python -m distutils pypi:apackage install

pypi has a download link, but it can be zipped or a tarball, and
sometimes neither, and it's not really required. And SourceForge has
its lame balancing system no matter what we do. But zip and tar can
both be handled, and SF just requires a custom link scraper. The rest
is a matter of getting people to use PyPI, and put in accurate download
links. And use trully unique names. And some other things I haven't
thought of.

 

[EP]

Building a CPyAN will be a big job, no question. But I think that for
the Python community and for the Python Software Foundation, it should
be job number one.

I must admit I found CPAN quite nice when I used Perl, and it would be terrific to have an equivalent for Python modules, at least for those of us using Windows boxes.

but... sometimes it's a matter of a bridge too far.

Afterall, the Perl guys had more than one way to do it (CPAN)...




P.S. if Python reflects original thinking, why are so many Python modules named " py "?

 

[bruno modulix]

Michael Ströder a écrit :

I'd rather call that a security and maintenance nightmare...

<aol>
We had some program (can't remember if it was spamassassin or an
antivir, I'm not the sysadmin !-) gone mad and all our customers mails
blocked for a full day because of such a wonderfull update of a perl
module...
</aol>

 

[Jonathon Blake]

Python's lack of a CPAN-like facility,

CPAN is useful, _if_ you know the name of the file you want.
When you don't know the name of the file, it isn't that useful.

I'll grant that something like CPAN would mean that Anaconda, PyAstro
and the like wouldn't disappear from the Python World.

xan

jonathon

 

[Gerhard Haering]

Michael Ströder a écrit :

<aol>
We had some program (can't remember if it was spamassassin or an
antivir, I'm not the sysadmin !-) gone mad and all our customers mails
blocked for a full day because of such a wonderfull update of a perl
module...
</aol>

Well, could have happened with a Debian [*] or *BSD, etc. update as well,
right?

-- Gerhard

[*] -testing or -unstable
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBiBR8dIO4ozGCH14RAuVDAJ4g1pz7d5BBDS0DvA6cd7Od3ly6AQCgsBG+
ApTriS1WdfxzQVmAM2VN28w=
=3Vz3
-----END PGP SIGNATURE-----

 

[Steve Holden]

Great point. Collectively, apt-get, yum et. al. and the underlying
package models will get far more development time, more polishing and
more real-world use than a PyPAN ever would.

Perhaps what Python needs is for Win32 to grow a working apt-get...;-)
-- G

And don't I remember that yum is written in ... Python?

regards
Steve

 

[Isaac To]

Gerhard> Well, could have happened with a Debian [*] or *BSD,
Gerhard> etc. update as well, right?

Yes, userss of systems with proper package management shouldn't want
to install anything from CPAN. On the other hand, there are many
people who are stuck with a braindead distribution or even OS that
doesn't have proper package management, or whose packages are severely
limited.

Regards,
Isaac.