Python does not play well with others

[Ben Finney]

Python is the only major open source project I've encountered where
there's so much hostility to bug reports.

Bear in mind that if you send a message only to this mailing list,
that's not a bug report. That's a discussion, which may be worth
having, but not one you can reasonably expect to result in a fix for a
bug.

A bug report should be sent to the bug tracker for the software
against which you're reporting a bug. Only at that point does it
become something on which you can comment about attitude toward bug
reports, because before that point the bug report doesn't exist in a
useful form.

For a while, I tried submitting long, detailed bug reports showing
where in the C code a problem lies, and pointing out ways to fix it.

That's great. I hope your attention to detail was well received.

But I don't want to take over maintenance on the SSL package; it's
too delicate.

That's fair enough. But if no-one maintains it to your satisfaction,
you do get what you pay for. If it's important to you, it should be
worth an investment of *some* kind from you, to ensure it is
maintained.

 

[John Nagle]

I'd say the functionality that John wants is the same that pretty much
everyone wants, and it's much easier to get for other languages than
for Python.

That's about it.

What's so striking is that this was a surprise. One would think
from what one reads about Python that it just works.

I've been able to pound through these problems. I finally
got M2Crypto, Python, and MySQLdb all working on a shared
hosting server. But it was quite a bit of work. I gave up on
getting the hosting provider to install current versions.

So I just spent four hours struggling with the build procedure
for M2Crypto, to get it to build with a combination of versions of OpenSSL,
M2Crypto, and SWIG that aren't the latest, but are supposed to work.
(Hint: you need to define "__i386__" or other machine if appropriate
as a command line argument to SWIG, so that OpenSSL's older conditional
includes work right.)

This is really a build system management and coordination issue.
Python has only two kinds of modules - nailed into the distribution,
or external and unsupported. Whether or not something is supported
should be independent of whether it's built by the main build file.
Components which have external dependencies, like SSL, M2Crypto,
and MySQLdb don't fit well into that model. They need to undergo
regression testing with each new Python distribution, but they don't
really need to be part of the main build. Right now, SSL is too
far inside, while the other two are too far outside. A middle
ground would help.

Otherwise, you get version hell.

John Nagle

 

[Dennis Lee Bieber]

What's so striking is that this was a surprise. One would think
from what one reads about Python that it just works.

Discounting the lag for the newest version.... Python does "just
works"... But problems with service providers not wanting to install
compliant add-ins is not a Python problem -- it's a provider problem.

One could always look for a co-location facility, supply their own
server, and configure that server for all needed pieces.

Java tends to be massively controlled by Sun -- though I will
concede that Python's non-static internals, WRT compiled extensions, is
an annoyance. It would help, greatly, if new versions of Python did not
break extensions -- Java may add features, but so far as I know, any
extensions (.jar files, say -- I don't know of any Java extension that
bypasses Java byte code and is not supplied by Sun) remain usable with
new interpreter versions. IOW, Python releases should NOT change
previously published interfaces.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Paul Boddie]

I'd say the functionality that John wants is the same that pretty much
everyone wants, and it's much easier to get for other languages than
for Python.

If the hosting provider doesn't want to install MySQLdb then it may
not be a technical issue - perhaps they just can't be bothered to
install it, possibly because there's no demand or benefit to the
bottom line in doing so. That said, I think that a large part of the
community does no-one any favours by "fashionably" requiring the most
cutting edge software, insisting on Python 2.5 in some cases, in order
for their stuff to work. Then again, getting stuff to work across
several versions (and combinations of several versions of several
packages) is an issue of release engineering, and it's no coincidence
that companies have made good business out of this kind of thing.

Perhaps Python needs a "backports" project where newer software is
backported to older Python releases. Alternatively, given the
increasing prominence of virtual server hosting, people might be
advised to consider one of those hosting plans: the benefits of being
able to install almost anything, often with modern operating system
distributions whose packages are updated, surely outweigh the
administrative demands and price differences in at least some
situations.

Paul

 

[Paul Boddie]

A bug report should be sent to the bug tracker for the software
against which you're reporting a bug. Only at that point does it
become something on which you can comment about attitude toward bug
reports, because before that point the bug report doesn't exist in a
useful form.

I think it's also worth considering various social factors when filing
bug reports. Whilst a report sent directly to the developers of a
particular module might ultimately be the way to get bugs fixed
"permanently", it is also worth investigating whether there are any
other parties affected by those bugs, whether bugs have been filed in
other places (eg. in a Linux distribution's bug tracker), and whether
there are communities who have an interest in fixing the bugs in a
more timely fashion.

I see this kind of thing every now and again with projects like KDE.
Often, in a distribution, the version of KDE won't be the absolute
latest from kde.org, and the level of interest amongst the original
developers to fix bugs in older releases is somewhat restrained.
Consequently, I would look to distributions to fix problems with their
KDE packages, especially since many of them appear to perform surgery
on KDE, often making the resulting software almost unmaintainable by
the original developers. Of course, if the bugs are genuine things
which are present "upstream" (ie. in the original software) then I'd
expect that any fixes ultimately make their way back to the original
developers, although various projects (including KDE) seem
uninterested even in merging ready-made fixes to older releases unless
it involves a major security flaw.

Paul

 

[Paul Rubin]

If the hosting provider doesn't want to install MySQLdb then it may
not be a technical issue - perhaps they just can't be bothered to
install it, possibly because there's no demand or benefit to the
bottom line in doing so.

Why should the hosting provider need to devote attention to something
like that? MySQLdb or something like it should be included with
Python, not added separately by the hosting provider. Python is
really making more demands on hosting providers than comparable
languages do. PHP hosting providers don't have to install a separate
PHP to MySQL interface gizmo as far as I know.

 

[Chris Mellon]

Why should the hosting provider need to devote attention to something
like that? MySQLdb or something like it should be included with
Python, not added separately by the hosting provider. Python is
really making more demands on hosting providers than comparable
languages do. PHP hosting providers don't have to install a separate
PHP to MySQL interface gizmo as far as I know.
--

How about because thats what you pay them for? Seriously. Do you even
think about what you're saying? Python needs to move MySQL (and only
MySQL, of course) into the core because installing packages is too
much of a burden for hosting companies?

Christ.

There are a number of languages which are primarily used for "web
development". PHP is the *only* one that ships with MySQL client
access.

Ruby doesn't (only primarily web development because of rails)
ASP, either .NET or classic, doesn't.
Java (in any form I'm aware of) doesn't.
Cold Fusion doesn't.
Perl doesn't.

Who wants to host at a company that can't install packages anyway?

And who wants to be "comparable" to PHP?

 

[Paul Rubin]

How about because thats what you pay them for? Seriously. Do you even
think about what you're saying? Python needs to move MySQL (and only
MySQL, of course) into the core because installing packages is too
much of a burden for hosting companies?

What does "batteries included" mean to you? To me, it means you don't
have to install add-ons.

If you don't like that, complain to the people who put that into the
Python sales pitch, not to me.

 

[Chris Mellon]

What does "batteries included" mean to you? To me, it means you don't
have to install add-ons.

Well, thats just stupid. I'm not sure where else to go with that.

 

[George Sakkis]

What does "batteries included" mean to you? To me, it means you don't
have to install add-ons.

So let's make a 500MB executable and add Numpy, Zope, Django, PIL,
pretty much everything actually. Even better, make CheeseShop just a
frontend to a build system that adds and updates automatically
submitted packages to the core. Problem solved ! <wink>.

George

 

[skip]

Paul> Why should the hosting provider need to devote attention to
Paul> something like that? MySQLdb or something like it should be
Paul> included with Python, ...

What about Firebird? Oracle? Sybase? Who maintains them in the core?
MySQLdb is quite stable outside the core. What would adding it to the core
do other than adding a lot of maintenance overhead to the developers' group?
How many versions of MySQL should the Python developers test with? How many
versions of Python? How many different platforms? Who decides? If *all*
you're talking about are relational database adaptors I see the obvious
candidates:

database: Sybase, Oracle, MySQL, SQLite, ODBC, Firebird
database versions: who knows? assume two per database
python versions: 2.4, 2.5, 2.6, 3.0?
platforms: windows, pick two linux flavors, solaris

Those numbers give me 6 * 2 * 4 * 4 == 192 combinations to test. Ignore
Python 2.4. That brings it down to 144. Only test on one linux variant and
skip solaris. Now you're at 72. Leave Firebird and Sybase out of the mix.
Now you're down to 48. That's still a fair sized burden. Now extend the
problem to non-database packages.

It seems to me that the PyBots project would be a better way to address the
combinatorial explosion caused by supporting multiple versions of multiple
packages on multiple platforms with multiple versions of Python:

http://www.pybots.org/

Do you have a favorite third-party package? A favorite platform? A
preferred version of your package? Set up one or more buildbots for it.
Whenever checkins occur on active Python branches occur (currently 2.5 and
SVN head, but easily extended to newer versions) your bots get built. You
can note the problem and either encourage a change to Python or to your
favorite package. This sort of solution has a couple advantages:

* Presence of a buildbot implies some level of interest by the community
at large. When the guy maintaining the HPUX-10 buildbot for the
OpenView widget set takes it down and nobody steps in to pick up the
slack, it suggests nobody cares enough about that combination anymore.

* It scales, both in people and compute resources.

Skip

 

[skip]

Paul> What does "batteries included" mean to you? To me, it means you
Paul> don't have to install add-ons.

Who decides which batteries are important to include?

Skip

 

[Paul Rubin]

So let's make a 500MB executable and add Numpy, Zope, Django, PIL,
pretty much everything actually. Even better, make CheeseShop just a
frontend to a build system that adds and updates automatically
submitted packages to the core. Problem solved ! <wink>.

Numpy should certainly be included and I think there are efforts in
that direction. There is also a movement to choose a web framework to
include and Django might be a good choice. I think the Zope
maintainers want to keep Zope separate and I think PIL has an
incompatible license. I'm not sure what you mean about making
CheeseShop a front end to a build system, but I certainly don't think
random user contributions should get added to the core automatically.
Including a module in the core should carry with it the understanding
that the module has undergone some reasonable evaluation by the core
maintainers and is maintained.

I do think the core should have more stuff than it does, so that its
functionality can be on a par with competing language distros like
J2SE and PHP. Both of those distros include database connectvity
modules and web frameworks. It could be that those other packages can
include more stuff because they have more active developer communities
and can therefore expend more resources maintaining their libraries.
But if that's the case, since the Java and PHP languages themselves
suck compared with Python, we have to ask ourselves why Python has not
been able to attract similar levels of effort and what it could be
doing differently.

 

[Paul Rubin]

database: Sybase, Oracle, MySQL, SQLite, ODBC, Firebird
database versions: who knows? assume two per database
python versions: 2.4, 2.5, 2.6, 3.0?
platforms: windows, pick two linux flavors, solaris

Those numbers give me 6 * 2 * 4 * 4 == 192 combinations to test. Ignore
Python 2.4. That brings it down to 144. Only test on one linux variant and
skip solaris. Now you're at 72. Leave Firebird and Sybase out of the mix.
Now you're down to 48. That's still a fair sized burden.

Why is it that PHP and J2SE manage deal with this problem but Python cannot?

 

[Diez B. Roggisch]

I do think the core should have more stuff than it does, so that its

functionality can be on a par with competing language distros like
J2SE and PHP. Both of those distros include database connectvity
modules and web frameworks.

This is simply not true. J2SE doesn't include a web-framework. J2EE
does, but it's a separate multi-megabyte-download.

PHP _is_ a web centered language, so to say it "includes" a webframework
sounds odd to me.

And _both_ of them don't contain DB connectivity modules! J2SE has JDBC
- which is just a bunch of interfaces. You need to download each and
every driver. Python has DB API, and because it isn't statically typed,
you just download a compliant driver/implementation.

And PHP - well, it hasn't even modules at all. Yes, there are DB
connectors for the important DBs. But for example on debian, you don't
get ORACLE, as they can't ship the needed libs. So you end up compiling
it yourself.

To conclude: given that easy_install stands up to it's promise IMHO, I'm
pretty happy with the batteries included.

Diez

 

[skip]

Paul> Numpy should certainly be included and I think there are efforts
Paul> in that direction. There is also a movement to choose a web
Paul> framework to include and Django might be a good choice.

Right off the bat you run into problems. While Numpy is popular in the
scientific computing community it's just dead weight for people without
numerical computing application needs. Besides, the Enthought folks seem to
be doing a pretty good job building a Python distribution geared toward
scientific computing. (Hint: Numpy is just the tip of that particular
iceberg.) The Pylons, web.py, Karrigell, Webware and TurboGears people
might (rightly) feel slighted if you include Django but not their
frameworks.

Where would you stop? At the boundaries of your particular application
interests?

Skip

 

[skip]

Paul> Why is it that PHP and J2SE manage deal with this problem but
Paul> Python cannot?

I can't speak authoritatively for either PHP or J2SE, but I suspect the
latter at least has some signifiant monetary support (if not outright gobs
of human resources) from Sun. PHP seems to have a more limited application
domain (web apps) from which it has expanded a bit. Can I build a PHP site
out of the box with a PostgreSQL or Oracle backend? Does J2SE have
something comparable to Numpy or scipy?

While might do the occasional bit of Python hacking for free, I still need
to put food on the table. My employer doesn't care if MySQLdb (to use one
of your examples) is delivered with Python or not. They aren't likely to
want to support me to solve your problems.

Skip

 

[Ben Finney]

Why should the hosting provider need to devote attention to
something like that? MySQLdb or something like it should be
included with Python, not added separately by the hosting provider.

"Something like it" *is* included in Python. Python 2.5 includes
SQLite in the standard library. Where do we draw the line? You want
MySQL, I want PostgreSQL, he wants Firebird, they want an interface to
something proprietary.

The standard library is for modules that are *small* and/or *generally
useful*. SQLite meets the former, and to some extent the latter, far
more than does a MySQL interface.

All the *extra* stuff is what you're paying the hosting company to
take care of in the first place.

 

[Paul Rubin]

"Something like it" *is* included in Python. Python 2.5 includes
SQLite in the standard library. Where do we draw the line? You want
MySQL, I want PostgreSQL, he wants Firebird, they want an interface to
something proprietary.

Since Python is being touted as good for web apps as a competitor to
PHP, it should offer the same db connectivity in its stdlib that PHP
offers in its. I think that includes MySQL, PostgreSQL, and Oracle,
but I dunno about Firebird.

All the *extra* stuff is what you're paying the hosting company to
take care of in the first place.

No. I'm paying the hosting company for access to a computer that's
connected to electricity and to the internet and which has a
straightforward OS, language package, web server, and db installed.
They need to know how to download a distro and type "make" to build
and install it and occasionally upgrade to a new version, but they
aren't expected to be experts on the intracacies of the software any
more than the guy who sells me potatoes at the supermarket is supposed
to be a gourmet chef. They shouldn't have to deal with dozens of
interdependent modules downloaded from different places just to
support one language. The fewer different packages they have to deal
with, the better off everyone involved is. And that means that a
package like Python should come preconfigured with all the modules
that that a typical user would expect to want to use with it.

 

[Ben Finney]

Since Python is being touted as good for web apps as a competitor to
PHP

Python is being touted as a good language for *many* purposes, not
just web applications. Python is also a "competitor" to Java, to Ruby,
to Perl, to many other languages. They all have strengths and
weaknesses.

it should offer the same db connectivity in its stdlib that PHP
offers in its.

That doesn't follow at all. Many consider the humungous function
library of PHP to be a significant downside of the system, which
criterion leaves Python ahead. More is not necessarily better.

I'm paying the hosting company for access to a computer that's
connected to electricity and to the internet and which has a
straightforward OS, language package, web server, and db installed.

In which case, there should be no problem with *you* installing
whatever software you need to use the system for what you want.

They shouldn't have to deal with dozens of interdependent modules
downloaded from different places just to support one language.

Either they are providing far more than the minimal set you describe
above, or this is entirely outside their domain. Make up your mind.

You can't claim both that the hosting company should have to maintain
a comprehensive set of functionality, *and* that they should not have
to.