Python does not play well with others

[Ben Sizer]

In the Perl, Java, PHP, and C/C++ worlds, the equivalent
functions just work. That's because, in those worlds, either the
development team for the language or the development team
for the subsystem takes responsibility for making them work.
Only Python doesn't do that.

I have much sympathy for your position. I think the problem is that
Python is quite capable in many areas, such that the people in the
community with the expertise to extend the language and libraries, are
usually the ones who've been happily using the polished features for
years and have found they need nothing more. And the ones who need
those features probably got bored of waiting for progress long ago. You
get the responses you do from years of natural selection in the
community. I think that is why many of the SIGs are stagnant, why the
standard library has so much fluff yet still lacks in key areas such as
multimedia and web development, etc.

People can say, "if you want it done, why aren't you doing it?", and is
a fair question, but it doesn't alter the fact of Python's deficiencies
in certain areas when compared with other languages.

 

[Paul Boddie]

Python is quite capable in many areas, such that the people in the
community with the expertise to extend the language and libraries, are
usually the ones who've been happily using the polished features for
years and have found they need nothing more. And the ones who need
those features probably got bored of waiting for progress long ago. You
get the responses you do from years of natural selection in the
community. I think that is why many of the SIGs are stagnant, why the
standard library has so much fluff yet still lacks in key areas such as
multimedia and web development, etc.

I think this is also a good insight into why things are as they are
within the core development section of the community, although one can
wonder what some people actively developing the language are actually
doing with it if they are satisfied with the state of some of the
standard library solutions. However, there are lots of factors which
motivate people and the proliferation (or otherwise) of solutions to
common problems: whether one develops one's own solutions as separate
projects and/or tries to push for a consensus, whether one cares about
other people using such solutions, whether one aspires to contributing
to the standard library.

Over the years, people have tended towards building their own
communities around their projects rather than attempting to engage the
wider Python community, and I think a motivation behind that has been
the intractability of improving parts of the standard library. For
example, how would one update the cgi module? Since lots of code
including various Web frameworks makes use of the cgi module, it can't
just go away or have its APIs changed radically, but this doesn't
exactly make for a pretty "out of the box" solution for Web
programming. Someone else was complaining recently about URL handling
and the required combination of urllib, cgi and urlparse (or perhaps
some other combination, depending on what one's needs are). Presented
with the problem of rationalising all this, the chosen solution (as
adopted by the Web-SIG) is, as it is frequently, to ignore the problem
and to work on other things instead.

People can say, "if you want it done, why aren't you doing it?", and is
a fair question, but it doesn't alter the fact of Python's deficiencies
in certain areas when compared with other languages.

True. It also doesn't address the issue of development priorities and
their role in defining the platform's own standards. We all know that
the Python language has issues (or "warts" as they are popularly
called), although some of the most notable ones don't seem to have been
addressed as yet in the plans for Python 3000 (eg. default argument
evaluation), but some of the most awkward aspects of using Python
involve libraries, not some deficiency of the language. However,
reforming the standard library is a topic which has arisen in all
seriousness only recently in the Python 3000 process (PEP 3108), and
the scope of the related activity is highly restricted - it's almost an
afterthought. Where the "why aren't you doing it?" part comes in
involves the observation that even if someone is motivated enough to
take something like this and to move forward in their own direction
[1], unless such initiatives are embraced by the core developers they
will remain peripheral, non-standard extensions that will at best only
become mainstream much later on.

I do wonder whether the interests of language/runtime project
developers eventually become completely aligned with the needs of such
projects, making things like "multimedia and web development" seem
irrelevant, uninteresting or tangential. This has worrying implications
for the perceived relevance of Python with regard to certain kinds of
solutions, despite the wealth of independently produced libraries
available for the language.

Paul

[1]
http://wiki.python.org/moin/CodingProjectIdeas/StandardLibrary/RestructuredStandardLibrary

 

[skip]

Paul> .... We all know that the Python language has issues (or "warts"
Paul> as they are popularly called), although some of the most notable
Paul> ones don't seem to have been addressed as yet in the plans for
Paul> Python 3000 (eg. default argument evaluation), but some of the
Paul> most awkward aspects of using Python involve libraries, not some
Paul> deficiency of the language....

I think EasyInstall might have some role to play in improving the library
status quo. Consider the cgi module. It was originally written in probably
1995 or earlier and for a long time was the only game in town.
Consequently, lots of applications used it. Times have changed though and
there's a whole new alphabet soup of web application frameworks available.
I installed Pylons the other day with "easy_install pylons". No fuss, no
muss. Over time (maybe in Python 3) modules like cgi, which arguably no
longer belong in the core can be maintained externally and installed only
when needed.

Skip

 

[John Nagle]

... I think this is also a good insight into why things are as they are
within the core development section of the community, although one can wonder
what some people actively developing the language are actually doing with it
if they are satisfied with the state of some of the standard library
solutions. However, there are lots of factors which motivate people and the
proliferation (or otherwise) of solutions to common problems: whether one
develops one's own solutions as separate projects and/or tries to push for a
consensus, whether one cares about other people using such solutions, whether
one aspires to contributing to the standard library.

Over the years, people have tended towards building their own communities
around their projects rather than attempting to engage the wider Python
community, and I think a motivation behind that has been the intractability
of improving parts of the standard library.

Yes. Working on "frameworks" is perceived as cooler than working
on libraries. Things like Ruby on Rails, Struts, Zope, and Twisted
get attention. There are papers and conferences on these things.
It's hard to get people excited about overhauling
the CGI library, or making mod_python work securely in shared-hosting
environments.

The key distinction between a framework and a library is that users
are expected to make their code fit the framework. In particular,
frameworks aren't expected to play well with each other. If you need
something from Zope and something from Twisted, you're probably not
going to be able to make it work. Libraries, on the other hand,
are expected to play well together. Which means that they have to
handle the hard cases correctly, not just the easy ones.

True. It also doesn't address the issue of development priorities and their
role in defining the platform's own standards ....
I do wonder whether the interests of language/runtime project developers
eventually become completely aligned with the needs of such projects, making
things like "multimedia and web development" seem irrelevant, uninteresting
or tangential. This has worrying implications for the perceived relevance of
Python with regard to certain kinds of solutions, despite the wealth of
independently produced libraries available for the language.

Something like that happened to the C++ standards committee.
The committee was captured by the template fanatics, and most new
standards work involves doing computation at compile time via template
expansion. That's seldom done in production code, yet most of the
standards effort is devoted to making cool template hacks work better.
Meanwhile, real problems, like doing something about memory leaks and buffer
overflows, are ignored by the C++ committee. As a result, C++ is
being displaced by Java and C#, which don't have elaborate templates but do have
memory safety.

I'm not sure how the Python development community will deal with this
problem. But what's happened in the C++ standards world has clearly
been very bad for users of the language. Learn from the mistakes there.

My main concern is with glue code to major packages. The connections
to OpenSSL, MySQL, and Apache (i.e. mod_python) all exist, but have major
weaknesses. If you're doing web applications, those are standard pieces
which need to work right. There's a tendency to treat those as abandonware
and re-implement them as event-driven systems in Twisted. Yet the
main packages aren't seriously broken. It's just that the learning curve
to make a small fix to any of them is substantial, so nobody new takes
on the problem.

John Nagle
Animats

 

[Paul Rubin]

My main concern is with glue code to major packages. The connections
to OpenSSL, MySQL, and Apache (i.e. mod_python) all exist, but have major
weaknesses. If you're doing web applications, those are standard pieces
which need to work right. There's a tendency to treat those as abandonware
and re-implement them as event-driven systems in Twisted. Yet the
main packages aren't seriously broken. It's just that the learning curve
to make a small fix to any of them is substantial, so nobody new takes
on the problem.

There is an attitude widespread in the Python community, maybe
inherited from Extreme Programming, that when writing some subsystem
it's enough to implement just enough functionality to satisfy the
application's immediate needs. Then if the application needs more
functionality from the subsystem later, go back and add it. That's
fairly reasonable when the application and subsystem are being done by
the same people and there's not such a learning curve. But it's
unwise for libraries. For all Java's ugliness, when the Java library
developers want to implement some protocol like SSL, they take the
specification for the protocol and they implement it completely.
Python's library modules are very often missing stuff, as you've
pointed out.

In the specific examples of OpenSSL, MySQL, and Apache, the modules
involved aren't even part of the Python stdlib, the way JSSE and JDBC
are part of the Java stdlib (I guess that doesn't apply to Tomcat
though).

 

[skip]

John> My main concern is with glue code to major packages. The
John> connections to OpenSSL, MySQL, and Apache (i.e. mod_python) all
John> exist, but have major weaknesses.

I have no SSL experience or direct mod_python experience (I do use Myghty
effectively), but I have used MySQLdb extensively in multi-threaded
environments. There is a Python database API (PEP 249). As far as I know,
MySQLdb implements this API well. I have personally never had problems.
Andy Dustman, the author of the package is quite responsive to requests
raised in the mysql-python forums on SourceForge
(http://sourceforge.net/forum/?group_id=22307). If you have problems with
MySQLdb, bring them up there. I'm sure Andy will respond.

Skip

 

[sjdevnull]

My main concern is with glue code to major packages. The connections
to OpenSSL, MySQL, and Apache (i.e. mod_python) all exist, but have major
weaknesses. If you're doing web applications, those are standard pieces
which need to work right. There's a tendency to treat those as abandonware
and re-implement them as event-driven systems in Twisted.

In the real world I've worked on Python web apps at my last 3 jobs, and
they've all used mod_python and either MySQL or Postgres. I haven't
had a need to do anything with OpenSSL from Python; all that takes
place in the Apache server (possibly with some mod_rewrite rules to
ensure that certain pages are only hit from https and so forth).

I think the impression that everyone's using some sexy new framework
like Django or TurboGears or Pylons on Twisted or lighttpd/wsgi is
pretty misleading, especially when it comes to companies with a
decent-size existing codebase (we're into the "fairly large" range with
about 325,000 lines of Python now--most is independent of the
architecture, but it'd still be an administratively sizeable
change)--the place I'm at currently is considering moving to
apache+wsgi, but that's a smaller change (and one that's been under
consideration for months now).

 

[Paul Rubin]

I haven't had a need to do anything with OpenSSL from Python; all
that takes place in the Apache server

That's a reasonable approach for a typical server-side web
application, but there are other types of Python apps that can also
want to use SSL. It's a pain to have to deal with OpenSSL at all, and
I hope that TLSLite (a very incomplete SSL implementation written in
pure Python) one day becomes complete and does all the stuff that JSSE
does.

(possibly with some mod_rewrite rules to ensure that certain pages
are only hit from https and so forth).

Normally you'd do that SSLRequireSSL or SSLRequire, if that matters.

 

[John Nagle]

That's a reasonable approach for a typical server-side web
application, but there are other types of Python apps that can also
want to use SSL. It's a pain to have to deal with OpenSSL at all, and
I hope that TLSLite (a very incomplete SSL implementation written in
pure Python) one day becomes complete and does all the stuff that JSSE
does.

That's the problem. We now have four SSL implementations for
Python, none of which let you do all the things OpenSSL can do.

The fact that people keep writing new ones, instead of fixing the
existing ones, indicates a problem with the development process.

Can you build the built-in SSL package without building all of
Python? That in itself would be useful. Then the thing could
be worked on without too much pain. The built-in SSL package
basically works, although it doesn't check anything. It just
doesn't get at enough OpenSSL functionality.

I think the right way out of this mess is to 1) break
out the SSL module so it can be built separately by the
usual "python setup.py build" process, 2) add more
access functions to the built-in SSL C glue module, 3) bring over
such Python parts of M2Crypto as are useful and make them
work with the built-in SSL module, and 4) get rid of SWIG.

If somebody will do 1), I'll take a look at 2).

John Nagle

 

[Paul Rubin]

That's the problem. We now have four SSL implementations for
Python, none of which let you do all the things OpenSSL can do.

I'm aware of some OpenSSL wrappers plus TLSlite. Am I missing a
couple more? I guess maybe the OpenSSL wrappers also work on GnuTLS.

Can you build the built-in SSL package without building all of
Python?

I thought there was no built-in SSL package-- it just calls OpenSSL.

 

[Jarek Zgoda]

Harry George napisa³(a):

You experience isn't shared by everyone. Some of us find Python the
most functional and portable of the candidates you mention.

The point on MySQLdb is rather common, many people who used it with
SQLObject will confirm. Fortunately for MySQLdb, the MySQL on its own is
much worse. Fortunately for us, the MySQLdb library is not a part of
Python distribution, so Python can not be blamed for its deficiencies.

 

[John Nagle]

I'm aware of some OpenSSL wrappers plus TLSlite. Am I missing a
couple more? I guess maybe the OpenSSL wrappers also work on GnuTLS.




I thought there was no built-in SSL package-- it just calls OpenSSL.

"import SSL" brings in a .dll or .so object called "_ssl.pyd".
This contains C glue code which connects Python to the
C-based OpenSSL library. That's the component which needs work.
Its source code is at

http://svn.python.org/view/python/trunk/Modules/_ssl.c

Part of the problem is that the build of _ssl.pyd
is too integrated into the main Python build to work on easily.
You can't just build and test a new SSL module and give it to others
to test. You have to rebuild the whole Python system, and
people who want to test the modified module need to run a
custom Python build.

SSL is somewhat integrated into the socket module internally,
although from the code I don't see that this is absolutely
necessary. The build machinery for SSL is in

http://svn.python.org/view/python/trunk/setup.py

After looking at that for a while, and the directory
structure, it looks like the Python directory layout and
build system makes the assumption that anything in the
standard distribution is built as part of the basic
Python build. The module itself could be separated from
the trunk build; Python will run without it. It's the
build process and directory structure which ties it up.

As far as I can tell, there's no concept in the
Python build system of a module that's built separately
but ships with the standard Python distribution. Is
that correct?

So now I understand why the standard SSL module doesn't
get fixed, and why there are three or four alternative
re-implementations of it. It's easier to replace than
to repair.

John Nagle

 

[Skip Montanaro]

Andy Dustman, the author of the package is quite responsive to requests
raised in the mysql-python forums on SourceForge
(http://sourceforge.net/forum/?group_id=22307). If you have problems with
MySQLdb, bring them up there. I'm sure Andy will respond.

I apologize in advance for beating this dead horse... This morning a
possible bug was reported in MySQLdb. Andy determined it was indeed a bug,
had a fix checked into the source repository in a couple hours and reported
that it would be in the next 1.2 beta release.

Skip

 

[Bruno Desthuilliers]

John Nagle a écrit :

(snip)

My main concern is with glue code to major packages. The connections
to OpenSSL, MySQL, and Apache (i.e. mod_python) all exist, but have major
weaknesses.

Neither MySQLdb nor mod_python are part of the Python's standard lib AFAIK.

If you're doing web applications,

I do.

those are standard pieces
which need to work right.

I avoid using MySQL - SQLite does a better job as a lighweight
SQL-compliant no-server solution, and PostgreSQL is years ahead of MySQL
when it comes to serious, rock-solid transactional RDBMS. But I had no
problem with the MySQLdb package so far. I also tend to favor
Apache-independant deployment solutions, so while I had some fun with
mod_python, I failed to clearly understand how broken it is. And I did
not have to worry about the ssl support in Python so far. FWIW, I had do
some LDAP stuff with both PHP and Python, and I would certainly not
advocate PHP's LDAP support.

There's a tendency to treat those as abandonware
and re-implement them as event-driven systems in Twisted.

While Twisted seems an interesting project, it's usually not the first
mentioned when talking about web development with Python.

Yet the
main packages aren't seriously broken. It's just that the learning curve
to make a small fix to any of them is substantial, so nobody new takes
on the problem.

If you feel you need it, then it's up to you.

 

[John Nagle]

Just a followup. I'm still trying to get Python, MySQL, MySQLdb,
M2Crypto, and maybe mod_python to work on a shared hosting server.
This is amazingly difficult, because of all the version incompatibility issues.
Python, MySQL, MySQLdb, SWIG, OpenSSL, and gcc all have to have the
right versions, and those aren't necessarily the versions shipping
in some major Linux distros.

From EZpublishing tech support:
"To make matters worse, we can't run the newer version of openSSL on any of our
current servers. It's a standard redhat rpm and redhat hasn't upgraded it. We
can install it on dedicated servers, but it is too risky for other users to be
installed on a shared environment."

From Aplus.net tech support:
"No, db connectivity with Python is not supported. Modules for Python
can't be installed by request on shared hosting."

And these are companies that say they support Python.

Python still isn't ready for prime time in the web hosting world.

John Nagle

 

[Ben Finney]

Just a followup. I'm still trying to get Python, MySQL, MySQLdb,
M2Crypto, and maybe mod_python to work on a shared hosting server.

From your description in the rest of the message, it seems that it's

the *hosting providers* who are unable to do this, not you.

And these are companies that say they support Python.

Python still isn't ready for prime time in the web hosting world.

That doesn't follow. It's just as valid to say that the web hosting
providers (that you've interacted with so far) aren't ready to support
the Python functionality you want.

 

[Paul Rubin]

That doesn't follow. It's just as valid to say that the web hosting
providers (that you've interacted with so far) aren't ready to support
the Python functionality you want.

I'd say the functionality that John wants is the same that pretty much
everyone wants, and it's much easier to get for other languages than
for Python.

 

[msoulier]

The major complaint I have about Python is that the packages
which connect it to other software components all seem to have
serious problems. As long as you don't need to talk to anything
outside the Python world, you're fine.

As one who has attempted to develop a wxPython application with an
sqlite backend, developing on both Linux and Windows, I would conclude
that the problem is version skew between third-party components and
the core.

Python 1.5.2 touted a simple core language that made Python easy to
learn. I would argue that the core is no longer simple, and it is very
easy to write modules that are not backwards-compatible. This results
in third-party components that you may need but are either not updated
against the new version of Python that you're using, or are not
compatible with the older version of Python that you shipped with.

I found it quite a frustrating experience. Love the language but I'd
rather it not keep reinventing itself quite so quickly, permitting
maintainers to keep up, especially when said maintenance is a
volunteer effort.

Mike

 

[msoulier]

Python is the only major open source project I've encountered where
there's so much hostility to bug reports.

Try telling the Perl community that their debugger is broken. That
didn't go well. ;-)

Mike

 

[skip]

John> From Aplus.net tech support:
John> "No, db connectivity with Python is not supported. Modules for
John> Python can't be installed by request on shared hosting."

John> And these are companies that say they support Python.

John> Python still isn't ready for prime time in the web hosting world.

Why do you lay the blame at the feet of the Python developers? Find another
hosting service.

Skip