Python does not play well with others

[Paul Rubin]

Paul> Yeah well, the Wxpython, PyQt, PyGTK etc. people may feel slighted
Paul> that Tkinter got included and their stuff didn't, and the Eric,
Paul> Eclipse, Komodo etc. people may feel slighted that IDLE got
Paul> included, but that doesn't stop Tkinter and IDLE from being useful
Paul> and worth shipping in Python.

They were both pretty much the only games in town when they were included
with Python. If they were developed in today's environment I doubt they
would be included.

That would diminish Python's popularity since being able to write GUI
apps without having to download additional crap is a Python selling
point, and IDLE is a considerably more pleasant Python editor than
Notepad is.

Paul> Basic competitive analysis. People ask here all the time "I'm
Paul> trying to write application XYZ, should I use language L or should
Paul> I use Python" (L is usually Java or PHP but can be other things).

So every time some lame ass PHP refugee gripes about something they think
Python is missing which PHP includes we should scan the horizon for all the
candidates and toss them in the next release?

No, just the functions that are requested frequently, like database
adapters.

 

[Paul Rubin]

Really? Then why does the MySQL AB download site at
http://dev.mysql.com/downloads/connector/php/ list both a
"Connector/PHP" AND a "native MySQL driver" for PHP? Along with ODBC,
.NET, and two variations of Java connectors.

Beats me. It does look like PHP5 has stopped bundling the MySQL
client library for licensing reasons:

http://www.php.net/manual/en/faq.databases.php#faq.databases.mysql.php5

It does say "there will always be MySQL support in PHP of one kind or
another" but I'm not sure what that means in this context.

On the other hand if MySQL itself supplies the client library, that's
almost as good. Remember that this isn't purely about minimizing the
number of downloads. It's also about minimizing the number of places
to download from, i.e. the number of different development entities
one has to deal with.

So if you're using Python with MySQL, and you can get everything you
need from python.org and mysql.com, then even if it takes multiple
downloads it's better than having to get additional stuff from random
third party sites, especially when Python's own docs haven't said
where to get the stuff.

Therefore, if someone can persuade mysql.com to offer downloadable
MySQL client drivers for Python at the same place where they supply
the PHP drivers, that's better than the current situation where the
drivers come from some totally unrelated party.

 

[Paul Rubin]

Python should only incorporate functionality in order to offer a
coherent experience (where the omission of functionality would
otherwise lead to a flawed experience). For example, having support
for SSL in the socket module offers a coherent experience because it
means that urllib and related modules can offer to support SSL-related
URLs out of the box.

But they can't, because the built-in socket module SSL interface
doesn't check certificates, causing total security failure if someone
spoofs the remote site. The built-in SSL functionality is broken and
users have to resort to external packages.

Then you have to ask why the stdlib includes anything like urllib in
the first place, under this "coherent experience" concept (I interpret
that as some kind of minimalist philosophy). Can't users have a
coherent experience if the stdlib doesn't include urllib? My own
answer is the one that I thought that the developers had settled on
years ago, namely "batteries included", i.e. ship a rich set of
libraries that provide a wide variety of useful functions, i.e. the
doctrine of minimalism has been explicitly rejected. We then get the
question of whether to include any specific function and that's where
comparisons with other languages come in.

Or a virtual private server.

Sure, that would count as "something similar".

If a hosting provider claims Python and MySQL support, then I'd hope
that they have worked out that the MySQLdb package glues the two
together.

I'd expect so too. The issue is there aren't very many of those
companies and that appears partly because of the hassle involved. So
minimizing the hassle suggests itself as a road to wider acceptance.

Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.

Perhaps better information is necessary for those hosting
companies who haven't worked such things out: you'd have a metapackage
for stuff like this in certain distributions.

That could help.

So, for the less forward-thinking providers a metapackage would be the
solution, then?

I'm not sure what you mean by metapackage but in general the goal is
to minimize the number of places that the hosting provider (or OS
distro maintainer, or whatever)

 

[Paul Rubin]

Paul> Why would I expect your employer to solve my problems anyway, even
Paul> if they relate to some module that you actually use?

Your reasoning seems to be that Python should contain the functional union
of everything at least in Java and PHP if not Perl, Ruby and Tcl as well.

I wouldn't go quite that far. I think there are specific application
areas, such as web server apps, where the Python advocates here on
clpy pitch Python relentlessly against those other languages. Given
that context, Python's stdlib should try to match the libraries of
those other languages in those areas. There are other areas where
Python doesn't get pitched as hard and the other languages have
acknowledged advantages. So it's ok if Python's stdlib gets less
attention in those areas.

Free software or not, each person who contributes to the development
of the Python core has to put food on the table. As an example, I
suspect most core Java development is funded directly by Sun who
pays its employees to develop libraries, JIT compilers and the like.
While my employer pays me to program in Python I'm not paid to
develop core Python code. What little I do is done on my own time.

Again I don't understand what your employer or your activities have to
do with any of this. If you don't want to spend your time developing
Python then don't. You didn't write the interpreter or the tkinter
library or the itertools library and yet there they are. There's
ongoing development that will continue with or without you or me, and
for that matter with or without funding from Sun.

If you want to turn the Python distribution into a kitchen sink,
make the argument on python-dev and be prepared to shoulder your
share of the burden should your arguments sway the group as a whole.

We've had this conversation before and I continue to think your
reasoning above is invalid. I'm not a Python developer, I'm just a
user, and my volunteer coding priorities are elsewhere, as I've
explained before. Python's developers and advocates have a declared
goal of reaching as many users as they can, and as a user I don't mind
offering suggestions about how to do that, but my responsibilities
don't go any further.

 

[Diez B. Roggisch]

Well, that's nowhere near as big a deal as having yet another set of
vendors or maintainer to deal with. Minimizing configuration is nice,
but getting your software from as few different sources as possible is
also a big win.

Ehm... .it _is_ another set of vendors and maintainers. It's just that
they are standard jdbc-compliant. As the python modules are DB API 2.0
compliant - hopefully.

A difference might be that most (not all) of the jdbc-drivers are
java-only, and not have any c-dependencies. Well - nice to have, but
simple inclusion of existing drivers doesn't remove that. Only a
complete rewrite in python would.

Diez

 

[Kirk Sluder]

Quite. I imagine that most GNU/Linux distributions (and various BSDs)
provide at least some version of MySQLdb as a package.

Bingo, I've rarely installed python from python.org, or other
libraries from sourceforge, etc., etc.. Usually I've installed
BSD-style ports, debian-style packages, or RedHat-style RPMs.

 

[Paul Boddie]

Bingo, I've rarely installed python from python.org, or other
libraries from sourceforge, etc., etc.. Usually I've installed
BSD-style ports, debian-style packages, or RedHat-style RPMs.

And while Python eggs may be useful for people managing additional
software as some unprivileged user, hosting providers (and virtual
private server administrators) will want packages that fit in with the
rest of the software being managed in the hosting environment.

Paul

 

[Paul Boddie]

But they can't, because the built-in socket module SSL interface
doesn't check certificates, causing total security failure if someone
spoofs the remote site. The built-in SSL functionality is broken and
users have to resort to external packages.

I was really advocating improvements to the built-in SSL support,
anyway, which was also what the complainant was suggesting before
people started asking him mistakenly why he thought that Python was
weakened by some third party packages (PyOpenSSL, M2Crypto). The
choice here involves either improving the built-in support or
unbundling SSL-based communications altogether. The former option
obviously demands a certain amount of engineering, and then one might
ask why there isn't a convenient framework for plugging in other
flavours of sockets, for example, although there arguably aren't any
as generally important as secure sockets. The latter option needs
everyone to think about how you'd plug such stuff back into Python in
a nice enough way, and then to get people to work on the right
projects to provide something which does the job.

Then you have to ask why the stdlib includes anything like urllib in
the first place, under this "coherent experience" concept (I interpret
that as some kind of minimalist philosophy). Can't users have a
coherent experience if the stdlib doesn't include urllib? My own
answer is the one that I thought that the developers had settled on
years ago, namely "batteries included", i.e. ship a rich set of
libraries that provide a wide variety of useful functions, i.e. the
doctrine of minimalism has been explicitly rejected.

Really, we have to ask whether including the batteries would save
people a lot of work, not just in whether the end-user has to find out
about an external package and then download something, but whether the
logistics around developing the code, integrating it with Python, and
maintaining it would be easier if people just included the stuff with
Python. Here, you need some kind of consensus that feature X needs
supporting and there's an approved way of supporting it which a group
of people would be happy to maintain. Would it benefit the Python
community (including core developers) more if Python shipped with SSL
support out of the box, and would the cost of doing so ultimately be
less than just pointing people at third party libraries and dealing
with their problems?

We then get the question of whether to include any specific function and that's where
comparisons with other languages come in.

Would it benefit the Python community more if Python shipped with
MySQL support out of the box? Is it likely that a user suddenly finds
him/herself needing to connect to a MySQL database? Is it more likely
that the user might suddenly find him/herself needing to download from
a secure site, particularly if some tool (eg. setuptools) suddenly
stumbles across an https URL. Some database systems have a choice of
drivers/libraries/modules (PostgreSQL has quite a few, for example):
choosing a module for standard library inclusion and integrating the
development can be too high a barrier for such questions of inclusion
to be resolved trivially.

[...]

I'm not sure what you mean by metapackage but in general the goal is
to minimize the number of places that the hosting provider (or OS
distro maintainer, or whatever)

On various distributions you get packages which don't actually contain
anything, but which indicate a suite of packages which are to be
installed. So, if you install the ubuntu-desktop package on Ubuntu
systems, you get the GNOME desktop environment and all the
dependencies (and a bunch of other stuff). Perhaps there should be a
python-mysql-hosting package for such providers.

Paul

 

[Jorge Godoy]

And while Python eggs may be useful for people managing additional
software as some unprivileged user, hosting providers (and virtual
private server administrators) will want packages that fit in with the
rest of the software being managed in the hosting environment.

And why eggs wouldn't satisfy them? Eggs can be installed globally as well,
making the package available to every client of this hosting server (if they
mount their libs from a unique NFS server then it would automatically be
available for all of their servers).

 

[Kirk Sluder]

Would it benefit the Python community more if Python shipped with
MySQL support out of the box? Is it likely that a user suddenly finds
him/herself needing to connect to a MySQL database?

The other problem is that it chains the python release schedule to
that of MySQL AB (and postgresql, and whatever.) One of the key
advantages to independent modules is that I don't need to update my
entire python system every time my database vendor releases a new
library, nor do I have to accept lag time as the module is tested
and rolled into the base distribution. Database client libraries
are much more of a moving target than core language features.

I find it interesting that PHP struggled with these issues and
decided to abandon embedded MySQL support partly because they
couldn't maintain their own parallel versions of the client
libraries.

http://us2.php.net/manual/en/faq.databases.php#faq.databases.mysql.ph
p5

Among the other problems faced by both PHP and Python in bundling
MySQL support is that they can't legally do it without adopting the
GPL.

Which leaves me wondering why the python core should adopt a feature
that was abandoned by PHP, and was never highly recommended or used?

 

[Ramon Diaz-Uriarte]

I find Paul Rubin's arguments compelling and convincing. As just a
Python user (i.e., someone who has contributed nothing) let me add a
few comments along the same lines.

I wouldn't go quite that far. I think there are specific application
areas, such as web server apps, where the Python advocates here on
clpy pitch Python relentlessly against those other languages. Given
that context, Python's stdlib should try to match the libraries of
those other languages in those areas. There are other areas where
Python doesn't get pitched as hard and the other languages have
acknowledged advantages. So it's ok if Python's stdlib gets less
attention in those areas.

In fact, it is quite frustrating to operate under the impression that
"Python is good for X, Y, Z, ...." and then realize "ooops, it is
becoming a pain in the ass to do X, whereas language L does X just
fine". It seems to me that Python advocates sometimes (often?) get
carried away. DB "sure, no problem"; web-frameworks "who needs Rails,
we have (lots of) frameworks that do it"; functional programming
"Python can do all the functional programming anyone in his/her mind
should ever try to do"; etc.

Compare this to the, in my opinion, equanimous, fair, "advertisement"
one finds in the Erlang page or the recognition by schemers that
scheme is not the only game in town.

We've had this conversation before and I continue to think your
reasoning above is invalid. I'm not a Python developer, I'm just a
user, and my volunteer coding priorities are elsewhere, as I've
explained before. Python's developers and advocates have a declared
goal of reaching as many users as they can, and as a user I don't mind
offering suggestions about how to do that, but my responsibilities
don't go any further.


R.

--
Ramon Diaz-Uriarte
Statistical Computing Team
Structural Biology and Biocomputing Programme
Spanish National Cancer Centre (CNIO)
http://ligarto.org/rdiaz

 

[Paul Boddie]

And why eggs wouldn't satisfy them? Eggs can be installed globally as well,
making the package available to every client of this hosting server (if they
mount their libs from a unique NFS server then it would automatically be
available for all of their servers).

Because Python is just another thing to support for various hosting
providers. One of the problems people supposedly have when persuading
such companies to add or update packages is, I imagine, a lack of
familiarity those companies have with Python technologies. Asking them
to use a technology-specific packaging system might be a bit much if
they aren't even familiar with (or interesting in knowing more about)
the packages required to satisfy their customers' basic needs.

Moreover, there's a lot of software in the average GNU/Linux or BSD
distribution, but if you had to "break out" into technology-specific
package/dependency managers to install some of the more heterogeneous
packages, it would be a nightmare: perhaps installing the
documentation would have you visit CPAN for some Perl scripts before
throwing you into the TeX package manager to get some missing TeX
libraries; then you might be off into setuptools for the Python
scripting extensions, possibly via equivalent tools for other
scripting extensions; finally, you'd be off via any other system
thought essential to manage software from a particular parochial
technological viewpoint. I've had to install software like this, and
it takes huge amounts of time for no good reason if you can use system
packages instead.

Paul

 

[Colin J. Williams]

Numpy should certainly be included and I think there are efforts in
that direction.

As I understand it, the effort is directed towards elaborating the
current array module so that it handles multi-dimensional array.

This would be a good step but numpy os much more and is of minority
interest and so should probably not, in my opinion, be included in the
Python core.

Colin W.
There is also a movement to choose a web framework to

 

[John Nagle]

I was really advocating improvements to the built-in SSL support,
anyway, which was also what the complainant was suggesting before
people started asking him mistakenly why he thought that Python was
weakened by some third party packages (PyOpenSSL, M2Crypto).

The real problems are with integration of modules that
aren't written in Python. Python-only modules don't have the
version compatibility problems which C modules do. Loading
a Python-only module from an external source is usually not
a big deal. Building a C module, especially one with
dependencies on other components, can be a big deal.
So, focus on problems with modules which have C
components.

John Nagle

 

[Paul Boddie]

The real problems are with integration of modules that
aren't written in Python. Python-only modules don't have the
version compatibility problems which C modules do. Loading
a Python-only module from an external source is usually not
a big deal. Building a C module, especially one with
dependencies on other components, can be a big deal.
So, focus on problems with modules which have C
components.

I think that the integration of certain "popular" extensions into the
Python distribution has been one of the tools employed to mitigate the
version compatibility situation: if the popular stuff gets released
with Python, perhaps fewer people will complain about "C modules"
whose developers/maintainers haven't updated them for the latest
release. I'm not really convinced that this makes for a sustainable
approach, however, since this just becomes a process of accretion
where the core development community must get larger and become very
well-coordinated in order to manage the dependencies within the
software, as well as tracking external dependencies.

Over the years there have been suggestions which might have made the
compatibility situation a bit better: a "Python in a tie" release was
an objective of the seemingly dormant Python Business Foundation;
Linux Standard Base inclusion of Python has been proposed and
discussed. I can't help feeling that without improvements in such
other areas, things like "C module" availability for different Python
versions (and for different versions of related libraries) is mainly
an issue of doing the almost thankless hard work of backporting,
testing and managing different build configurations. Still, there may
be demand for a backports community in all this.

Paul

 

[Graham Dumpleton]

Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.

To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.

First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process. These
distinct sub interpreters can be linked to different parts of the URL
namespace. This means that it is possible to host more than one
mod_python application where each executes within in their own
distinct sub interpreter. The outcome of this is that each application
can have their own sys.path, own os.environ, own sets of modules and
potentially with different versions of some module.

Maintaining separation using sub interpreters eliminates the bulk of
problems with applications interfering which each other at least
within a process. Some problems can still arise though where third
party extension modules for Python aren't written so as to be usable
from multiple sub interpreters at the same time however. This is not a
failing of mod_python though, but a failing of the module writers.

The main area where interference can occur is where applications needs
to write to the file system. This is because all code will be
executing as the user that Apache runs as. Thus, distinct applications
could overwrite each others data within the file system. On one level
this just means that applications need to be configured to always use
their own part of the file system. For example, if using the support
in mod_python for sessions, the distinct applications should perhaps
use separate session databases rather than use the same common
database. Ultimately though, a rogue application could write where
ever it wants to, but from what I know (and could be wrong), this
isn't different to other languages systems within Apache such as PHP
and mod_perl.

Another possibility for interference is where an application simply
does something bad like get stuck in a tight loop or consume lots of
memory. Such an event can possibly interfere with other applications,
even across language boundaries, however, how bad the impact will be
depend on what MPM is used by Apache.

If "prefork" MPM is used, then the request being handled by that
application is the only thing which would be running within that child
process at that particular time. Thus, if it stops the functioning of
just that one process it doesn't matter as Apache will just farm
requests off to other child processes. If that initial child process
crashes because of the problem, then again it doesn't matter as Apache
will just create another child process to replace it and will
otherwise keep running.

If the "worker" MPM is used the impact can be greater as there could
be other requests being handled concurrently within the same child
process and the performance of those requests may be hindered. If the
worst happens and the child process crashes, only other requests being
handled within that specific child process would be affected, those in
other child processes would again continue unaffected as would Apache
as a whole.

The worst case is the "winnt" MPM (ie., Windows boxes). This is
because there is only one Apache process and thus a rogue application
can affect the whole Apache web server.

It should be highlighted though that this class of problem is not
unique to Python or mod_python as you could get rogue code in a PHP
page or mod_perl application just as easily.

What it all really comes down to is that the only features that are
really missing are the ability for distinct applications to run as
distinct users, or for applications to run inside of some sort of
chroot environment. Some aspects of this are addressed by FCGI and
SCGI, but again lack of this feature within mod_python itself is not
unique to it and that as far as I know is also going to be an issue
for other language systems for Apache such as PHP or mod_perl.

Having said all that, perhaps those who are complaining about lack of
support for specific features in mod_python can now clarify what
actually you are talking about. At the moment the brief comments being
made seem to possibly cover some things that mod_python can already do
but may not be obvious.

Graham

 

[Paul Boddie]

Having said all that, perhaps those who are complaining about lack of
support for specific features in mod_python can now clarify what
actually you are talking about. At the moment the brief comments being
made seem to possibly cover some things that mod_python can already do
but may not be obvious.

I had a more complete response to this before Google Groups and my
browser went into "stupid mode" together, but I'd like to say that I
don't have any complaints about mod_python, but I know that there were
people who claimed [1] that PHP had its "safe mode" [2] which gave
isolation to applications belonging to different users in the same
Apache instance, asserting that mod_python not having that particular
feature prevented wider adoption of Python in the hosting business.

It's good to see a clarification of mod_python and its position in
relation to such issues, though. I don't really share the concerns
mentioned, and wouldn't want to be using or providing services of the
kind previously discussed, anyway. Instead, I'd want to deploy
applications using other techniques such as virtualisation, rather
than throwing customers into the same Web server and, apparently in
the case of PHP, relying on hearsay about whether they can interfere
with each other. I note that the creator of mod_python seems to have a
similar perspective on virtualisation, given the nature of his hosting
company's services.

Paul

[1] http://groups.google.com/group/comp.lang.python/msg/
469dd47f5c1ad521
[2] http://no.php.net/features.safe-mode

 

[John Nagle]

To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.

First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process.

Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.

FastCGI, though, can get all the CPUs going. It takes more
memory, though, since each instance has a full copy of Python
and all the libraries in use.

(FastCGI is a straightforward transaction processing engine.
Each transaction program is launched in a separate process, and,
once done with one transaction, can be used to do another one
without reloading. When things are slow, the extra transaction processes
are told to exit; when load picks up, more of them are forked.
Security is comparable to CGI.)

John Nagle

 

[Graham Dumpleton]

Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.

That is not true if 'prefork' MPM is used for Apache which is how most
people seem to run it. This is because each Apache child process only
run one request at a time and so there isn't normally going to be any
contention on the GIL at all. The only case where there would be
contention in this arrangement is if the request handlers within
Apache had spawned off distinct threads themselves to do stuff. Even
then, in this arrangement the main request handler is usually not
doing anything and is just waiting for the created thread to finish
what it was doing. Thus if only one thread was spawned to do some work
or a blocking operation to allow the main thread to timeout, then
again there isn't really any contention as only one thread is actually
doing anything. If you are trying to embed very intensive operations
with threads within Apache then I would suggest it is not really the
best design you could use anyway as such things would be much better
farmed off to a long running backend process using XML-RPC or some
other interprocess communication mechanism.

If one is using the "worker" MPM then yes there will be some
contention if multiple requests are being handled by mod_python at the
same time within the same Apache child process. The downside of this
is lessened however by the fact that there are still multiple Apache
child processes and Apache will spread requests across all the Apache
child processes, thus the amount that may be running concurrently
within any one process is less.

The basic problem of GIL contention here is no different to a single
Python backend process which is handling everything behind Apache. In
some respects the Apache approach actually works better as there are
multiple processes spreading the load. Your comment on the GIL is
therefore partly unjustified in that respect for Apache and
mod_python. Your statement in some respect still stands for Python
itself when run as a single process, but you linked it to mod_python
and Apache which lessens the impact through its architecture of using
multiple child processes.

Finally we have 'winnt' MPM, again, because this is all in the one
process you will have GIL contention in a much more substantial
manner. However, I'd suggest that most wouldn't choose Apache on
Windows as a major deployment platform.

FastCGI, though, can get all the CPUs going. It takes more
memory, though, since each instance has a full copy of Python
and all the libraries in use.

How is that any different to Apache child processes. Each Apache child
process has a full copy of Python and the libraries in use. Each
Apache child process can be making use of different CPUs. Further,
static file requests, plus other requests against PHP, mod_perl etc
can when mod_python is also running be on separate CPUs within the
same child process when 'worker' MPM is being used. Thus you haven't
lost all forms or parallelism that may be possible, it is only within
the mod_python world that there will be some GIL contention and only
with 'worker' and 'winnt' MPMs, not 'prefork'. It isn't going to lock
out non Python stuff from making use of additional CPUs.

(FastCGI is a straightforward transaction processing engine.
Each transaction program is launched in a separate process, and,
once done with one transaction, can be used to do another one
without reloading. When things are slow, the extra transaction processes
are told to exit; when load picks up, more of them are forked.
Security is comparable to CGI.)

Apache will also kill off excess child processes when it deems they
are no longer required, or create new ones as demand dictates.

So, I am still not sure where the big issue is, the architecture of
Apache limits the impact of GIL contention in ways that Python alone
doesn't.

Graham

 

[sjdevnull]

Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.

The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.