Python does not play well with others

[John Nagle]

The major complaint I have about Python is that the packages
which connect it to other software components all seem to have
serious problems. As long as you don't need to talk to anything
outside the Python world, you're fine. But once you do, things
go downhill. MySQLdb has version and platform compatibility
problems. So does M2Crypto. The built-in SSL support is weak.
Even basic sockets don't quite work right; the socket module
encapsulates the timeout mechanism but doesn't get it right.

In the Perl, Java, PHP, and C/C++ worlds, the equivalent
functions just work. That's because, in those worlds, either the
development team for the language or the development team
for the subsystem takes responsibility for making them work.
Only Python doesn't do that.

Python has been around long enough that this should have
been fixed by now.

John Nagle

 

[skip]

John> MySQLdb has version and platform compatibility problems.

Got specific examples? I've successfully used MySQLdb on Linux, Mac and
Solaris with no compatibility problems at all.

John> Even basic sockets don't quite work right; the socket module
John> encapsulates the timeout mechanism but doesn't get it right.

As has been reported here a number of times, this simply awaits someone with
the time and inclination to do the work. If it itches you then scratch the
itch.

John> In the Perl, Java, PHP, and C/C++ worlds, the equivalent functions
John> just work. That's because, in those worlds, either the
John> development team for the language or the development team for the
John> subsystem takes responsibility for making them work. Only Python
John> doesn't do that.

You seem to be expecting something for free from the volunteer development
team. If these things are important enough you can always pay someone to do
the work and contribute it back to the community. The work didn't happen
for free in any of the communities you mentioned. Someone (Sun, AT&T,
whoever) paid for it in either time or money.

Skip

 

[Gabriel Genellina]

The major complaint I have about Python is that the packages
which connect it to other software components all seem to have
serious problems. As long as you don't need to talk to anything
outside the Python world, you're fine. But once you do, things
go downhill.

That appears to be your situation.

MySQLdb has version and platform compatibility
problems.

Which, specifically?

So does M2Crypto.

Both of them are third-party libraries maintained by volunteer people
(and I could ask why do you menction M2Crypto specifically?)

The built-in SSL support is weak.
Even basic sockets don't quite work right; the socket module
encapsulates the timeout mechanism but doesn't get it right.

I use several packages connecting to the "outside Python" world
without problems. If you have these specific problems, perhaps you
could try to fix them? Get funding to make them fixed? Do the funding yourself?

In the Perl, Java, PHP, and C/C++ worlds, the equivalent
functions just work. That's because, in those worlds, either the
development team for the language or the development team
for the subsystem takes responsibility for making them work.
Only Python doesn't do that.

I think that it was done because *someone* was sponsoring the
development (like IBM, Sun, and others).


--
Gabriel Genellina
Softlab SRL






__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

 

[Harry George]

The major complaint I have about Python is that the packages
which connect it to other software components all seem to have
serious problems. As long as you don't need to talk to anything
outside the Python world, you're fine. But once you do, things
go downhill. MySQLdb has version and platform compatibility
problems. So does M2Crypto. The built-in SSL support is weak.
Even basic sockets don't quite work right; the socket module
encapsulates the timeout mechanism but doesn't get it right.

In the Perl, Java, PHP, and C/C++ worlds, the equivalent
functions just work. That's because, in those worlds, either the
development team for the language or the development team
for the subsystem takes responsibility for making them work.
Only Python doesn't do that.

Python has been around long enough that this should have
been fixed by now.

John Nagle

You experience isn't shared by everyone. Some of us find Python the
most functional and portable of the candidates you mention.

Perl - excellent modules and bindings for just about everything you
can think of, but the whole thing is painful to watch. Once you've
done a few code reviews on 10,000 line perl packages where even the
authors have no idea what the code is doing, you tend to look
elesewhere.

Java - a world of its own. They reinvent the wheel instead of linking
to existing libraries. In the process you get libraries upon
libraries upon libraries. Even if there isn't a performance hit, you
(as a human) can get lost. And the language is just too verbose to
live with.

PHP - are we talking web scripts or serious programs? Are you doing
numerical analysis, NLP, computational chemistry, or bit twiddling in
PHP?

C - the portable assembler. Solid, trusted, tunable performance,
bindings for everything. Of course memory bugs can stop your project
in its tracks for indeterminant periods.

C++ - objects tacked onto C; but that didn't work so invent a whole
world of templates and rewrite everything again, but now trickier than
C to bind to other languages. Good work can be done in C++, but that
is a testimony to the programmers and not to the language.

Python - it just works. Same scripts run on every platform. Bindings
available to every C/C++/FORTRAN library I've needed so far. Often
the bindings are not complete, but oddly enough the binding developers
have chosen to do just the functions I need, so who cares. A clean
architecture for adding more function bindings if I'm so inclined.

 

[hg]

What about C# ?
e

arf arf

 

[John Nagle]

John Nagle <[email protected]> writes:

You experience isn't shared by everyone. Some of us find Python the
most functional and portable of the candidates you mention.

The language is fine. It's the bindings to other packages that
are the problem. There are three different packages for talking
to OpenSSL, and they're all broken in some important way.

What's actually needed on the SSL side, I think, is to
add bindings to the built-in SSL to export the functionality
the M2Crypto C binding module has. Preferably with better
attention to reference count problems, and without using SWIG.
Then move over the Python portions of M2Crypto.

Some problems, all of which are known and logged bugs:

- The built in SSL package doesn't actually validate anything,
and will happily accept bogus SSL certificates.
- The built in SSL package doesn't allow access to most of the
fields of an SSL certificate, and the ones you can get
are returned in a debug format that's not parseable.
- M2Crypto has OpenSSL and SWIG version dependencies beyond
what is documented. The latest version of SWIG has
a problem which breaks builds with older versions of
OpenSSL.
- M2Crypto may still have a memory leak associated with contexts.
(Check out "close" in "Context".)
- M2Crypto doesn't understand SSL certificates which support
a list of sites.
- M2Crypto and the socket library don't play nice about timeouts.

Most of these problems have been known for years. The last person to
try to fix this was treated so badly he stopped contributing. Read
the bug history for "[1114345] Add SSL certificate validation".
It's sad.

John Nagle

 

[Chris Mellon]

The major complaint I have about Python is that the packages
which connect it to other software components all seem to have
serious problems. As long as you don't need to talk to anything
outside the Python world, you're fine. But once you do, things
go downhill. MySQLdb has version and platform compatibility
problems. So does M2Crypto. The built-in SSL support is weak.
Even basic sockets don't quite work right; the socket module
encapsulates the timeout mechanism but doesn't get it right.

In the Perl, Java, PHP, and C/C++ worlds, the equivalent
functions just work. That's because, in those worlds, either the
development team for the language or the development team
for the subsystem takes responsibility for making them work.
Only Python doesn't do that.

Python has been around long enough that this should have
been fixed by now.

John Nagle

This just isn't correct. You think that socket libraries don't have
incompatibilities in C and that C++ libraries don't have versioning
problems? You think that SSL works right from Java in all cases, or
that changing your mysql client libraries underneath PHP won't affect
it?

 

[Chris Mellon]

John Nagle <[email protected]> writes:

You experience isn't shared by everyone. Some of us find Python the
most functional and portable of the candidates you mention.

The language is fine. It's the bindings to other packages that
are the problem. There are three different packages for talking
to OpenSSL, and they're all broken in some important way.

What's actually needed on the SSL side, I think, is to
add bindings to the built-in SSL to export the functionality
the M2Crypto C binding module has. Preferably with better
attention to reference count problems, and without using SWIG.
Then move over the Python portions of M2Crypto.

Some problems, all of which are known and logged bugs:

- The built in SSL package doesn't actually validate anything,
and will happily accept bogus SSL certificates.
- The built in SSL package doesn't allow access to most of the
fields of an SSL certificate, and the ones you can get
are returned in a debug format that's not parseable.
- M2Crypto has OpenSSL and SWIG version dependencies beyond
what is documented. The latest version of SWIG has
a problem which breaks builds with older versions of
OpenSSL.
- M2Crypto may still have a memory leak associated with contexts.
(Check out "close" in "Context".)
- M2Crypto doesn't understand SSL certificates which support
a list of sites.
- M2Crypto and the socket library don't play nice about timeouts.

Most of these problems have been known for years. The last person to
try to fix this was treated so badly he stopped contributing. Read
the bug history for "[1114345] Add SSL certificate validation".
It's sad.

Just to be clear: The problem here is "my personal itch is not being
scratched by other people for me", not "Python doesn't play well with
others". You are not any more important than anyone else and assuming
that anyone cares about your problems is a major fallacy. You have a
specific library that doesn't meet your needs, and that you are not
interested in maintaining yourself, so you are attempting to leverage
some sort of community guilt trip to get other people to do it for
you. That's dishonest and off-putting.

Not everyone has the ability or desire to contribute. That is fine.
They might even ask if anyone is interested in doing it for them,
which is also fine - there are many people who enjoy working on
something that is of use to others, even if it's not important to them
personally. But stop acting like you're entitled to special
consideration, or that the "Python community" is failing some sort of
duty by not catering to you specifically.

 

[Paul Boddie]

The major complaint I have about Python is that the packages
which connect it to other software components all seem to have
serious problems. As long as you don't need to talk to anything
outside the Python world, you're fine.

I think you're overstating your case here, since there is a lot of
Python code out there (much of it under the radar of many readers of
this newsgroup/list) whose purpose is precisely that of integration
with other software components. Having read what you've written
elsewhere with regard to SSL integration with Python, though, the
causes of the symptoms you're seeing are mostly correctly identified in
your own analysis.

In the Perl, Java, PHP, and C/C++ worlds, the equivalent functions just work.

We may have differing experiences here. Java stuff, in my experience,
usually needs lots of precisely versioned .jar files and then
frequently doesn't "just work". C and C++ aren't complete environments,
but there are admittedly some libraries/frameworks/environments based
on those languages which might provide solutions which often work
satisfactorily. I don't really want to think of the cocktail of
solutions available for Perl to solve any given problem, but people
often report that the quality varies somewhat between them.

That's because, in those worlds, either the
development team for the language or the development team
for the subsystem takes responsibility for making them work.
Only Python doesn't do that.

And this is where I'd almost reach agreement with you. It's not enough
for the language to keep growing new features if the libraries are
broken or appear archaic, and this task seems to be outside the "core
developers" area of interest. Of course, people like you and I could
help update the libraries, although finding the time for making more
than just suggestions can be quite difficult, but the core developers
merely signalling that the libraries are a priority area for
development would probably lead to fewer people pushing their own
favourite syntax brush-ups or esoteric constructs, hopefully leading
some people (with enough time) to update, document and improve what's
already there.

Paul

 

[egbert]

Perl - excellent modules and bindings for just about everything ...
Java - a world of its own. They reinvent the wheel instead of ...
PHP - are we talking web scripts or serious programs? Are you ...
C - the portable assembler. Solid, trusted, tunable ...
C++ - objects tacked onto C; but that didn't work so invent ...
Python - it just works. Same scripts run on every platform ...

What about C# ?
e

 

[John Nagle]

Just to be clear: The problem here is "my personal itch is not being
scratched by other people for me", not "Python doesn't play well with
others". You are not any more important than anyone else and assuming
that anyone cares about your problems is a major fallacy. You have a
specific library that doesn't meet your needs, and that you are not
interested in maintaining yourself, so you are attempting to leverage
some sort of community guilt trip to get other people to do it for
you. That's dishonest and off-putting.

No, the problem is that these things are broken.

Python is the only major open source project I've encountered where
there's so much hostility to bug reports.

For a while, I tried submitting long, detailed bug reports showing
where in the C code a problem lies, and pointing out ways to fix it.
But I don't want to take over maintenance on the SSL package;
it's too delicate.

John Nagle

 

[Chris Mellon]

No, the problem is that these things are broken.

That's *your* problem. I don't write any code that uses SSL and so
it's not *my* problem. In a community effort, you either need to fix
your own problems or convince other people that they should fix them.
Saying "hey guys, this is broken" is a start. "You guys are shirking
your responsibilities and not supporting me the way I want to be
supported" is not, because it assumes an entitlement that you do not
have.

Python is the only major open source project I've encountered where
there's so much hostility to bug reports.

You've never submitted a duplicate bug to Gnome, have you? Regardless,
I see very little hostility toward bug reports. You confuse people
failing to satisfy your needs with hostility.

For a while, I tried submitting long, detailed bug reports showing
where in the C code a problem lies, and pointing out ways to fix it.
But I don't want to take over maintenance on the SSL package;
it's too delicate.

And yet, you are perfectly willing to claim that everyone else who
makes the same decision is failing, somehow. Could one not point out
that you are failing your responsibility to the Python community,
following the same logic?

 

[skip]

John> Python is the only major open source project I've encountered
John> where there's so much hostility to bug reports.

You're misinterpreting lack of time (or itches that need scratching by the
people willing to do the work) for hostility. Lest you think that there is
no activity on bugs, scan the python-checkins mailing list archive:

http://mail.python.org/pipermail/python-checkins/

Grepping through the January mbox archive I see 96 different bug ids
mentioned in the checkins. For December I see 116 different ids. I could
go back farther, but I'll let you do that.

Skip

 

[Kay Schluehr]

for the language to keep growing new features if the libraries are
broken or appear archaic, and this task seems to be outside the "core
developers" area of interest.

But it shouldn't be - although this is not a request for the core
developers to maintain any 3rd party package in the world, of course. A
while ago someone on python-dev suggested to use builtbot to monitor
were 3rd party packages get broken by new Python releases and offered
this as a service to library developers. I do think this and similar
ideas are relevant to improve overall quality, not only that of the
CPython interpreter. I would also suggest stricter policies for PyPI
were unmaintained packages might be removed after a period ( or at
least tagged in a certain way ).

But maybe admitting quality problems of free software that is affecting
the whole community ( and not just isolated projects ) is just too much
negative PR on a "competitive language market"? We all know Ruby makes
us happy and Haskell is driven by real academics, who are doing
software engineering right, by default. Maybe Python could survive only
leaving the impression that it binds to everything and has an ever
growing code base of high quality?

 

[Dennis Lee Bieber]

What about C# ?

Uhm... Unless something has happened that I don't know about, isn't
C# a M$ specific product?
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Stephan Kuhagen]

Perl - excellent modules and bindings for just about everything you ....
Java - a world of its own. They reinvent the wheel instead of linking ....
PHP - are we talking web scripts or serious programs? Are you doing ....
C - the portable assembler. Solid, trusted, tunable performance, ....
C++ - objects tacked onto C; but that didn't work so invent a whole ....
Python - it just works. Same scripts run on every platform.

The world has become very small these days...

Although I do not agree to the OP, because Python works for me very good on
different platforms (i.e. different Versions of Windows, Linux and Mac OS
X), there are some other old fellows around, which are very good for cross
platform programming. C and C++ are definitely *not* one of them, I would
say. True, you can write cross platform with them, but... I'm missing
especially some of the older cross platform languages in your list (I would
set Tcl at the top of the list, but Lisp and others will appear in such a
list also), which have grown and became mature over the years. Most cross
platform problems and others issues have been fixed and really worked out
in them. New kids on the block may be more dynamic (Python really is), but
they can learn much from the old guys. Sometimes one should have a look and
learn from still good working ancestors...

Regards
Stephan

 

[Stephan Kuhagen]

Uhm... Unless something has happened that I don't know about, isn't
C# a M$ specific product?

Mono?

Regards
Stephan

 

[Gabriel Genellina]

Uhm... Unless something has happened that I don't know about, isn't
C# a M$ specific product?

Not exactly. It's defined by an ISO standard; there are several
implementations apart from MS, like Borland C# Builder and the Mono project.


--
Gabriel Genellina
Softlab SRL






__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

 

[=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=]

Python - it just works. Same scripts run on every platform. Bindings
available to every C/C++/FORTRAN library I've needed so far. Often
the bindings are not complete, but oddly enough the binding developers
have chosen to do just the functions I need, so who cares. A clean
architecture for adding more function bindings if I'm so inclined.

Although that is true on Linux, it is often not the case on Windows.
Sometimes the binding lacks a binary for the minor version of Python
you use or it depends on something which depends on something which
sometimes is only available in source form. Still, I would rather take
Python's approach than Java's.

 

[Steve Holden]

But it shouldn't be - although this is not a request for the core
developers to maintain any 3rd party package in the world, of course. A
while ago someone on python-dev suggested to use builtbot to monitor
were 3rd party packages get broken by new Python releases and offered
this as a service to library developers. I do think this and similar
ideas are relevant to improve overall quality, not only that of the
CPython interpreter. I would also suggest stricter policies for PyPI
were unmaintained packages might be removed after a period ( or at
least tagged in a certain way ).

But maybe admitting quality problems of free software that is affecting
the whole community ( and not just isolated projects ) is just too much
negative PR on a "competitive language market"? We all know Ruby makes
us happy and Haskell is driven by real academics, who are doing
software engineering right, by default. Maybe Python could survive only
leaving the impression that it binds to everything and has an ever
growing code base of high quality?

Looking at the history of the bug, it ends with a request from Martin
von Loewis that James Eagan (the author of the patch, for which we
should be grateful) provide unit tests and documentation to go with the
code already contributed, finally followed by a comment from James that
it will be a while before he has time to make the necessary changes.

I don't think there's any attempt here to avoid admitting that there's a
quality problem with the existing code. Adding new functionality without
tests and documentation certainly wouldn't improve things.

regards
Steve