New versions breaking extensions, etc.

[Cameron Laird]

Short answer to Jive's question: (1) free non-MS C compiler (either
MinGW or Borland) (2) inner calm.

I really can't understand what all the screaming and yelling is about.
Windows Python is built using an MS compiler. Those extension
developers who can't/won't buy the MS compiler use either the free
MinGW compiler or the free Borland 5.5 compiler (or both!). Yes, you

.
.
.
? As I read your description, it obscures the existence of
the free lcc and the no-charge Microsoft compilers <URL:
http://msdn.microsoft.com/visualc/vctoolkit2003/ > (along
with such other popular alternatives as Cygwin, the Intel
compiler, ...).

It's been years since I've tried any of these with Python.
They *ought* to work, though ...

 

[Cameron Laird]

The problem is the Python C/API. At the moment, it exposes things
directly (like
data structures) that may change size between major version releases. The other
issue is that the interpreter and the extensions may be linked to different
versions of the Microsoft runtime.

This is a solvable problem, but it would require:
1. Major additions to the C/API - a "Python Major Version Agnostic" API that
hides all data structures behind opaque pointers, never passes file descriptors
between the interpreter and the extension and never gets Python to
delete memory
allocated by the extension (or vice-versa)
2. Rewriting extensions to use the new API functions instead of the current
major version specific ones.

Such an API is likely to be both harder to work with and slower, simply due to
the limitations of C.

To this date, no-one has cared enough about the problem to put in the effort
required to make the C API version agnostic. Given that the API almost always

.
.
.
Tcl went through this epiphany a few years, and has largely,
though not exclusively, committed to use of the once-more-
redirected API called Stubs <URL: http://wiki.tcl.tk/stubs >.
Some Tcl-ers are orgasmic about the wonders of Stubs, some
ignore it, a few don't like it, and most, perhaps, aren't
even aware of its existence. Performance doesn't appear to
be much of an issue, though.

 

[Cameron Laird]

.
.
.

Doesn't Microsoft have an answer for that? There are (at last count) nine
skillion ActiveX
components in the wild. Surely Microsoft didn't blast them into oblivion
with dot-net -- did it?

.
.
.
Microsoft *constantly* obsoletes all sorts of third-party creations.
There's no simple formula for their actions, but, in this case, it's
fairly safe to say that substantial portions of the Redmond Monster
are actively seeking to eliminate ActiveX.

 

[Jive]

Tcl went through this epiphany a few years, and has largely,
though not exclusively, committed to use of the once-more-
redirected API called Stubs <URL: http://wiki.tcl.tk/stubs >.
Some Tcl-ers are orgasmic about the wonders of Stubs, some
ignore it, a few don't like it, and most, perhaps, aren't
even aware of its existence. Performance doesn't appear to
be much of an issue, though.

It uses function pointer tables (like COM), but in the "other direction"
apparently.

Bit by bit, I am remembering now just how stupifyingly brain-dead the MS DLL
scheme is.
Back in '87 or so, I wrote a better dynamic linker (for a Unix) in a couple
of days.

 

[Cameron Laird]

.
.
.

Bit by bit, I am remembering now just how stupifyingly brain-dead the MS DLL
scheme is.

.
.
.
Part of the trick is that it demands deep understanding
to detect the antisynergies that arise from the interac-
tions of the DLL, registry, and filesystem schemes. I
know it was only this year that I realized the whole
installation-requires-reboot absurdity is a consequence
of DLL (mis-)design.

That's judgmental of me. What I'm saying is *I* wouldn't
do an OS that way. It's certain, though, that I'll never
extract as much consumer surplus as Microsoft has, so my
evaluations should count for little.

 

[Jive]

It's not hard-coded in the linker, but hard-coded in the import library.
So if you link with msvcrt.lib (which might not be the precise name
of the import library - I cannot look up the precise name right now),
msvcrt.lib will in turn refer to msvcr71.dll.

But it makes no difference, no? The problem is that both Python.exe and the
extensions are *compiled* to link with a *particular* crt. (How "dyanamic"
is that?) We could probably kluge around the problem if it were not for the
fact that one crt might perversely define some struct, (FILE, for example),
with difference layouts in different crt DLL's. Right? The header files
contain the poison.

So it would seem that as far as the crt goes, we are at the mercy of the
micro soft ones. They could introduce an incompatible crt at any time.

By the way, I've googled around and found others outside the Python
community who have run into the same problem. In some cases, they have
opted just to stay with VC 6.0. One said, "At least the 6.0 compiler is 2.5
times as fast." Groan.

 

[Robin Becker]

Cameron Laird wrote:
......

.
Part of the trick is that it demands deep understanding
to detect the antisynergies that arise from the interac-
tions of the DLL, registry, and filesystem schemes. I
know it was only this year that I realized the whole
installation-requires-reboot absurdity is a consequence
of DLL (mis-)design.

That's judgmental of me. What I'm saying is *I* wouldn't
do an OS that way. It's certain, though, that I'll never
extract as much consumer surplus as Microsoft has, so my
evaluations should count for little.

what seems strange is that although the OS hasn't changed we need a
whole new framework just because the compiler underwent a version change.

That would not happen in a *nix style OS because the framework is the
OS. In the M$ world the $ value of causing reams of people to get the
new compiler outways any sense of what would be a decent approach to OS
design etc.

It's sad that people who are otherwise sensible about opensource seem to
be a bit silly about the poisoned apples. There was no rational reason
for me to upgrade to VC 7.x, but now I'm forced to by my preferred language.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

But it makes no difference, no? The problem is that both Python.exe and the
extensions are *compiled* to link with a *particular* crt.

No, that is not (really) the case. They are compiled to link with
msvcrt.lib, which could, at link time, then become msvcrt.dll,
msvcrt40.dll, or msvcr71.dll.

(How "dyanamic" is that?)

The actual code of the DLL is not provided until run-time - only
at run-time, the precise implementation of the CRT routines is
available - and it indeed may vary from system to system, as different
versions of msvcrt40.dll might be installed (or of python24.dll, for
that matter).

We could probably kluge around the problem if it were not for the
fact that one crt might perversely define some struct, (FILE, for example),
with difference layouts in different crt DLL's. Right?

Wrong. struct FILE binds us the the MS CRT (so the Borland or glibc CRTs
are incompatible), but struct FILE has stayed the same for more than 10
years now, so you can, in principle, mix object files compiled with
different compilers. One would need to check the header files to
determine they are really compatible, but that appears to be the intent,
and Microsoft does not dare to change it (except perhaps for the _ctype
thing, which I don't fully understand).

What really hurts is that the CRT has a number of global variables: the
global FILE objects (FILE 0 to FILE 63, I believe), the global heap,
and the global atexit handlers. If you have multiple CRTs, you get
multiple copies of these globals, and the different routines in the
different CRTs each operate on their own copy of the global variable.

So it would seem that as far as the crt goes, we are at the mercy of the
micro soft ones. They could introduce an incompatible crt at any time.

They could, but they won't. I believe they have long given up changing
or improving the CRT, since that would make things worse in most cases.
The only things they still do is to add new features; the existing ABI
apparently is never touched (again, perhaps except for a _ctype change
I don't understand).

By the way, I've googled around and found others outside the Python
community who have run into the same problem. In some cases, they have
opted just to stay with VC 6.0. One said, "At least the 6.0 compiler is 2.5
times as fast." Groan.

This is what Python did for 2.3. Already at that time, user pressure
was quite high to move to a more recent compiler; with 2.4, it was
clear that we do need some of the new features (in particular,
ongoing commercial availability, and IPv6 support in winsock.h).

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Part of the trick is that it demands deep understanding
to detect the antisynergies that arise from the interac-
tions of the DLL, registry, and filesystem schemes. I
know it was only this year that I realized the whole
installation-requires-reboot absurdity is a consequence
of DLL (mis-)design.

Can you elaborate? To me, that problem only originates from
the OS lack of support for deleting open files. If you could
delete a shared libary that is still in use (as you can on
Unix), the put the new version of the DLL in the place, it
would all work fine: new processes would use the new DLL.
Existing processes continue to use the old DLL, which would
be only deleted if the last process using it terminates.
It might be sensible to restart a few long-running processes
so that they also use the new DLL, but apart from that,
installation does need to imply reboot even on a system
that uses DLLs.

Regards,
Martin

 

[Jive]

Cameron Laird wrote:
There was no rational reason
for me to upgrade to VC 7.x, but now I'm forced to by my preferred language.

That's the way I feel about it too. Actually, I'm getting pushed toward
Microsoft from other directions, also for no good reason -- and for a whole
lot of computers, not just for the one on my desk.

Open letter to Bill Gates:

Dear Bill,

Okay. You win again. It was silly of me to think Python would help me
escape your clutches. Look at how good I am being now. I am typing this
into an Outlook Express window, using a Microsoft operating system that you
will no doubt find a way to make me replace very soon. You beat me fair and
square. Well, square, anyway. Congratulations. I give up.

Jive

 

[David Bolen]

Can you elaborate? To me, that problem only originates from
the OS lack of support for deleting open files. If you could
delete a shared libary that is still in use (as you can on
Unix), the put the new version of the DLL in the place, (...)

Note that at least on NT-based systems, you can at least rename the
existing file out of the way while it is in use in order to install a
new version.

I do think however, that until more recent changes (not sure whether
in 2K versus XP) in how DLL searches work (e.g., permitting local
versions), even with that operation, if a named DLL was available
loaded into memory, it would be used by a subsequent process
attempting to reference it regardless of the state of the filesystem.

-- David