New versions breaking extensions, etc.

[Jive]

Can someone explain to me why Python 2.4 on MS Windows has these backward
compatibility problems? What am I missing? Why won't extensions compiled
to run with 2.3 also work with 2.4? Why does it matter whether a component
was compiled with VC++ 6.0 or 7.1? I've been using MS stuff for 6 years.
Before I started using Python, I don't remember ever having a program tell
me I had to use an OLDER version of something to make the program work.
Newer, maybe, but older, never. I had to revert to Python 2.3 because I
have applications that will not run under Python 2.4.

Although I have been a full-time software developer since the late 70's, I
don't know much about Microsoft stuff. (Amazing trick that, eh?) But I've
been reading some things on the MS web pages that say dot-NET and all that
should NOT break existing stuff.
http://msdn.microsoft.com/library/en-us/vclib/html/_crt_c_run.2d.time_libraries.asp I
don't pretend to understand it all, but certainly one would not expect MS to
screw things up too badly on purpose. The value of MS Windows lies in the
applications that it can run. That's part of the genius of the Evil Genius.
He understood when others didn't that when Joe Blow in his basement develops
a program to run under MS Windows, Joe Blow is making money for the E.G.,
and it doesn't cost the E.G. a penny.

In my own small domain, I cannot even consider coming out with new releases
that are not drop-in replacements for what went before. Customers won't go
for it. It's an absolute deal-breaker.

What puzzles me is that, apparently, the incompatibility of Python 2.4 and
extensions built for 2.3 did not come as a nasty surprise. I get the
impression that it was expected. Can someone explain that to me?

I hope this doesn't sound like I'm complaining. I think Python is great.
And the price can't be beat!

 

[Robert]

VS7 is a really a vastly different beastie than VS6.

 

[News M Claveau /Hamster-P]

Hi !

But, if Python is as much sensitive to the passage of an external software,
version 6 (obsolete) with a version 7 (obsolete also), it is worrying.

Michel Claveau

 

[Nick Coghlan]

Can someone explain to me why Python 2.4 on MS Windows has these backward
compatibility problems? What am I missing?

The problem is the Python C/API. At the moment, it exposes things directly (like
data structures) that may change size between major version releases. The other
issue is that the interpreter and the extensions may be linked to different
versions of the Microsoft runtime.

This is a solvable problem, but it would require:
1. Major additions to the C/API - a "Python Major Version Agnostic" API that
hides all data structures behind opaque pointers, never passes file descriptors
between the interpreter and the extension and never gets Python to delete memory
allocated by the extension (or vice-versa)
2. Rewriting extensions to use the new API functions instead of the current
major version specific ones.

Such an API is likely to be both harder to work with and slower, simply due to
the limitations of C.

To this date, no-one has cared enough about the problem to put in the effort
required to make the C API version agnostic. Given that the API almost always
remains *source* compatible, within a month or two of a new Python release, the
extension developers have generally tweaked their build environments to also
produce binaries for the new release.

In the current case (Python 2.4), the transition from msvcrt to msvcrt71, and
the poor quality of the free development tools offered by Microsoft has
exacerbated the problem on Windows. The Python developers are looking at
improving the documentation and support for building extensions with Windows
compilers other than Visual Studio .Net 2003 (mainly MinGW and the free MS tools).

Although key extension developers could always try asking the PSF to buy them a
Visual Studio license

Cheers,
Nick.

 

[Jive]

The problem is the Python C/API. At the moment, it exposes things directly (like
data structures) that may change size between major version releases.

Okay. That could be fixed.

The other
issue is that the interpreter and the extensions may be linked to different
versions of the Microsoft runtime.

Doesn't Microsoft have an answer for that? There are (at last count) nine
skillion ActiveX
components in the wild. Surely Microsoft didn't blast them into oblivion
with dot-net -- did it?

This is a solvable problem, but it would require:
1. Major additions to the C/API - a "Python Major Version Agnostic" API that
hides all data structures behind opaque pointers, never passes file descriptors
between the interpreter and the extension and never gets Python to delete memory
allocated by the extension (or vice-versa)

I remember having trouble with that last bit once. But I seem to recall
there was an easy
answer. Something to do with the main ap and the DLL using the same heap or
some
such thing.

2. Rewriting extensions to use the new API functions instead of the current
major version specific ones.

That's a one-time thing, for those who choose to do it. Certainly the old
API
would still be available.

Such an API is likely to be both harder to work with and slower, simply due to
the limitations of C.

To this date, no-one has cared enough about the problem to put in the effort
required to make the C API version agnostic. Given that the API almost always
remains *source* compatible, within a month or two of a new Python release, the
extension developers have generally tweaked their build environments to also
produce binaries for the new release.

It would be a Good Thing to put a stop to that requirement, no?

In the current case (Python 2.4), the transition from msvcrt to msvcrt71, and
the poor quality of the free development tools offered by Microsoft has
exacerbated the problem on Windows.

No doubt. I don't understand what the deal is with msvcr71 and all that,
but I have
figured out that it's a dirty rotten shame. If you google for msvcr71, you
will
get pages and pages of links to plaintive cries for help. We are not alone.

The Python developers are looking at
improving the documentation and support for building extensions with Windows
compilers other than Visual Studio .Net 2003 (mainly MinGW and the free MS tools).

That would be good. But is using VC++ 6.0 right out? As it stands, I can
no longer
build extensions that will run with a standard dot-net Python 2.4 release.
Or at least I don't
know how. And if I build 2.4 under VC++ 6.0 and distribute that around the
company, it's not
clear to me if we can then use new third party extensions. Some
documentation
clearing it all up would be very helpful. Maybe it exists somewhere
already. This is
a Microsoft problem, not a Python problem.

Like I said, dirty rotten shame.

Thinking it over, I'm wondering why the Python crowd went with dot-NET in
the first place.
Surely the world would be a better, happier place without MS jerking
everyone around.
We work our fingerprints to the bone writing code to make their stinking
little OS do something
useful, and what have they ever given us in return? -- Besides the
aquaducts?

Jive

 

[Jive]

P.s. Does anyone know how to make Outlook Express leave my damned line-ends
alone? If I want line-ends. I know where to find the ENTER key.

 

[JanC]

Jive schreef:

P.s. Does anyone know how to make Outlook Express leave my damned
line-ends alone? If I want line-ends. I know where to find the ENTER
key.

Google for "oe-quotefix", but the best solution is to use a proper
newsreader. ;-)

 

[Mike Meyer]

It would be a Good Thing to put a stop to that requirement, no?

Actually, there's a problem on Unix that may not exist on
Windows. Python is installed in <PREFIX>/lib/python<version>/. This
lets us have multiple versions of Python installed at the same time,
which is a good thing.

Now, installed packages and extensions go in
<PREFIX>/lib/python<version>/site-packages. This means that you can't
use *any* of the previously installed packages in the new
version, even if they were pure python. Since I use TMDA, my mail
stopped working when I installed python 2.4.

The real solution is a database of installed packages (which may well
be needed for the apt-get/CPAN functionality that people want) so that
a single python script can fetch and install all the installed
packages.

<mike

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Why won't extensions compiled to run with 2.3 also work with 2.4?

I believe nobody has answered *this* question, yet:

Python extensions built for 2.3 link with python23.dll, Python
extensions build for 2.4 link with python24.dll.

pythonxy.dll has global variables, e.g. the pointers to True,
False, None, or the dict type object. If you have two copies
of the Python runtime, you get two copies of each local object.
This cannot work: we really need to rely on having only one
dict type, or else "is" comparisons with the dict type fail.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Doesn't Microsoft have an answer for that?

Microsoft's answer to this question is: don't do that. Never
mix different versions of the CRT in a single application.

There are (at last count) nine skillion ActiveX
components in the wild. Surely Microsoft didn't blast them into oblivion
with dot-net -- did it?

No, it didn't - but so what? The point is that ActiveX controls normally
don't use the CRT in dangerous ways, since they are constrained to use
Co* function calls only.

I remember having trouble with that last bit once. But I seem to recall
there was an easy
answer. Something to do with the main ap and the DLL using the same heap or
some
such thing.

Something like that, for sure.

[require that extensions be recompiled]

It would be a Good Thing to put a stop to that requirement, no?

No. The maintenance cost of such a new API would be significantly
higher, and Python would lose some of its attractiveness for C
developers.

No doubt. I don't understand what the deal is with msvcr71 and all that,

You probably either need to learn what the deal is, or else you have to
trust others that there is no better way.

That would be good. But is using VC++ 6.0 right out? As it stands, I can
no longer build extensions that will run with a standard dot-net Python
> 2.4 release. Or at least I don't know how.

The most easy way to do this is to use VC7.1 (VS.NET 2003). That will
work out of the box (you need to box first, of course).

Thinking it over, I'm wondering why the Python crowd went with dot-NET in
the first place.

We did not go to .NET. Python 2.4 has nothing to do with .NET. It just
happens that the C compiler we use ships as part of a product that has
..NET in its name - the C compiler itself is Microsoft Visual C 7.1,
aka Microsoft C 13.10.

That compiler is now used because it has a number of improvements over
Microsoft C 12.00, and because VC6 is no longer commercially available.
Many developers have only VS .NET 2003 available, so if Python would
continue to be built with VC6, those people could not build
extensions for it - they could not even buy the compiler they needed,
anymore. OTOH, people who only have VC6 just need to buy VS.NET 2003,
which is still available. Or else they find out what alternative
compiler arrangements can be made.

Surely the world would be a better, happier place without MS jerking
everyone around.

It would surely be better if the operating system shipped with a
compiler, so anybody could rebuild things, and you wouldn't need
binary distributions in the first place

Regards,
Martin

 

[Jive]

OTOH, people who only have VC6 just need to buy VS.NET 2003,
which is still available.

I don't even know how to do that! What's the difference between VC++
..net Standard and Visual Studio .net Pro? (Besides $370?) Is the former
C++ only, but with the IDE, and the later the whole shebang with SourceSafe,
VBASIC, and all that?

OH NO! I've gone seriously off-topic. Please don't call the Spanish
Inquisiton. Allow me to re-phrase the question: What do I need to build
(on-topic) Python extensions?

 

[John Machin]

I don't even know how to do that! What's the difference between VC++
.net Standard and Visual Studio .net Pro? (Besides $370?) Is the former
C++ only, but with the IDE, and the later the whole shebang with SourceSafe,
VBASIC, and all that?

OH NO! I've gone seriously off-topic. Please don't call the Spanish
Inquisiton. Allow me to re-phrase the question: What do I need to build
(on-topic) Python extensions?

Short answer to Jive's question: (1) free non-MS C compiler (either
MinGW or Borland) (2) inner calm.

I really can't understand what all the screaming and yelling is about.
Windows Python is built using an MS compiler. Those extension
developers who can't/won't buy the MS compiler use either the free
MinGW compiler or the free Borland 5.5 compiler (or both!). Yes, you
have to be careful about mixing the runtimes. An extension that tries
to use a FILE * that was created by Python will crash. Using free() on
a pointer that was malloc()ed by the other party isn't a bright idea
either. There are adequate solutions to these problems, involving
Python-supplied higher-level functions instead of C runtime functions.
Otherwise, not a problem. Distutils has made the process of using MinGW
and bcpp a snap. The documentation is adequate. When a new version of
Python comes out, one rebuilds and tests one's extensions. So ... now
there are THREE compilers that can be used instead of the one that
Python's built with; what's the big deal?

 

[It's me]

On the other hand, it can be annoying.

I can't use Python 2.4 right now because NumPy won't run. So, I need to
wait for NumPy to get updated.

Of course, one would say: but NumPy is open source, go build it yourself.

My answer is simple: If there are more then 24 hours to a day, I definitely
would...

--
It's me

I don't even know how to do that! What's the difference between VC++
.net Standard and Visual Studio .net Pro? (Besides $370?) Is the former
C++ only, but with the IDE, and the later the whole shebang with SourceSafe,
VBASIC, and all that?

OH NO! I've gone seriously off-topic. Please don't call the Spanish
Inquisiton. Allow me to re-phrase the question: What do I need to build
(on-topic) Python extensions?

Short answer to Jive's question: (1) free non-MS C compiler (either
MinGW or Borland) (2) inner calm.

I really can't understand what all the screaming and yelling is about.
Windows Python is built using an MS compiler. Those extension
developers who can't/won't buy the MS compiler use either the free
MinGW compiler or the free Borland 5.5 compiler (or both!). Yes, you
have to be careful about mixing the runtimes. An extension that tries
to use a FILE * that was created by Python will crash. Using free() on
a pointer that was malloc()ed by the other party isn't a bright idea
either. There are adequate solutions to these problems, involving
Python-supplied higher-level functions instead of C runtime functions.
Otherwise, not a problem. Distutils has made the process of using MinGW
and bcpp a snap. The documentation is adequate. When a new version of
Python comes out, one rebuilds and tests one's extensions. So ... now
there are THREE compilers that can be used instead of the one that
Python's built with; what's the big deal?

 

[Steven Bethard]

My answer is simple: If there are more then 24 hours to a day, I definitely
would...

Can we get a patch in for this?
datetime.timedelta(1)

would be much preferable to the current:
datetime.timedelta(1, 3600)



Steve

 

[Jeremy Bowers]

Actually, there's a problem on Unix that may not exist on
Windows. Python is installed in <PREFIX>/lib/python<version>/. This
lets us have multiple versions of Python installed at the same time,
which is a good thing.
...
The real solution is...

Interestingly, I saw you going somewhere else for this. I expected,

"The real solution is, at least for the pure Python packages, a
system-wide directory for those, reserving site-packages for things that
are truly version specific."

I'm not sure I'm ready to "propose" this yet, but thoughts? Bad idea, good
idea?

Obviously it isn't perfect, but it seems to me that reverse compatibility
has been good enough for a while now that the imperfections are minor in
impact...

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

I don't even know how to do that! What's the difference between VC++
.net Standard and Visual Studio .net Pro? (Besides $370?) Is the former
C++ only, but with the IDE, and the later the whole shebang with SourceSafe,
VBASIC, and all that?

According to

http://msdn.microsoft.com/howtobuy/vstudio/

there is, for VS.NET, "Academic", "Professional", "Enterprise
Developer", and "Enterprise Architect". For Python extensions, all
these releases will work; see

http://msdn.microsoft.com/howtobuy/vstudio/features/

for a checkmark list. For VC++ .NET 2003 Standard, it appears that
you get just the C++ compiler, and the IDE, see

http://msdn.microsoft.com/howtobuy/visualc/features/default.aspx

OH NO! I've gone seriously off-topic. Please don't call the Spanish
Inquisiton. Allow me to re-phrase the question: What do I need to build
(on-topic) Python extensions?

Either VS.NET 2003 or VC++ .NET 2003 should do (although I don't know
anybody who owns the latter to be sure). The core issue is that it needs
a "native" C++ compiler (ie. not just managed C++), and that it needs
mscvcr71.dll.

Regards,
Martin

 

[Jive]

Either VS.NET 2003 or VC++ .NET 2003 should do (although I don't know
anybody who owns the latter to be sure). The core issue is that it needs
a "native" C++ compiler (ie. not just managed C++), and that it needs
mscvcr71.dll.

Sorry if I'm being dense. If I had that DLL, why couldn't I use VC++ 6.0?

It's still not clear to me exactly what's in each package. One would think
there would be a more technical description on the MS web pages.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Sorry if I'm being dense. If I had that DLL, why couldn't I use VC++ 6.0?

Sorry for being imprecise. You don't need just msvcr71.dll, you need an
import library for it, and you need to cause link.exe to use that import
library. VC6 will continue to link your extension modules with
msvcrt40.dll - regardless of what other DLLs you have on your system.

It might be that you also need the header files for msvcr71.dll,
although I believe you could get away with using the "wrong" (i.e.
VC6) header files.

Regards,
Martin

 

[Jive]

6.0?

Sorry for being imprecise. You don't need just msvcr71.dll, you need an
import library for it, and you need to cause link.exe to use that import
library. VC6 will continue to link your extension modules with
msvcrt40.dll - regardless of what other DLLs you have on your system.

It might be that you also need the header files for msvcr71.dll,
although I believe you could get away with using the "wrong" (i.e.
VC6) header files.

Regards,
Martin

Well, ain't that enough to gag a maggot? I was aware that DLL's don't
really link dynamically. I was not aware that the crt dll file name was
hard-coded into the linker. But I looked on the link line and, sure enough,
that particular dll was not listed among the others.

Malice or incompetence? You be the judge.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Well, ain't that enough to gag a maggot? I was aware that DLL's don't
really link dynamically. I was not aware that the crt dll file name was
hard-coded into the linker. But I looked on the link line and, sure enough,
that particular dll was not listed among the others.

It's not hard-coded in the linker, but hard-coded in the import library.
So if you link with msvcrt.lib (which might not be the precise name
of the import library - I cannot look up the precise name right now),
msvcrt.lib will in turn refer to msvcr71.dll.

In addition, through #pragma comment(lib, foo.lib), the Microsoft C
*header* files force references to import libraries into the object
files, so that the import libraries don't even appear on the linker
line - instead, the linker gets additional command line options from
the object files; use dumpbin to find out what those are.

On the plus side, I believe that the name of the import library for
the CRT was always the same (msvcrt.lib), so different versions of
that file refer to different actual CRTs (msvcrt.dll, msvcrt40.dll,
msvcr70.dll, msvcr71.dll). So just replacing your msvcrt.lib file
of VC6 with the VC71 one might be enough - but then, perhaps there
also where changes to the header files which you need to get.

Regards,
Martin