pygame and python 2.5

[siggi]

@Ben Sizer

Hi Ben,

in January I received your message re Pygame and Python 2.5:

pygame and python 2.5
Ben Sizer kylotan at gmail.com
Fri Jan 12 11:01:00 CET 2007
--------------------------------------------------------------------------------


For now, yes. This is a long-standing problem with Python really,
requiring extensions to always be recompiled for newer versions. I
usually have to wait about 6 months to a year after any new release
before I can actually install it, due to the extension lag.

As a Python (and programming ) newbie allow me a - certainly naive -
question:

What is this time consuming part of recompiling an extension, such as
Pygame, from source code to Windows? Is it a matter of spare time to do the
job? Or do you have to wait for some Windows modules that are necessary for
compiling?

I am just asking for sake of "scientific" interest; building, compiling
from source code is a mystery to me poor Windows user ;-)

Thank you,

siggi

 

[Ben Sizer]

@Ben Sizer

Lucky I spotted this...

As a Python (and programming ) newbie allow me a - certainly naive -
question:

What is this time consuming part of recompiling an extension, such as
Pygame, from source code to Windows? Is it a matter of spare time to do the
job? Or do you have to wait for some Windows modules that are necessary for
compiling?

The problem is something like this:
- Python extensions written in C require recompilation for each new
version of Python, due to Python limitations.
- Recompiling such an extension requires you to have a C compiler set
up on your local machine.
- Windows doesn't come with a C compiler, so you have to download
one.
- The compiler that Python expects you to use (Visual Studio 2003) is
no longer legally available.
- The other compiler that you can use (MinGW) is requires a slightly
convoluted set of steps in order to build an extension.

Hopefully in the future, some of those convoluted steps will be fixed,
but that requires someone putting in the effort to do so. As is often
the case with Python, and indeed many open source projects, the people
who are knowledgeable enough to do such things usually don't need to
do them, as their setup already works just fine.

 

[skip]

Ben> Python extensions written in C require recompilation for each new
Ben> version of Python, due to Python limitations.

Can you propose a means to eliminate this limitation?

Skip

 

[John Nagle]

The problem is something like this:
- Python extensions written in C require recompilation for each new
version of Python, due to Python limitations.
- Recompiling such an extension requires you to have a C compiler set
up on your local machine.
- Windows doesn't come with a C compiler, so you have to download
one.
- The compiler that Python expects you to use (Visual Studio 2003) is
no longer legally available.
- The other compiler that you can use (MinGW) is requires a slightly
convoluted set of steps in order to build an extension.

Hopefully in the future, some of those convoluted steps will be fixed,
but that requires someone putting in the effort to do so. As is often
the case with Python, and indeed many open source projects, the people
who are knowledgeable enough to do such things usually don't need to
do them, as their setup already works just fine.

True. There really should be no need to recompile a C extension unless
the linkage format of the C compiler changes, which is a very rare event.
Binary compatibility needs to be improved.

In the GCC world, any compiler since 3.2 should generate interchangeable
output.

http://gcc.gnu.org/onlinedocs/gcc/Compatibility.html

In the Windows world, I'm not sure about compatibility across the
VC6/.NET transition, but I think you only need one version for either
side of that one.

John Nagle

 

[Mike C. Fletcher]

Ben> Python extensions written in C require recompilation for each new
Ben> version of Python, due to Python limitations.

Can you propose a means to eliminate this limitation?

Sure, write your wrapper-style extensions in ctypes . For example,
pygame-ctypes[1] should work on Python 2.5. Of course, you need to get
the PyGame dependencies (SDL) installed via some external mechanism, but
the ctypes-based code should run in Python 2.5 today (with the caveat
that it's not finished software).

[1] http://www.pygame.org/ctypes/

Have fun,
Mike

--
________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://www.vrplumber.com
http://blog.vrplumber.com

 

[skip]

Ben> Python extensions written in C require recompilation for each new
Ben> version of Python, due to Python limitations.

Mike> Sure, write your wrapper-style extensions in ctypes .

I was think more along the lines of how could the Python extension module
API change so that for example, modules compiled for Python 2.6 would
continue to work without warning under Python 2.7. Maybe ctypes is the
answer, but suspect it addresses a different API than I was thinking of.

Skip

 

[Ben Sizer]

Ben> Python extensions written in C require recompilation for each new
Ben> version of Python, due to Python limitations.

Can you propose a means to eliminate this limitation?

By putting an intermediate layer between the extensions and the
language. I suppose this is essentially what ctypes does, except from
the other direction.

If someone could explain the limitation in detail, I expect ways could
be found around it. After all, I don't know of any other systems that
require you to recompile all the extensions when you upgrade the
application.

Winamp is one application that comes to mind which has kept plugins
working across many upgrades. I doubt they're still compiling with
Visual Studio 6. Perhaps it works because they have a more restrictive
API that isn't passing non-primitive types across the DLL boundary.

 

[skip]

Ben> If someone could explain the limitation in detail, I expect ways
Ben> could be found around it. After all, I don't know of any other
Ben> systems that require you to recompile all the extensions when you
Ben> upgrade the application.

Python used to work that way. You'd then silently get errors if the API
changed between version A and version B and you neglected to recompile the
extensions you compiled against version A. Maybe the Python extension API
is mature enough now that it can be frozen, but I sort of doubt it.

Skip

 

[bearophileHUGS]

Skip:

Python used to work that way. You'd then silently get errors if the API
changed between version A and version B and you neglected to recompile the
extensions you compiled against version A.

Can't the compiled module have one or more test functions that can be
used during linking to see if the compiled module respects the
expected standard?

Bye,
bearophile

 

[Ben Sizer]

Ben> If someone could explain the limitation in detail, I expect ways
Ben> could be found around it. After all, I don't know of any other
Ben> systems that require you to recompile all the extensions when you
Ben> upgrade the application.

Python used to work that way. You'd then silently get errors if the API
changed between version A and version B and you neglected to recompile the
extensions you compiled against version A. Maybe the Python extension API
is mature enough now that it can be frozen, but I sort of doubt it.

The only reason this is an issue is because the system is tightly
bound on a binary level. Decouple that and the problem goes away.
These 'silent' errors will all stem from a small number of specific
things, each of which can be addressed. eg. PyFile_AsFile returns a
FILE*, which is all well and good if both the extension's compiler and
the language's compiler agree on what you get when you dereference
that type, and probably not so good when they don't. The answer there
is not to make assumptions about the structure of complex types across
the boundary. The same may well go for the multitude of macros that
make assumptions about the structure of a PyObject.

It's not really much to do with the maturity, since functions don't
seem to be getting regularly removed from the API. It's more the
choices made about how to implement it.

 

[Hendrik van Rooyen]

Ben> Python extensions written in C require recompilation for each new
Ben> version of Python, due to Python limitations.

Can you propose a means to eliminate this limitation?

Yes. - Instead of calling something, send it a message...

- Hendrik

 

[mensanator]

Lucky I spotted this...



The problem is something like this:
 - Python extensions written in C require recompilation for each new
version of Python, due to Python limitations.
 - Recompiling such an extension requires you to have a C compiler set
up on your local machine.
 - Windows doesn't come with a C compiler, so you have to download
one.
 - The compiler that Python expects you to use (Visual Studio 2003) is
no longer legally available.
 - The other compiler that you can use (MinGW) is requires a slightly
convoluted set of steps in order to build an extension.

Hopefully in the future, some of those convoluted steps will be fixed,
but that requires someone putting in the effort to do so. As is often
the case with Python, and indeed many open source projects, the people
who are knowledgeable enough to do such things usually don't need to
do them, as their setup already works just fine.

So you're saying the knowledgeable people's attitude
is "**** everyone else as lomg as it's not MY problem"?

And you people complain about Microsoft.

 

[Siggi]

:
[snip]

Hopefully in the future, some of those convoluted steps will be fixed,
but that requires someone putting in the effort to do so. As is often
the case with Python, and indeed many open source projects, the people
who are knowledgeable enough to do such things usually don't need to
do them, as their setup already works just fine.

Thank you, and all those putting in their comments to my thread.
As a python newbie, I conclude now that I will be better off to drop
Windows and install Linux on my next PC, to be able to reap the full
benefits of Python.

Thanks,

siggi

 

[Steve Holden]

Yes. - Instead of calling something, send it a message...

I suppose you are proposing to use the ISO 19999.333 generic
message-passing interface for this? The one that doesn't actually call a
function to pass a message?

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note: http://holdenweb.blogspot.com
See you at PyCon? http://us.pycon.org/TX2007

 

[skip]

bearophile> Can't the compiled module have one or more test functions
bearophile> that can be used during linking to see if the compiled
bearophile> module respects the expected standard?

Given the complexity of the formal API how would you test to see if the
extension violated a particular aspect of the API? What if one of the API
bits used is implemented as a C macro (as parts are) and it was changed
simply to fix a bug. Wouldn't you want to know with a high degree of
certainty that you should recompile? How would a test function tell you
that? A simple API versioning scheme does that. It means you have to
recompile when a new version of Python comes out. In fact, you can think of
it as the test function you suggest. It's just that it's noted at the time
a module is imported, not strictly speaking at link time. It tells you,
"Hey buddy. You're using an outdated version of the API." What it can't
tell you is if the parts of the API your particular module uses are used
incorrectly.

Skip

 

[skip]

Siggi> ... I conclude now that I will be better off to drop Windows and
Siggi> install Linux on my next PC, to be able to reap the full benefits
Siggi> of Python.

Darn tootin'... (*)

Skip

(*) http://en.wikipedia.org/wiki/You're_Darn_Tootin'

 

[Ben Sizer]

So you're saying the knowledgeable people's attitude
is "**** everyone else as lomg as it's not MY problem"?

And you people complain about Microsoft.

Am I one of "those people"? You don't exactly make it clear.

But yes, there is a lot of "well, it works for me" going around. If
you do that long enough, people stop complaining, so people wrongly
assume there's no longer a problem. This is partly why Python has
various warts on Windows and why the standard libraries are oddly
biased, why configuring Linux almost always ends up involving hand-
editing a .conf file, why the leading cross-platform multimedia
library SDL still doesn't do hardware graphics acceleration a decade
after such hardware became mainstream, and so on.

However, the difference between the open-source people and Microsoft
is the the open-source people aren't being paid by you for the use of
their product, so they're not obligated in any way to help you. After
all, they have already given freely and generously, and if they choose
not to give more on top of that, it's really up to them. Yes, it's
occasionally very frustrating to the rest of us, but that's life. The
best I feel I can do is raise these things on occasion, on the off-
chance that I manage to catch the attention of someone who is
altruistic, knowledgeable, and who has some spare time on their hands!

 

[Ben Sizer]

I suppose you are proposing to use the ISO 19999.333 generic
message-passing interface for this? The one that doesn't actually call a
function to pass a message?

I'm assuming you're being facetious here..?

Of course, functions get called at the ends of the message passing
process, but those functions can stay the same across versions while
the messages themselves change. The important part is reducing the
binary interface between the two sides to a level where it's trivial
to guarantee that part of the equation is safe.

eg.
Instead of having PySomeType_FromLong(long value) exposed to the API,
you could have a PyAnyObject_FromLong(long value, char*
object_type_name). That function can return NULL and set up an
exception if it doesn't understand the object you asked for, so Python
versions earlier than the one that implement the type you want will
just raise an exception gracefully rather than not linking.

The other issue comes with interfaces that are fragile by definition -
eg. instead of returning a FILE* from Python to the extension, return
the file descriptor and create the FILE* on the extension side with
fdopen.

 

[mensanator]

Am I one of "those people"? You don't exactly make it clear.

I'm talking about the people who complain about Microsoft
making the VC6 compiler no longer legally available and
yet are so irresponsible that they use it for the latest
release.

But yes, there is a lot of "well, it works for me" going around. If
you do that long enough, people stop complaining, so people wrongly
assume there's no longer a problem. This is partly why Python has
various warts on Windows and why the standard libraries are oddly
biased, why configuring Linux almost always ends up involving hand-
editing a .conf file, why the leading cross-platform multimedia
library SDL still doesn't do hardware graphics acceleration a decade
after such hardware became mainstream, and so on.

However, the difference between the open-source people and Microsoft
is the the open-source people aren't being paid by you for the use of
their product, so they're not obligated in any way to help you.

This argument has become tiresome. The Python community
wants Python to be a big fish in the big pond. That's why
they make Windows binaries available.

After all, they have already given freely and generously, and if they choose
not to give more on top of that, it's really up to them.

Right. Get people to commit and then abandon them. Nice.

Yes, it's
occasionally very frustrating to the rest of us, but that's life.

As the Kurds are well aware.

The best I feel I can do is raise these things on occasion,
on the off-chance that I manage to catch the attention of
someone who is
altruistic, knowledgeable, and who has some spare time on
their hands!

Someone who, say, solved the memory leak in the GMPY
divm() function even though he had no way of compiling
the source code?

Just think of what such an altruistic, knowedgeable
person could do if he could use the current VC compiler
or some other legally available compiler.

 

[skip]

mensanator> This argument has become tiresome. The Python community
mensanator> wants Python to be a big fish in the big pond. That's why
mensanator> they make Windows binaries available.

I suspect the main reason Windows binaries are produced is because a)
Microsoft doesn't ship Python installed on Windows, and b) your garden
variety Windows user doesn't have the tools necessary to build Python from
source. Not being a Windows user myself I don't understand all the ins and
outs of VC6 v. VC7, legal or technical. Is there nothing Microsoft could
have done to make VC7 compatible with the existing VC6-based build
procedure?

Skip