Python in sci/tech applications

[mattf]

I've discovered Python and have been trying it out lately as a possible
replacement for computations that would ordinarily be done with a
commercial package like Matlab or IDL. I'd like to mention a few things
I've run across that have either surprised me or kept me from doing
things the way I'd like to.

1) -There's a large and active sci/tech Python community out there.-
This was something of a surprise. If you look at the python.org site
and click down a couple of levels past the front page, there's a rather
brief mention of scientific and numeric applications-- but I don't
think this does justice to the current levels of activity and
accomplishment.

2) -There's a very impressive set of libraries out there-
NumPy, SciPy, Enthought. It's really kind of stunning how mature these
libraries are and how much I had to poke around to figure that out.

3) -There's a problem with development under Windows.
A typical task will entail writing a 'pure python' prototype to get the
'data in, data out' part of a problem straightened out, then writing a
module in C to get adequate performance in production runs. But the C
compiler that my employer provides (the current version of MSVS)
doesn't produce libraries that work with the current version of Python.
Ooops. This, in the real world, is a big problem. I -love- Python. And
I think I could convince other people to use it. But I've got to have a
way to produce compiled modules painlessly, i.e., without installing a
new operating system.

 

[Paul Boddie]

I'd like to mention a few things I've run across that have either surprised me or kept me
from doing things the way I'd like to.

1) -There's a large and active sci/tech Python community out there.-
This was something of a surprise. If you look at the python.org site
and click down a couple of levels past the front page, there's a rather
brief mention of scientific and numeric applications-- but I don't
think this does justice to the current levels of activity and
accomplishment.

If you want an in-depth impression of how widely Python is used,
python.org is not really the place that manages to provide it. Sure,
"NASA uses Python" (appropriate response: dressed in a flight suit,
leap up and punch the air?) but isn't OLPC (One Laptop per Child:
http://www.laptop.org/) worth a mention, given the central role of
Python in a device that will potentially reach millions? To the site's
credit, the link to scientific and numeric applications does yield a
lot of information, and I suppose it's hard to focus the site on any
one thing that any given group considers of utmost importance.

However, I think you're going to find that the python.org worldview can
be somewhat alien, especially if your primary interest isn't Python
(the language). To take another domain as an example, if you do desktop
development, the treatment of graphical user interface programming on
the site is quite possibly going to be completely disaligned with what
you'd expect: Python is used extensively in various desktop
environments (not that you'd get that impression from a cursory
examination of python.org), but the python.org (or comp.lang.python)
worldview mostly excludes such things as if the desktop is something
you abstract away rather than take advantage of.

And on the subject of how well "core Pythoneers" know of Python's
successes: how many people knew that Python was used in the OLPC device
before Alan Kay mentioned it at EuroPython? It's hard to promote Python
when you don't know where it's being successfully used already.

2) -There's a very impressive set of libraries out there-
NumPy, SciPy, Enthought. It's really kind of stunning how mature these
libraries are and how much I had to poke around to figure that out.

I guess a lot of the poking around comes about because people are
reluctant to provide any semi-official guide to the best libraries, and
that the most promotion occurs for the most fashionable products in the
most fashionable domains. Some of us have tried to provide better
information about libraries for Python, but such details remain
relatively hidden in the python.org Wiki:

http://wiki.python.org/moin/UsefulModules

With respect to content on python.org, I think all that remains is an
acknowledgement that certain resources are never going to return to
their original vitality, and that alternative mechanisms for publishing
content are required. For example, the XML topic guide surely hasn't
seen many updates in a very long time, and despite a reasonable level
of maintenance, the database topic guide could surely be more actively
maintained by aggregating package information from the package index.

Paul

 

[bearophileHUGS]

mattf:

3) -There's a problem with development under Windows.

It's possibile to compile Python with MinGW, and to create extensions
with it. So some site can host a single zip file that contains both
MinGW and Python compiled with it, all ready and set. A person not much
expert can then create compiled small extensions in a short time
without too much complexities. (Or maybe the D language can be packed
into that, instead of MinGW, to do similar things. D can be a good
language used with Python) But then probably PIL, scipy, etc have to be
compiled again for such alternative official or semi-official Python
distribution.

Bye,
bearophile

 

[Robert Kern]

mattf:

It's possibile to compile Python with MinGW, and to create extensions
with it. So some site can host a single zip file that contains both
MinGW and Python compiled with it, all ready and set. A person not much
expert can then create compiled small extensions in a short time
without too much complexities. (Or maybe the D language can be packed
into that, instead of MinGW, to do similar things. D can be a good
language used with Python) But then probably PIL, scipy, etc have to be
compiled again for such alternative official or semi-official Python
distribution.

With a few caveats, mingw will simply work with the standard 2.4 (and presumably
2.5) interpreter. We distribute mingw set up to do this with our "Enthought
Edition" Python distribution.

http://code.enthought.com/enthon/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[robert]

mattf:

It's possibile to compile Python with MinGW, and to create extensions
with it. So some site can host a single zip file that contains both
MinGW and Python compiled with it, all ready and set. A person not much
expert can then create compiled small extensions in a short time
without too much complexities. (Or maybe the D language can be packed
into that, instead of MinGW, to do similar things. D can be a good
language used with Python) But then probably PIL, scipy, etc have to be
compiled again for such alternative official or semi-official Python
distribution.

Is it really not possible to create extension libs with older MSVC or Mingw, which work with regular Python binaries version 2.4 and 2.5 ?
Maybe with some special import libraries?

(another option is to stay with Python2.3, which has a small memory footprint and compiles with crtl4.2 libs (VC6, ...) )

-robert

 

[bearophileHUGS]

Robert Kern:

We distribute mingw set up to do this with our "Enthought
Edition" Python distribution.
http://code.enthought.com/enthon/

Sorry, maybe I'm blind but I don't see MinGW listed in that page...
Maybe it's included but not listed...

Bye,
bearophile

 

[Fredrik Lundh]

Is it really not possible to create extension libs with
> older MSVC or Mingw, which work with regular Python binaries
> version 2.4 and 2.5 ?

last time I tried, it took me 20 minutes from that I typed "mingw" into
google until I had built and tested my first non-trivial extension. your
milage may vary.

</F>

 

[Robert Kern]

Robert Kern:

Sorry, maybe I'm blind but I don't see MinGW listed in that page...
Maybe it's included but not listed...

It's there.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[robert]

last time I tried, it took me 20 minutes from that I typed "mingw" into
google until I had built and tested my first non-trivial extension. your
milage may vary.

thats far beyond my dexterity.

When one follows ..
http://docs.python.org/inst/tweak-flags.html#SECTION000622000000000000000
http://www.zope.org/Members/als/tips/win32_mingw_modules

...this seems only to cover the immediate python dll issues. What happens with the C runtime libraries? You'll bind 2 different C-RTLs (DLLs) etc.?
And what happens for example with heap objects created with one C-RTL and deleted/free'd with the other?

-robert

 

[bearophileHUGS]

Fredrik Lundh:

last time I tried, it took me 20 minutes from that I typed "mingw" into
google until I had built and tested my first non-trivial extension. your
milage may vary.

But probably before those 20 minutes there is a lot of time of
experience of yours with CPython sources, other compilers, and so on,
so for most people this timing comparison just means "It can be done"


Bye,
bearophile

 

[Steve Holden]

It's there.

Well I tend to believe you, but Firefox is calling you a liar: a search
fails at "min".

Do you anticipate a 2.5-based release any time soon?

regards
Steve

 

[Robert Kern]

Well I tend to believe you, but Firefox is calling you a liar: a search
fails at "min".

Sorry, by "it's there" I meant "it is actually included in the distribution
regardless of any omissions on the web page."

Do you anticipate a 2.5-based release any time soon?

We'll start working on it soon, at least.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[John Coleman]

Maybe I don't know what I'm looking for, but I downloaded Enthought a
few days ago and don't seem to find MinGW on my system. There are 2
relatively small (totalling about 13 kb IIRC) *python* files deeply
buried in the distribution with mingw in their filename but nothing
like a gcc compiler. I've mostly used visual studio or code warrior for
C compiling, so again - maybe I don't know what I am looking for.

-John Coleman

p.s. Pretty cool sig line - the older I get, the more enigmatic the
world seems. I appreciate all of the effort that Enthought puts into
their distribution.

 

[Robert Kern]

Maybe I don't know what I'm looking for, but I downloaded Enthought a
few days ago and don't seem to find MinGW on my system. There are 2
relatively small (totalling about 13 kb IIRC) *python* files deeply
buried in the distribution with mingw in their filename but nothing
like a gcc compiler. I've mostly used visual studio or code warrior for
C compiling, so again - maybe I don't know what I am looking for.

It is in c:\Python24\Enthought\mingw32\, IIRC (I don't have it in front of me).
Add c:\Python24\Enthought\mingw32\bin\ to your PATH environment variable, and
gcc.exe should be executable.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[John Coleman]

It is in c:\Python24\Enthought\mingw32\, IIRC (I don't have it in front of me).
Add c:\Python24\Enthought\mingw32\bin\ to your PATH environment variable, and
gcc.exe should be executable.

I have no such sub-directory. I'll try an uninstall-install. When I did
the original installation the installer hung at the end and I had to
manually close it. The Python interpreter was there and worked properly
and all of the goodies like numby and chaco were there so I wrote it
off as nothing more then a failure to close properly, but evidently I
missed an essential part of the installation.

-John Coleman

 

[Magnus Lycka]

When one follows ..
http://docs.python.org/inst/tweak-flags.html#SECTION000622000000000000000
http://www.zope.org/Members/als/tips/win32_mingw_modules

..this seems only to cover the immediate python dll issues. What happens
with the C runtime libraries? You'll bind 2 different C-RTLs (DLLs)
etc.? And what happens for example with heap objects created with one
C-RTL and deleted/free'd with the other?

As far as I understand, there should be only one runtime library
involved. MinGW relies on the Microsoft runtime. It seems some
people have had problems using MSVCRT71 with MinGW, but I'm sure
that e.g. the Enthought people can explain those issues, since they
bundle MinGW in their Python 2.4 distribution.

 

[John Coleman]

I have no such sub-directory. I'll try an uninstall-install. When I did
the original installation the installer hung at the end and I had to
manually close it. The Python interpreter was there and worked properly
and all of the goodies like numby and chaco were there so I wrote it
off as nothing more then a failure to close properly, but evidently I
missed an essential part of the installation.

-John Coleman

Problem fixed. I had clicked on the 10-05-06 download (thinking that
enthought_setup-1.1.0.msi led to the set up the full enthought
package), but this is evidently only a partial download. I tried it a
second time with the same results, but perhaps I wasn't using the
"browse and install" option correctly (clicking that button on the
enstaller caused a lot of files to be transfered, but there didn't seem
to be any real list to browse). In any event, the August download
worked like a charm and had it all, including mingw. Thanks for your
help.

-John Coleman

 

[Robert Kern]

As far as I understand, there should be only one runtime library
involved. MinGW relies on the Microsoft runtime. It seems some
people have had problems using MSVCRT71 with MinGW, but I'm sure
that e.g. the Enthought people can explain those issues, since they
bundle MinGW in their Python 2.4 distribution.

No, there are two if you use an unmodified Mingw distribution. Mingw is built
around MSVCRT.dll . For many extensions, this doesn't cause a problem since
things like FILE pointer don't cross C runtime boundaries. One can edit gcc's
spec file to make it use MSVCR71.dll . However, Mingw's headers are still
written for MSVCRT.dll and Microsoft changed some of the internal details that
they rely upon. For many (most?) extensions, this won't matter, either. C++
extensions using iostreams have a problem, IIRC. Unfortunately, neither approach
works for all extensions. It may even be the case that some extensions won't
work under *either* method.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[robert]

No, there are two if you use an unmodified Mingw distribution. Mingw is
built around MSVCRT.dll . For many extensions, this doesn't cause a
problem since things like FILE pointer don't cross C runtime boundaries.
One can edit gcc's spec file to make it use MSVCR71.dll . However,
Mingw's headers are still written for MSVCRT.dll and Microsoft changed
some of the internal details that they rely upon. For many (most?)
extensions, this won't matter, either. C++ extensions using iostreams
have a problem, IIRC. Unfortunately, neither approach works for all
extensions. It may even be the case that some extensions won't work
under *either* method.

In past I asked for linking Python2.5 and next Pythons on Win against such standard DLL (MSVCRT4 or MSVCRT.DLL)- yet Martin v. Löwis somehow explained in
http://groups.google.de/group/comp.lang.python/msg/fcbe41f9df595c35
somehow that MSVCRT.dll is not intended for normal apps - only for "system-level components".

Now I'm stun that MingW by default works with that library by default!
Another reason for my former request (in addition to ease of use with other compilers) was, that you don't have to package the big CRT7/8.. libs with frozen apps.


Robert

 

[Robert Kern]

In past I asked for linking Python2.5 and next Pythons on Win against such standard DLL (MSVCRT4 or MSVCRT.DLL)- yet Martin v. Löwis somehow explained in
http://groups.google.de/group/comp.lang.python/msg/fcbe41f9df595c35
somehow that MSVCRT.dll is not intended for normal apps - only for "system-level components".

Now I'm stun that MingW by default works with that library by default!

This is what Martin said:

"""
It used to be possible to link with it. See

http://msdn2.microsoft.com/en-us/library/abx4dbyh(VS.80).aspx

This is now a "known DLL", and meant for use by system-level components
only.
"""

Note the words "used to" and "now". Mingw is a somewhat old project and was
initially developed at a time when MSVCRT.dll *was* intended to be used by
applications. It simply hasn't received enough attention recently to move to a
different runtime. If you would like to volunteer your time to do the necessary
work to allow it to link to modern runtimes, please do so. You will receive the
undying thanks of many, many Pythoneers.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco