Python in sci/tech applications

Discussion in 'Python' started by mattf, Nov 2, 2006.

  1. mattf

    mattf Guest

    I've discovered Python and have been trying it out lately as a possible
    replacement for computations that would ordinarily be done with a
    commercial package like Matlab or IDL. I'd like to mention a few things
    I've run across that have either surprised me or kept me from doing
    things the way I'd like to.

    1) -There's a large and active sci/tech Python community out there.-
    This was something of a surprise. If you look at the python.org site
    and click down a couple of levels past the front page, there's a rather
    brief mention of scientific and numeric applications-- but I don't
    think this does justice to the current levels of activity and
    accomplishment.

    2) -There's a very impressive set of libraries out there-
    NumPy, SciPy, Enthought. It's really kind of stunning how mature these
    libraries are and how much I had to poke around to figure that out.

    3) -There's a problem with development under Windows.
    A typical task will entail writing a 'pure python' prototype to get the
    'data in, data out' part of a problem straightened out, then writing a
    module in C to get adequate performance in production runs. But the C
    compiler that my employer provides (the current version of MSVS)
    doesn't produce libraries that work with the current version of Python.
    Ooops. This, in the real world, is a big problem. I -love- Python. And
    I think I could convince other people to use it. But I've got to have a
    way to produce compiled modules painlessly, i.e., without installing a
    new operating system.
     
    mattf, Nov 2, 2006
    #1
    1. Advertising

  2. mattf

    Paul Boddie Guest

    mattf wrote:
    > I'd like to mention a few things I've run across that have either surprised me or kept me
    > from doing things the way I'd like to.
    >
    > 1) -There's a large and active sci/tech Python community out there.-
    > This was something of a surprise. If you look at the python.org site
    > and click down a couple of levels past the front page, there's a rather
    > brief mention of scientific and numeric applications-- but I don't
    > think this does justice to the current levels of activity and
    > accomplishment.


    If you want an in-depth impression of how widely Python is used,
    python.org is not really the place that manages to provide it. Sure,
    "NASA uses Python" (appropriate response: dressed in a flight suit,
    leap up and punch the air?) but isn't OLPC (One Laptop per Child:
    http://www.laptop.org/) worth a mention, given the central role of
    Python in a device that will potentially reach millions? To the site's
    credit, the link to scientific and numeric applications does yield a
    lot of information, and I suppose it's hard to focus the site on any
    one thing that any given group considers of utmost importance.

    However, I think you're going to find that the python.org worldview can
    be somewhat alien, especially if your primary interest isn't Python
    (the language). To take another domain as an example, if you do desktop
    development, the treatment of graphical user interface programming on
    the site is quite possibly going to be completely disaligned with what
    you'd expect: Python is used extensively in various desktop
    environments (not that you'd get that impression from a cursory
    examination of python.org), but the python.org (or comp.lang.python)
    worldview mostly excludes such things as if the desktop is something
    you abstract away rather than take advantage of.

    And on the subject of how well "core Pythoneers" know of Python's
    successes: how many people knew that Python was used in the OLPC device
    before Alan Kay mentioned it at EuroPython? It's hard to promote Python
    when you don't know where it's being successfully used already.

    > 2) -There's a very impressive set of libraries out there-
    > NumPy, SciPy, Enthought. It's really kind of stunning how mature these
    > libraries are and how much I had to poke around to figure that out.


    I guess a lot of the poking around comes about because people are
    reluctant to provide any semi-official guide to the best libraries, and
    that the most promotion occurs for the most fashionable products in the
    most fashionable domains. Some of us have tried to provide better
    information about libraries for Python, but such details remain
    relatively hidden in the python.org Wiki:

    http://wiki.python.org/moin/UsefulModules

    With respect to content on python.org, I think all that remains is an
    acknowledgement that certain resources are never going to return to
    their original vitality, and that alternative mechanisms for publishing
    content are required. For example, the XML topic guide surely hasn't
    seen many updates in a very long time, and despite a reasonable level
    of maintenance, the database topic guide could surely be more actively
    maintained by aggregating package information from the package index.

    Paul
     
    Paul Boddie, Nov 2, 2006
    #2
    1. Advertising

  3. mattf

    Guest

    mattf:
    > 3) -There's a problem with development under Windows.


    It's possibile to compile Python with MinGW, and to create extensions
    with it. So some site can host a single zip file that contains both
    MinGW and Python compiled with it, all ready and set. A person not much
    expert can then create compiled small extensions in a short time
    without too much complexities. (Or maybe the D language can be packed
    into that, instead of MinGW, to do similar things. D can be a good
    language used with Python) But then probably PIL, scipy, etc have to be
    compiled again for such alternative official or semi-official Python
    distribution.

    Bye,
    bearophile
     
    , Nov 2, 2006
    #3
  4. mattf

    Robert Kern Guest

    wrote:
    > mattf:
    >> 3) -There's a problem with development under Windows.

    >
    > It's possibile to compile Python with MinGW, and to create extensions
    > with it. So some site can host a single zip file that contains both
    > MinGW and Python compiled with it, all ready and set. A person not much
    > expert can then create compiled small extensions in a short time
    > without too much complexities. (Or maybe the D language can be packed
    > into that, instead of MinGW, to do similar things. D can be a good
    > language used with Python) But then probably PIL, scipy, etc have to be
    > compiled again for such alternative official or semi-official Python
    > distribution.


    With a few caveats, mingw will simply work with the standard 2.4 (and presumably
    2.5) interpreter. We distribute mingw set up to do this with our "Enthought
    Edition" Python distribution.

    http://code.enthought.com/enthon/

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Nov 2, 2006
    #4
  5. mattf

    robert Guest

    wrote:
    > mattf:
    >> 3) -There's a problem with development under Windows.

    >
    > It's possibile to compile Python with MinGW, and to create extensions
    > with it. So some site can host a single zip file that contains both
    > MinGW and Python compiled with it, all ready and set. A person not much
    > expert can then create compiled small extensions in a short time
    > without too much complexities. (Or maybe the D language can be packed
    > into that, instead of MinGW, to do similar things. D can be a good
    > language used with Python) But then probably PIL, scipy, etc have to be
    > compiled again for such alternative official or semi-official Python
    > distribution.
    >


    Is it really not possible to create extension libs with older MSVC or Mingw, which work with regular Python binaries version 2.4 and 2.5 ?
    Maybe with some special import libraries?

    (another option is to stay with Python2.3, which has a small memory footprint and compiles with crtl4.2 libs (VC6, ...) )

    -robert
     
    robert, Nov 2, 2006
    #5
  6. mattf

    Guest

    Robert Kern:
    > We distribute mingw set up to do this with our "Enthought
    > Edition" Python distribution.
    > http://code.enthought.com/enthon/


    Sorry, maybe I'm blind but I don't see MinGW listed in that page...
    Maybe it's included but not listed...

    Bye,
    bearophile
     
    , Nov 2, 2006
    #6
  7. robert wrote:

    > Is it really not possible to create extension libs with
    > older MSVC or Mingw, which work with regular Python binaries
    > version 2.4 and 2.5 ?


    last time I tried, it took me 20 minutes from that I typed "mingw" into
    google until I had built and tested my first non-trivial extension. your
    milage may vary.

    </F>
     
    Fredrik Lundh, Nov 2, 2006
    #7
  8. mattf

    Robert Kern Guest

    wrote:
    > Robert Kern:
    >> We distribute mingw set up to do this with our "Enthought
    >> Edition" Python distribution.
    >> http://code.enthought.com/enthon/

    >
    > Sorry, maybe I'm blind but I don't see MinGW listed in that page...
    > Maybe it's included but not listed...


    It's there.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Nov 2, 2006
    #8
  9. mattf

    robert Guest

    Fredrik Lundh wrote:
    > robert wrote:
    >
    >> Is it really not possible to create extension libs with
    > > older MSVC or Mingw, which work with regular Python binaries
    > > version 2.4 and 2.5 ?

    >
    > last time I tried, it took me 20 minutes from that I typed "mingw" into
    > google until I had built and tested my first non-trivial extension. your
    > milage may vary.


    thats far beyond my dexterity.

    When one follows ..
    http://docs.python.org/inst/tweak-flags.html#SECTION000622000000000000000
    http://www.zope.org/Members/als/tips/win32_mingw_modules

    ...this seems only to cover the immediate python dll issues. What happens with the C runtime libraries? You'll bind 2 different C-RTLs (DLLs) etc.?
    And what happens for example with heap objects created with one C-RTL and deleted/free'd with the other?

    -robert
     
    robert, Nov 2, 2006
    #9
  10. mattf

    Guest

    Fredrik Lundh:
    > last time I tried, it took me 20 minutes from that I typed "mingw" into
    > google until I had built and tested my first non-trivial extension. your
    > milage may vary.


    But probably before those 20 minutes there is a lot of time of
    experience of yours with CPython sources, other compilers, and so on,
    so for most people this timing comparison just means "It can be done"
    :)

    Bye,
    bearophile
     
    , Nov 2, 2006
    #10
  11. mattf

    Steve Holden Guest

    Robert Kern wrote:
    > wrote:
    >
    >>Robert Kern:
    >>
    >>>We distribute mingw set up to do this with our "Enthought
    >>>Edition" Python distribution.
    >>>http://code.enthought.com/enthon/

    >>
    >>Sorry, maybe I'm blind but I don't see MinGW listed in that page...
    >>Maybe it's included but not listed...

    >
    >
    > It's there.
    >

    Well I tend to believe you, but Firefox is calling you a liar: a search
    fails at "min".

    Do you anticipate a 2.5-based release any time soon?

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://holdenweb.blogspot.com
    Recent Ramblings http://del.icio.us/steve.holden
     
    Steve Holden, Nov 2, 2006
    #11
  12. mattf

    Robert Kern Guest

    Steve Holden wrote:
    > Robert Kern wrote:
    >> wrote:
    >>
    >>> Robert Kern:
    >>>
    >>>> We distribute mingw set up to do this with our "Enthought
    >>>> Edition" Python distribution.
    >>>> http://code.enthought.com/enthon/
    >>> Sorry, maybe I'm blind but I don't see MinGW listed in that page...
    >>> Maybe it's included but not listed...

    >>
    >> It's there.
    >>

    > Well I tend to believe you, but Firefox is calling you a liar: a search
    > fails at "min".


    Sorry, by "it's there" I meant "it is actually included in the distribution
    regardless of any omissions on the web page."

    > Do you anticipate a 2.5-based release any time soon?


    We'll start working on it soon, at least.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Nov 2, 2006
    #12
  13. mattf

    John Coleman Guest

    Maybe I don't know what I'm looking for, but I downloaded Enthought a
    few days ago and don't seem to find MinGW on my system. There are 2
    relatively small (totalling about 13 kb IIRC) *python* files deeply
    buried in the distribution with mingw in their filename but nothing
    like a gcc compiler. I've mostly used visual studio or code warrior for
    C compiling, so again - maybe I don't know what I am looking for.

    -John Coleman

    p.s. Pretty cool sig line - the older I get, the more enigmatic the
    world seems. I appreciate all of the effort that Enthought puts into
    their distribution.

    Robert Kern wrote:
    > Steve Holden wrote:
    > > Robert Kern wrote:
    > >> wrote:
    > >>
    > >>> Robert Kern:
    > >>>
    > >>>> We distribute mingw set up to do this with our "Enthought
    > >>>> Edition" Python distribution.
    > >>>> http://code.enthought.com/enthon/
    > >>> Sorry, maybe I'm blind but I don't see MinGW listed in that page...
    > >>> Maybe it's included but not listed...
    > >>
    > >> It's there.
    > >>

    > > Well I tend to believe you, but Firefox is calling you a liar: a search
    > > fails at "min".

    >
    > Sorry, by "it's there" I meant "it is actually included in the distribution
    > regardless of any omissions on the web page."
    >
    > > Do you anticipate a 2.5-based release any time soon?

    >
    > We'll start working on it soon, at least.
    >
    > --
    > Robert Kern
    >
    > "I have come to believe that the whole world is an enigma, a harmless enigma
    > that is made terrible by our own mad attempt to interpret it as though it had
    > an underlying truth."
    > -- Umberto Eco
     
    John Coleman, Nov 2, 2006
    #13
  14. mattf

    Robert Kern Guest

    John Coleman wrote:
    > Maybe I don't know what I'm looking for, but I downloaded Enthought a
    > few days ago and don't seem to find MinGW on my system. There are 2
    > relatively small (totalling about 13 kb IIRC) *python* files deeply
    > buried in the distribution with mingw in their filename but nothing
    > like a gcc compiler. I've mostly used visual studio or code warrior for
    > C compiling, so again - maybe I don't know what I am looking for.


    It is in c:\Python24\Enthought\mingw32\, IIRC (I don't have it in front of me).
    Add c:\Python24\Enthought\mingw32\bin\ to your PATH environment variable, and
    gcc.exe should be executable.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Nov 2, 2006
    #14
  15. mattf

    John Coleman Guest

    Robert Kern wrote:
    > John Coleman wrote:
    > > Maybe I don't know what I'm looking for, but I downloaded Enthought a
    > > few days ago and don't seem to find MinGW on my system. There are 2
    > > relatively small (totalling about 13 kb IIRC) *python* files deeply
    > > buried in the distribution with mingw in their filename but nothing
    > > like a gcc compiler. I've mostly used visual studio or code warrior for
    > > C compiling, so again - maybe I don't know what I am looking for.

    >
    > It is in c:\Python24\Enthought\mingw32\, IIRC (I don't have it in front of me).
    > Add c:\Python24\Enthought\mingw32\bin\ to your PATH environment variable, and
    > gcc.exe should be executable.
    >

    I have no such sub-directory. I'll try an uninstall-install. When I did
    the original installation the installer hung at the end and I had to
    manually close it. The Python interpreter was there and worked properly
    and all of the goodies like numby and chaco were there so I wrote it
    off as nothing more then a failure to close properly, but evidently I
    missed an essential part of the installation.

    -John Coleman


    > --
    > Robert Kern
    >
    > "I have come to believe that the whole world is an enigma, a harmless enigma
    > that is made terrible by our own mad attempt to interpret it as though it had
    > an underlying truth."
    > -- Umberto Eco
     
    John Coleman, Nov 2, 2006
    #15
  16. mattf

    Magnus Lycka Guest

    robert wrote:
    > When one follows ..
    > http://docs.python.org/inst/tweak-flags.html#SECTION000622000000000000000
    > http://www.zope.org/Members/als/tips/win32_mingw_modules
    >
    > ..this seems only to cover the immediate python dll issues. What happens
    > with the C runtime libraries? You'll bind 2 different C-RTLs (DLLs)
    > etc.? And what happens for example with heap objects created with one
    > C-RTL and deleted/free'd with the other?


    As far as I understand, there should be only one runtime library
    involved. MinGW relies on the Microsoft runtime. It seems some
    people have had problems using MSVCRT71 with MinGW, but I'm sure
    that e.g. the Enthought people can explain those issues, since they
    bundle MinGW in their Python 2.4 distribution.
     
    Magnus Lycka, Nov 3, 2006
    #16
  17. mattf

    John Coleman Guest

    John Coleman wrote:
    > Robert Kern wrote:
    > > John Coleman wrote:
    > > > Maybe I don't know what I'm looking for, but I downloaded Enthought a
    > > > few days ago and don't seem to find MinGW on my system. There are 2
    > > > relatively small (totalling about 13 kb IIRC) *python* files deeply
    > > > buried in the distribution with mingw in their filename but nothing
    > > > like a gcc compiler. I've mostly used visual studio or code warrior for
    > > > C compiling, so again - maybe I don't know what I am looking for.

    > >
    > > It is in c:\Python24\Enthought\mingw32\, IIRC (I don't have it in front of me).
    > > Add c:\Python24\Enthought\mingw32\bin\ to your PATH environment variable, and
    > > gcc.exe should be executable.
    > >

    > I have no such sub-directory. I'll try an uninstall-install. When I did
    > the original installation the installer hung at the end and I had to
    > manually close it. The Python interpreter was there and worked properly
    > and all of the goodies like numby and chaco were there so I wrote it
    > off as nothing more then a failure to close properly, but evidently I
    > missed an essential part of the installation.
    >
    > -John Coleman


    Problem fixed. I had clicked on the 10-05-06 download (thinking that
    enthought_setup-1.1.0.msi led to the set up the full enthought
    package), but this is evidently only a partial download. I tried it a
    second time with the same results, but perhaps I wasn't using the
    "browse and install" option correctly (clicking that button on the
    enstaller caused a lot of files to be transfered, but there didn't seem
    to be any real list to browse). In any event, the August download
    worked like a charm and had it all, including mingw. Thanks for your
    help.

    -John Coleman
     
    John Coleman, Nov 3, 2006
    #17
  18. mattf

    Robert Kern Guest

    Magnus Lycka wrote:
    > robert wrote:
    >> When one follows ..
    >> http://docs.python.org/inst/tweak-flags.html#SECTION000622000000000000000
    >> http://www.zope.org/Members/als/tips/win32_mingw_modules
    >>
    >> ..this seems only to cover the immediate python dll issues. What happens
    >> with the C runtime libraries? You'll bind 2 different C-RTLs (DLLs)
    >> etc.? And what happens for example with heap objects created with one
    >> C-RTL and deleted/free'd with the other?

    >
    > As far as I understand, there should be only one runtime library
    > involved. MinGW relies on the Microsoft runtime. It seems some
    > people have had problems using MSVCRT71 with MinGW, but I'm sure
    > that e.g. the Enthought people can explain those issues, since they
    > bundle MinGW in their Python 2.4 distribution.


    No, there are two if you use an unmodified Mingw distribution. Mingw is built
    around MSVCRT.dll . For many extensions, this doesn't cause a problem since
    things like FILE pointer don't cross C runtime boundaries. One can edit gcc's
    spec file to make it use MSVCR71.dll . However, Mingw's headers are still
    written for MSVCRT.dll and Microsoft changed some of the internal details that
    they rely upon. For many (most?) extensions, this won't matter, either. C++
    extensions using iostreams have a problem, IIRC. Unfortunately, neither approach
    works for all extensions. It may even be the case that some extensions won't
    work under *either* method.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Nov 3, 2006
    #18
  19. mattf

    robert Guest

    Robert Kern wrote:
    > Magnus Lycka wrote:
    >> robert wrote:
    >>> When one follows ..
    >>> http://docs.python.org/inst/tweak-flags.html#SECTION000622000000000000000
    >>> http://www.zope.org/Members/als/tips/win32_mingw_modules
    >>>
    >>> ..this seems only to cover the immediate python dll issues. What
    >>> happens with the C runtime libraries? You'll bind 2 different C-RTLs
    >>> (DLLs) etc.? And what happens for example with heap objects created
    >>> with one C-RTL and deleted/free'd with the other?

    >>
    >> As far as I understand, there should be only one runtime library
    >> involved. MinGW relies on the Microsoft runtime. It seems some
    >> people have had problems using MSVCRT71 with MinGW, but I'm sure
    >> that e.g. the Enthought people can explain those issues, since they
    >> bundle MinGW in their Python 2.4 distribution.

    >
    > No, there are two if you use an unmodified Mingw distribution. Mingw is
    > built around MSVCRT.dll . For many extensions, this doesn't cause a
    > problem since things like FILE pointer don't cross C runtime boundaries.
    > One can edit gcc's spec file to make it use MSVCR71.dll . However,
    > Mingw's headers are still written for MSVCRT.dll and Microsoft changed
    > some of the internal details that they rely upon. For many (most?)
    > extensions, this won't matter, either. C++ extensions using iostreams
    > have a problem, IIRC. Unfortunately, neither approach works for all
    > extensions. It may even be the case that some extensions won't work
    > under *either* method.


    In past I asked for linking Python2.5 and next Pythons on Win against such standard DLL (MSVCRT4 or MSVCRT.DLL)- yet Martin v. Löwis somehow explained in
    http://groups.google.de/group/comp.lang.python/msg/fcbe41f9df595c35
    somehow that MSVCRT.dll is not intended for normal apps - only for "system-level components".

    Now I'm stun that MingW by default works with that library by default!
    Another reason for my former request (in addition to ease of use with other compilers) was, that you don't have to package the big CRT7/8.. libs with frozen apps.


    Robert
     
    robert, Nov 3, 2006
    #19
  20. mattf

    Robert Kern Guest

    robert wrote:
    > In past I asked for linking Python2.5 and next Pythons on Win against such standard DLL (MSVCRT4 or MSVCRT.DLL)- yet Martin v. Löwis somehow explained in
    > http://groups.google.de/group/comp.lang.python/msg/fcbe41f9df595c35
    > somehow that MSVCRT.dll is not intended for normal apps - only for "system-level components".
    >
    > Now I'm stun that MingW by default works with that library by default!


    This is what Martin said:

    """
    It used to be possible to link with it. See

    http://msdn2.microsoft.com/en-us/library/abx4dbyh(VS.80).aspx

    This is now a "known DLL", and meant for use by system-level components
    only.
    """

    Note the words "used to" and "now". Mingw is a somewhat old project and was
    initially developed at a time when MSVCRT.dll *was* intended to be used by
    applications. It simply hasn't received enough attention recently to move to a
    different runtime. If you would like to volunteer your time to do the necessary
    work to allow it to link to modern runtimes, please do so. You will receive the
    undying thanks of many, many Pythoneers.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Nov 3, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. anonaki
    Replies:
    77
    Views:
    1,841
    Dale King
    Apr 15, 2006
  2. Stan
    Replies:
    22
    Views:
    706
    David B. Held
    Oct 18, 2003
  3. Replies:
    0
    Views:
    326
  4. Wade Leftwich

    Excellent sci-fi novel featuring Python

    Wade Leftwich, Nov 17, 2007, in forum: Python
    Replies:
    5
    Views:
    319
    Simon Brunning
    Nov 17, 2007
  5. Matjaz Bezovnik
    Replies:
    0
    Views:
    218
    Matjaz Bezovnik
    Aug 25, 2009
Loading...

Share This Page