suitability of python

[Rudra Banerjee]

Dear friends,
I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation?
Any comment or reference is welcome.

 

[Laurent Claessens]

Le 24/11/2011 13:31, Rudra Banerjee a écrit :

Dear friends,
I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation?
Any comment or reference is welcome.

If you need mathematical power (especially symbolic computations), you
should also consider Sage[1] which is kind of a module of math over
python. In some situations, Sage is the "correct" successor of Fortran
instead of plain python.

Well, it does not answers the question, but ...

Laurent

[1] http://sagemath.org

 

[Dave Angel]

Dear friends,
I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation?
Any comment or reference is welcome.

If I take your description at face value, then I'd say that stock
CPython would be slower than Fortran. If the CPU-intensive parts had to
be rewritten in CPython, they'd be slower than the Fortran they replace,
by somewhere between 10:1 and 500:1. Further, if you've already got
those Fortran algorithms written and debugged, why rewrite them? And
finally, even for new code, you might be getting ideas for your
algorithms from journals and other resources, where the examples may
well be done in Fortran, so productivity might be best in Fortran as well.

HOWEVER, you don't have to use stock CPython, alone. It could be that
some of your Fortran algorithms are written in shared libraries, and
that you could get your CPython code to call them to do the "heavy
lifting." Or it could be that numpy, sage, or other 3rd party libraries
might be usable for your particular problems, and that speed is then
comparable to Fortran. Or it could be that one of the alternative
Python implementations might be fast enough.

Or it could even be that you're mistaken that the present code is even
CPU intensive.

Or it could be that by the time you recode the problem in Python, you
discover a more efficient algorithm, and that way gain back all the
speed you theoretically lost.

There are tools to measure things, though I'm not the one to recommend
specifics. And those probably depend on your platform as well.

The last Fortran that I wrote was over 40 years ago. I'm afraid when I
need speed, I usually use C++. But if I were starting a personal
math-intensive project now, I'd try to prototype it in Python, and only
move portions of it to Fortran or other compiled language. Only the
portions that measurably took too long. And those portions might be
rewritten in Cython, C++, or Fortran, depending on what kind of work
they actually did.

Another alternative that might make sense is to use Python as a "macro
language" to Fortran, where you call out to Python to automate some
tasks within the main program. I have no experience with doing that,
but I assume it'd be something like how MSWord can call out to VBA
routines. And it'd make the most sense when the main app is already
written, and the macro stuff is an afterthought.

I think the real point is that it doesn't have to be "all or nothing."
I suspect that the pieces you're already doing in Python are calling out
to pre-existing libraries as well. So your plotting code does some
massaging, and then calls into some plotting library, or even execs a
plotting executable.

 

[Terry Reedy]

Dear friends,
I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation?

The first killer app for Python was running Fortran code from within
Python. People use Python for both pre- and post-processing. For small
jobs, this enabled running Fortran interactively.

This lead to Numerical Python, now Numpy, SciPy, and later Sage and
other scientific and Python packages. I believe SciPy has an f2py
(fortran to py) module to help with running Fortran under Python (but it
has been years since I read the details).

Detailed questions might get better answers on, for instance, a scipy list.

 

[88888 Dihedral]

The first killer app for Python was running Fortran code from within
Python. People use Python for both pre- and post-processing. For small
jobs, this enabled running Fortran interactively.

This lead to Numerical Python, now Numpy, SciPy, and later Sage and
other scientific and Python packages. I believe SciPy has an f2py
(fortran to py) module to help with running Fortran under Python (but it
has been years since I read the details).

Detailed questions might get better answers on, for instance, a scipy list.

If pyhthon just handles the user interface and glue logics of well written python modules that are most written c, the speed of running python pyc is OK.

Of course the object reference updating required in OOP is completely supported by python.

 

[88888 Dihedral]

The first killer app for Python was running Fortran code from within
Python. People use Python for both pre- and post-processing. For small
jobs, this enabled running Fortran interactively.

This lead to Numerical Python, now Numpy, SciPy, and later Sage and
other scientific and Python packages. I believe SciPy has an f2py
(fortran to py) module to help with running Fortran under Python (but it
has been years since I read the details).

Detailed questions might get better answers on, for instance, a scipy list.

If pyhthon just handles the user interface and glue logics of well written python modules that are most written c, the speed of running python pyc is OK.

Of course the object reference updating required in OOP is completely supported by python.

 

[alex23]

This lead to Numerical Python, now Numpy, SciPy, and later Sage and
other scientific and Python packages. I believe SciPy has an f2py
(fortran to py) module to help with running Fortran under Python (but it
has been years since I read the details).

Andrew Dalke recently did some work on f2pypy, as a step toward
running Fortran under PyPy:

http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html

If PyPy's Numpy support was more advanced, I'd probably recommend the
OP start there.

 

[Alan Meyer]

Dear friends,
I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation?
Any comment or reference is welcome.

I would expect that a language that compiles intensive math programming
to machine language will be much more than an order of magnitude faster
than a program that does the same thing by interpreting byte code.

If you study all of the Python math libraries I'm guessing you'll find
modules that do a lot, conceivably all, of what you want in compiled
machine language, but when held together with Python it may or may not
be as efficient as fortran. I'm guessing there's not much out there
that is as efficient as fortran for purely numerical work.

I think your division of labor using fortran for the CPU intensive math
parts and python for post-processing is a pretty good one. It takes
advantage of the strength of each language. In addition, it completely
separates the two parts so that they aren't really dependent on each
other. You can change the fortran any way you want without breaking the
python code as long as you output the same format, and of course you can
change the python any way you want. Programs in each language don't even
have to know that any other language is involved.

My only suggestion is to see if you can get a profiler to see what's
happening inside that weeks long running fortran program. You might
find some surprises. I once wrote a 5,000 line program that was slower
than I had hoped. I ran it through a profiler and it showed me that I
was spending more than 50 percent of my time on one single line of my
code that called a simple library routine ("strcpy"). I wrote the
simple library routine inline instead adding just a few lines of code.
It cut the total execution time of the whole program in half.

Alan

 

[Grant Edwards]

I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation?

You'll have to tell us what "these type of heavy computation" are
before we can answer. There are a _lot_ of heavy-duty computational
libraries (many of them written in FORTAN) that have been interfaced
to Python (BLAS and so on). If the heavy lifting can be done by those
libraries, Python might be very suitable.

You might want to check out scipy, Scientific Python, and the
Enthought python distro.

http://www.scipy.org/
http://dirac.cnrs-orleans.fr/plone/software/scientificpython/overview/
http://www.enthought.com/products/epd.php

 

[Stefan Behnel]

Rudra Banerjee, 24.11.2011 12:31:

I am a newbie in python and basically i use python for postprocessing
like plotting, data manipulation etc.
Based on ease of programming on python I am wondering if I can consider
it for the main development as well. My jobs (written on fortran) runs
for weeks and quite CPU intensive. How python works on these type of
heavy computation?

You already got a lot of answers that pointed you to the scientific
computing tools that are available for Python. The reason why they exist is
because (and nowadays also "why") Python is so extremely popular in that
field: it's an easy to learn and use language and the standard
implementation (often referred to as CPython) makes it really easy to
interface with external code (C/C++/Fortran/etc.) in a very efficient way.

In addition to looking at NumPy/SciPy and/or Sage (depending on the kind of
computations you are involved with), you should also look at fwrap and
Cython. They will allow you to easily wrap your existing Fortran code for
Python, and to quickly write very fast glue code for the two language
environments. Thus, you can keep your existing code as it is, and use and
control it from Python, using all the nice tools that Python provides for
quickly writing anything from distributed code and test suites to graphical
user interfaces for visualising your data. Since you specifically asked
about plotting, don't miss out on matplotlib.

Stefan