suitability of python

Discussion in 'Python' started by Rudra Banerjee, Nov 24, 2011.

  1. Dear friends,
    I am a newbie in python and basically i use python for postprocessing
    like plotting, data manipulation etc.
    Based on ease of programming on python I am wondering if I can consider
    it for the main development as well. My jobs (written on fortran) runs
    for weeks and quite CPU intensive. How python works on these type of
    heavy computation?
    Any comment or reference is welcome.
    Rudra Banerjee, Nov 24, 2011
    #1
    1. Advertising

  2. Le 24/11/2011 13:31, Rudra Banerjee a écrit :
    > Dear friends,
    > I am a newbie in python and basically i use python for postprocessing
    > like plotting, data manipulation etc.
    > Based on ease of programming on python I am wondering if I can consider
    > it for the main development as well. My jobs (written on fortran) runs
    > for weeks and quite CPU intensive. How python works on these type of
    > heavy computation?
    > Any comment or reference is welcome.



    If you need mathematical power (especially symbolic computations), you
    should also consider Sage[1] which is kind of a module of math over
    python. In some situations, Sage is the "correct" successor of Fortran
    instead of plain python.

    Well, it does not answers the question, but ...

    Laurent

    [1] http://sagemath.org
    Laurent Claessens, Nov 24, 2011
    #2
    1. Advertising

  3. Rudra Banerjee

    Dave Angel Guest

    On 11/24/2011 07:31 AM, Rudra Banerjee wrote:
    > Dear friends,
    > I am a newbie in python and basically i use python for postprocessing
    > like plotting, data manipulation etc.
    > Based on ease of programming on python I am wondering if I can consider
    > it for the main development as well. My jobs (written on fortran) runs
    > for weeks and quite CPU intensive. How python works on these type of
    > heavy computation?
    > Any comment or reference is welcome.
    >

    If I take your description at face value, then I'd say that stock
    CPython would be slower than Fortran. If the CPU-intensive parts had to
    be rewritten in CPython, they'd be slower than the Fortran they replace,
    by somewhere between 10:1 and 500:1. Further, if you've already got
    those Fortran algorithms written and debugged, why rewrite them? And
    finally, even for new code, you might be getting ideas for your
    algorithms from journals and other resources, where the examples may
    well be done in Fortran, so productivity might be best in Fortran as well.

    HOWEVER, you don't have to use stock CPython, alone. It could be that
    some of your Fortran algorithms are written in shared libraries, and
    that you could get your CPython code to call them to do the "heavy
    lifting." Or it could be that numpy, sage, or other 3rd party libraries
    might be usable for your particular problems, and that speed is then
    comparable to Fortran. Or it could be that one of the alternative
    Python implementations might be fast enough.

    Or it could even be that you're mistaken that the present code is even
    CPU intensive.

    Or it could be that by the time you recode the problem in Python, you
    discover a more efficient algorithm, and that way gain back all the
    speed you theoretically lost.

    There are tools to measure things, though I'm not the one to recommend
    specifics. And those probably depend on your platform as well.

    The last Fortran that I wrote was over 40 years ago. I'm afraid when I
    need speed, I usually use C++. But if I were starting a personal
    math-intensive project now, I'd try to prototype it in Python, and only
    move portions of it to Fortran or other compiled language. Only the
    portions that measurably took too long. And those portions might be
    rewritten in Cython, C++, or Fortran, depending on what kind of work
    they actually did.

    Another alternative that might make sense is to use Python as a "macro
    language" to Fortran, where you call out to Python to automate some
    tasks within the main program. I have no experience with doing that,
    but I assume it'd be something like how MSWord can call out to VBA
    routines. And it'd make the most sense when the main app is already
    written, and the macro stuff is an afterthought.

    I think the real point is that it doesn't have to be "all or nothing."
    I suspect that the pieces you're already doing in Python are calling out
    to pre-existing libraries as well. So your plotting code does some
    massaging, and then calls into some plotting library, or even execs a
    plotting executable.

    --

    DaveA
    Dave Angel, Nov 24, 2011
    #3
  4. Rudra Banerjee

    Terry Reedy Guest

    On 11/24/2011 7:31 AM, Rudra Banerjee wrote:
    > Dear friends,
    > I am a newbie in python and basically i use python for postprocessing
    > like plotting, data manipulation etc.
    > Based on ease of programming on python I am wondering if I can consider
    > it for the main development as well. My jobs (written on fortran) runs
    > for weeks and quite CPU intensive. How python works on these type of
    > heavy computation?


    The first killer app for Python was running Fortran code from within
    Python. People use Python for both pre- and post-processing. For small
    jobs, this enabled running Fortran interactively.

    This lead to Numerical Python, now Numpy, SciPy, and later Sage and
    other scientific and Python packages. I believe SciPy has an f2py
    (fortran to py) module to help with running Fortran under Python (but it
    has been years since I read the details).

    Detailed questions might get better answers on, for instance, a scipy list.

    --
    Terry Jan Reedy
    Terry Reedy, Nov 25, 2011
    #4
  5. On Friday, November 25, 2011 8:51:10 AM UTC+8, Terry Reedy wrote:
    > On 11/24/2011 7:31 AM, Rudra Banerjee wrote:
    > > Dear friends,
    > > I am a newbie in python and basically i use python for postprocessing
    > > like plotting, data manipulation etc.
    > > Based on ease of programming on python I am wondering if I can consider
    > > it for the main development as well. My jobs (written on fortran) runs
    > > for weeks and quite CPU intensive. How python works on these type of
    > > heavy computation?

    >
    > The first killer app for Python was running Fortran code from within
    > Python. People use Python for both pre- and post-processing. For small
    > jobs, this enabled running Fortran interactively.
    >
    > This lead to Numerical Python, now Numpy, SciPy, and later Sage and
    > other scientific and Python packages. I believe SciPy has an f2py
    > (fortran to py) module to help with running Fortran under Python (but it
    > has been years since I read the details).
    >
    > Detailed questions might get better answers on, for instance, a scipy list.
    >
    > --
    > Terry Jan Reedy


    If pyhthon just handles the user interface and glue logics of well written python modules that are most written c, the speed of running python pyc is OK.

    Of course the object reference updating required in OOP is completely supported by python.
    88888 Dihedral, Nov 25, 2011
    #5
  6. On Friday, November 25, 2011 8:51:10 AM UTC+8, Terry Reedy wrote:
    > On 11/24/2011 7:31 AM, Rudra Banerjee wrote:
    > > Dear friends,
    > > I am a newbie in python and basically i use python for postprocessing
    > > like plotting, data manipulation etc.
    > > Based on ease of programming on python I am wondering if I can consider
    > > it for the main development as well. My jobs (written on fortran) runs
    > > for weeks and quite CPU intensive. How python works on these type of
    > > heavy computation?

    >
    > The first killer app for Python was running Fortran code from within
    > Python. People use Python for both pre- and post-processing. For small
    > jobs, this enabled running Fortran interactively.
    >
    > This lead to Numerical Python, now Numpy, SciPy, and later Sage and
    > other scientific and Python packages. I believe SciPy has an f2py
    > (fortran to py) module to help with running Fortran under Python (but it
    > has been years since I read the details).
    >
    > Detailed questions might get better answers on, for instance, a scipy list.
    >
    > --
    > Terry Jan Reedy


    If pyhthon just handles the user interface and glue logics of well written python modules that are most written c, the speed of running python pyc is OK.

    Of course the object reference updating required in OOP is completely supported by python.
    88888 Dihedral, Nov 25, 2011
    #6
  7. Rudra Banerjee

    alex23 Guest

    Terry Reedy <> wrote:
    > This lead to Numerical Python, now Numpy, SciPy, and later Sage and
    > other scientific and Python packages. I believe SciPy has an f2py
    > (fortran to py) module to help with running Fortran under Python (but it
    > has been years since I read the details).


    Andrew Dalke recently did some work on f2pypy, as a step toward
    running Fortran under PyPy:

    http://www.dalkescientific.com/writings/diary/archive/2011/11/09/f2pypy.html

    If PyPy's Numpy support was more advanced, I'd probably recommend the
    OP start there.
    alex23, Nov 25, 2011
    #7
  8. Rudra Banerjee

    Alan Meyer Guest

    On 11/24/2011 07:31 AM, Rudra Banerjee wrote:
    > Dear friends,
    > I am a newbie in python and basically i use python for postprocessing
    > like plotting, data manipulation etc.
    > Based on ease of programming on python I am wondering if I can consider
    > it for the main development as well. My jobs (written on fortran) runs
    > for weeks and quite CPU intensive. How python works on these type of
    > heavy computation?
    > Any comment or reference is welcome.
    >


    I would expect that a language that compiles intensive math programming
    to machine language will be much more than an order of magnitude faster
    than a program that does the same thing by interpreting byte code.

    If you study all of the Python math libraries I'm guessing you'll find
    modules that do a lot, conceivably all, of what you want in compiled
    machine language, but when held together with Python it may or may not
    be as efficient as fortran. I'm guessing there's not much out there
    that is as efficient as fortran for purely numerical work.

    I think your division of labor using fortran for the CPU intensive math
    parts and python for post-processing is a pretty good one. It takes
    advantage of the strength of each language. In addition, it completely
    separates the two parts so that they aren't really dependent on each
    other. You can change the fortran any way you want without breaking the
    python code as long as you output the same format, and of course you can
    change the python any way you want. Programs in each language don't even
    have to know that any other language is involved.

    My only suggestion is to see if you can get a profiler to see what's
    happening inside that weeks long running fortran program. You might
    find some surprises. I once wrote a 5,000 line program that was slower
    than I had hoped. I ran it through a profiler and it showed me that I
    was spending more than 50 percent of my time on one single line of my
    code that called a simple library routine ("strcpy"). I wrote the
    simple library routine inline instead adding just a few lines of code.
    It cut the total execution time of the whole program in half.

    Alan
    Alan Meyer, Nov 25, 2011
    #8
  9. On 2011-11-24, Rudra Banerjee <> wrote:

    > I am a newbie in python and basically i use python for postprocessing
    > like plotting, data manipulation etc.
    > Based on ease of programming on python I am wondering if I can consider
    > it for the main development as well. My jobs (written on fortran) runs
    > for weeks and quite CPU intensive. How python works on these type of
    > heavy computation?


    You'll have to tell us what "these type of heavy computation" are
    before we can answer. There are a _lot_ of heavy-duty computational
    libraries (many of them written in FORTAN) that have been interfaced
    to Python (BLAS and so on). If the heavy lifting can be done by those
    libraries, Python might be very suitable.

    You might want to check out scipy, Scientific Python, and the
    Enthought python distro.

    http://www.scipy.org/
    http://dirac.cnrs-orleans.fr/plone/software/scientificpython/overview/
    http://www.enthought.com/products/epd.php
    Grant Edwards, Nov 27, 2011
    #9
  10. Rudra Banerjee, 24.11.2011 12:31:
    > I am a newbie in python and basically i use python for postprocessing
    > like plotting, data manipulation etc.
    > Based on ease of programming on python I am wondering if I can consider
    > it for the main development as well. My jobs (written on fortran) runs
    > for weeks and quite CPU intensive. How python works on these type of
    > heavy computation?


    You already got a lot of answers that pointed you to the scientific
    computing tools that are available for Python. The reason why they exist is
    because (and nowadays also "why") Python is so extremely popular in that
    field: it's an easy to learn and use language and the standard
    implementation (often referred to as CPython) makes it really easy to
    interface with external code (C/C++/Fortran/etc.) in a very efficient way.

    In addition to looking at NumPy/SciPy and/or Sage (depending on the kind of
    computations you are involved with), you should also look at fwrap and
    Cython. They will allow you to easily wrap your existing Fortran code for
    Python, and to quickly write very fast glue code for the two language
    environments. Thus, you can keep your existing code as it is, and use and
    control it from Python, using all the nice tools that Python provides for
    quickly writing anything from distributed code and test suites to graphical
    user interfaces for visualising your data. Since you specifically asked
    about plotting, don't miss out on matplotlib.

    Stefan
    Stefan Behnel, Nov 27, 2011
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Sanjay Minni

    XML suitability

    Sanjay Minni, Jul 15, 2005, in forum: XML
    Replies:
    1
    Views:
    431
    =?ISO-8859-1?Q?J=FCrgen_Kahrs?=
    Jul 15, 2005
  2. Replies:
    11
    Views:
    589
    Tom Anderson
    Sep 21, 2005
  3. Replies:
    0
    Views:
    740
  4. Michael R. Copeland

    Python's Suitability?

    Michael R. Copeland, Aug 28, 2007, in forum: Python
    Replies:
    2
    Views:
    326
    Nick Craig-Wood
    Aug 28, 2007
  5. Replies:
    2
    Views:
    420
    Roedy Green
    Apr 23, 2008
Loading...

Share This Page