Making a time series analysis package in python - advice or assistancesought

[Ray Tomes]

Hi Folks

I am an old codger who has much experience with computers
in the distant past before all this object oriented stuff.
Also I have loads of software in such languages as FORTRAN
and BASIC, QBASIC etc that is very useful except that it
really doesn't like to run on modern operating systems and
has hopeless graphics resolution and lack of ease of use in
some ways.

My desire is to get all the facilities available in my
old programs working in a modern platform with flexible
and high-res graphics and easy to use. Ideally I might
find some good coders that are interested in the subject
who would assist me, alternatively some help in getting
started because there is so much info and so many resources
and libraries etc that I don't know where to start.

My package will have the following capabilities:
1. Able to read time series data in a variety of formats.
2. Able to create, manipulate and save time series files.
3. Able to do vector arithmetic on time series, including
dozens of functions.
4. Loop and macro facilities to simplify repetitive stuff.
5. Flexible high-resolution graphic presentation.
6. Built in functions to include:
FFT / fourier analysis, MESA / maximum entropy spectral analysis,
multiple regression, canonical correlation etc etc etc.
I have code for all these mostly in FORTRAN, some QBASIC.

The applications of the package include:
1. Analysis of time series data from many branches of science.
2. Economic / business models including forecasting.
3. Markets, stocks, commodities forecasting.
4. Interdisciplinary causal analysis.
5. Many more

If you are seriously interested in this, then please contact
me by email at ray(at)tomes(dot)biz which is the email from
which this message was sent without the ".remove" part (anti-
spam measure).

Ray Tomes
http://ray.tomes.biz/
http://www.cyclesresearchinstitute.org/

 

[Tim Leslie]

Hi Ray,

As a first step you might want to look at numpy/scipy/matplotlib

numpy (numpy.scipy.org) provides the underlying data structures (array
and matrices among other things) you require. This will handle all
your vector stuff, reading/writing to and from files, "loop macros",
etc.

scipy (www.scipy.org) provides a set of scientific programming
libraries, including stats, fft and many other things. Have a look
around and see if it already does what you want.

matplotlib (http://matplotlib.sourceforge.net/) takes care of all your
plotting needs, and plays nice with numpy and scipy.

HTH

Tim

 

[Terry Reedy]

My package will have the following capabilities:
1. Able to read time series data in a variety of formats.
2. Able to create, manipulate and save time series files.
3. Able to do vector arithmetic on time series, including
dozens of functions.
4. Loop and macro facilities to simplify repetitive stuff.
5. Flexible high-resolution graphic presentation.
6. Built in functions to include:
FFT / fourier analysis, MESA / maximum entropy spectral analysis,
multiple regression, canonical correlation etc etc etc.
I have code for all these mostly in FORTRAN, some QBASIC.

As Tim said, work with numpy/scipy. I believe it include programs or
modules to wrap Fortran and make Fortran function accessible from Python.
I believe these are the same as what they used for some of the functions
included.

If you find some broadly useful functions missing, consider making a
contribution. I know FFT is already included but I do not know about MESA.

Terry Jan Reedy

 

[bearophileHUGS]

Ray Tomes:

My package will have the following capabilities:
1. Able to read time series data in a variety of formats.
2. Able to create, manipulate and save time series files.
3. Able to do vector arithmetic on time series, including
dozens of functions.
4. Loop and macro facilities to simplify repetitive stuff.
5. Flexible high-resolution graphic presentation.
6. Built in functions to include:
FFT / fourier analysis, MESA / maximum entropy spectral analysis,
multiple regression, canonical correlation etc etc etc.
I have code for all these mostly in FORTRAN, some QBASIC.

It seems quite doable. Beside doing vector arithmetic with SciPy, and
plotting/generating graphs with MatPlotLib, you may need a GUI toolkit,
like Tkinter (built-in), wxpython, etc:
http://pythoncard.sourceforge.net/
http://www.wxpython.org/
etc.

Designing good GUIs requires some time.

The "Loop and macro facilities" can be done in Python itself.

To use Fortran from Python:
http://cens.ioc.ee/projects/f2py2e/

Bye,
bearophile

 

[beliavsky]

Hi Folks

I am an old codger who has much experience with computers
in the distant past before all this object oriented stuff.
Also I have loads of software in such languages as FORTRAN
and BASIC, QBASIC etc that is very useful except that it
really doesn't like to run on modern operating systems and
has hopeless graphics resolution and lack of ease of use in
some ways.

The Fortran code, which I assume is Fortran 77 or earlier, should run
fine on "modern operating systems" using free (g77, g95, or gfortran)
or commercial compilers.

My desire is to get all the facilities available in my
old programs working in a modern platform with flexible
and high-res graphics and easy to use. Ideally I might
find some good coders that are interested in the subject
who would assist me, alternatively some help in getting
started because there is so much info and so many resources
and libraries etc that I don't know where to start.

My package will have the following capabilities:
1. Able to read time series data in a variety of formats.
2. Able to create, manipulate and save time series files.
3. Able to do vector arithmetic on time series, including
dozens of functions.

Fortran 90 and later versions have array operations, as does NumPy. You
could convert parts of the FORTRAN code to F90

4. Loop and macro facilities to simplify repetitive stuff.
5. Flexible high-resolution graphic presentation.
6. Built in functions to include:
FFT / fourier analysis, MESA / maximum entropy spectral analysis,
multiple regression, canonical correlation etc etc etc.
I have code for all these mostly in FORTRAN, some QBASIC.

The applications of the package include:
1. Analysis of time series data from many branches of science.
2. Economic / business models including forecasting.
3. Markets, stocks, commodities forecasting.
4. Interdisciplinary causal analysis.
5. Many more

There exist public domain codes for many of the topics you mention, and
I think several are part of NumPy. Many statistical algorithms are in
R, for which the underlying C and Fortran code is available. I suggest
that you identify which of your algorithms are not publicly available
and focus on those, making an R package of them. I am interested in
MESA. Then you can exploit the R graphics and language (called S) and
have your work easily accessible to many users.

 

[Cameron Laird]

The Fortran code, which I assume is Fortran 77 or earlier, should run
fine on "modern operating systems" using free (g77, g95, or gfortran)
or commercial compilers.


Fortran 90 and later versions have array operations, as does NumPy. You
could convert parts of the FORTRAN code to F90


There exist public domain codes for many of the topics you mention, and
I think several are part of NumPy. Many statistical algorithms are in
R, for which the underlying C and Fortran code is available. I suggest
that you identify which of your algorithms are not publicly available
and focus on those, making an R package of them. I am interested in
MESA. Then you can exploit the R graphics and language (called S) and
have your work easily accessible to many users.

The original poster has received much good advice. I'll reinforce
a couple of points:
1. Flexibility, high usability, and appealing
graphics indeed are worth the effort. They
can be achieved withOUT object orientation,
though, and you absolutely should consider
modernization of your existing *BASIC,
Fortran, and so on. Don't let lack of a
compiler block your progress; I'm sure we
can help locate appropriate ones for you.
2. Python is indeed a great vehicle for this
sort of work, as I've argued in the past
<URL: http://phaseit.net/claird/comp.programming/open_source_science.html >.
For your particular circumstances, though,
I applaud Mr. Beliavsky's suggestion that
you look into R <URL:
http://www-106.ibm.com/developerworks/linux/library/l-sc16.html >.
You might get even quicker satisfaction,
with a somewhat lower long-term ceiling,
through Yorick <URL: http://wiki.tcl.tk/yorick >.

I understand that you were thinking in terms of enlistment of fellow
developers. You might well be best off, though, with another round
of research and experimentation on your own.