Historical Data

[philippon]

Hello

I am quite new to Java.

The program that I would like to build is studying time series, which
means that I deal with dates, and a fairly large amount of data that
are stored in ".dat" files.
I have started programming. Each time I create an object, I actually
load the entire historical data, the historical data being an attibute
of the object. So in my constructor, I load the historical data for
the object as soon as it is created, so that it is readily available.
The problem is that I study a lot of time series and it seems to be
too much data to load. Also, I think it is a bit stupid to store
everything as I never need the whole data at one time, only part of
it.

What is the solution to that ? Build a program that opens the file to
pick the right amount of historical data in the file when needed for a
calculation ? Shall I use a database to store the data so that the
access is quicker than with a ".dat" file ? Shall I write the
intermediray results of the calculation I run on the time series in a
file (or database) and access it afterward ?

If you have some code somewhere that deal with time series analysis,
and/or if you have a few minutes to give me your opinion, I would be
very grateful.

Many thanks

 

[Paul Lutus]

Hello

I am quite new to Java.

The program that I would like to build is studying time series, which
means that I deal with dates, and a fairly large amount of data that
are stored in ".dat" files.
I have started programming. Each time I create an object, I actually
load the entire historical data, the historical data being an attibute
of the object. So in my constructor, I load the historical data for
the object as soon as it is created, so that it is readily available.
The problem is that I study a lot of time series and it seems to be
too much data to load. Also, I think it is a bit stupid to store
everything as I never need the whole data at one time, only part of
it.

What is the solution to that ?

Write a program that only loads the data you need. Use a database to filter
out the extraneous fields and speed up access as well. More or less as you
suggest in your own post.

/ ...

If you have some code somewhere that deal with time series analysis,

We need to see your code, and you have not said what kind of analysis you
are doing.

 

[Michael Borgwardt]

It definitely sounds as if storing the data in a database instead of
a file would be exactly the right thing to do.