Need help reading file data

T

Tom

I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
.. . .
.. . .
.. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");

for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------
 
T

Thad Smith

Tom said:
I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");

for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------


If the data is in strict character columns you may be able to seek to
the exact positions of the file where each datum is. I believe it is
not guaranteed for text files, but usually works.

Another approach is to transpose the original data file before processing.
 
C

Charlie Gordon

Tom said:
I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

It should not: the amount of data is roughly 5 million values, which can be
loaded in memory as float or double for a total of 20 or 40 megabytes. If
you are running XP, you must have a lot more RAM than that. It could also
be a limitation of the C compiler you are using. Try cygwin and gcc.
All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");

for(i=0;i<=499;i++)

Better use the actual number of elements:

for (i = 0; i < 500; i++)
{
fscanf(fp,"%f",&(a));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}


try removing this second loop and just scan for the '\n':

{
int c;
while ((c = getc(fp)) != EOF && c != '\n')
continue;
}

This should make the program faster.
 
B

Barry Schwarz

I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

What error? Compile time or run time?
All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");

You should check that fopen succeeded.
for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a));


You should check that fscanf succeeded. You have now read the first
column of a record.
for(j=1;j<=10002;j++)

This loop will try to read the next 10002 columns of the record.
Unfortunately, there are only 10001 columns left to be read. At the
end of this loop you will have read the first column of the next
record.

You might want to try using fgets and strtod to insure you never lose
synchronization.
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);

It seems like you should be using a and b here.
}
---------------------------------------------------------------------------------------------


Remove del for email
 
A

alex

I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");

for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------


Since only numeric data presents, my suggestion is first convert this
text file into binary, and then seek to right
position as needed.

for example,
position(i, j) = i * 10002 * sizeof(float) + j * sizeof(float)
(i, j) is (row, col) index, starting from 0
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top