Need help reading file data

Discussion in 'C Programming' started by Tom, Sep 30, 2007.

  1. Tom

    Tom Guest

    I have a datafile which has a form of

    10.10 10.20 10.30 ...
    20.10 20.20 20.30 ...
    30.10 30.20 30.30 ...
    .. . .
    .. . .
    .. . .

    it has 500 rows and 10002 colums.

    I tried to read the whole data in a single variable.
    My pc shows error. (Windows XP, notebook)

    All I need is to read each column data and do some integration.
    and so on for every colum.

    my naive code is following but, it is too slow.
    (it is also limited to read first column, but have to expand to read
    every column)

    ---------------------------------------------------------------------------------------------
    FILE *fp;
    float b;
    int i,j;
    float a[500];

    fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");

    for(i=0;i<=499;i++)
    {
    fscanf(fp,"%f",&(a));

    for(j=1;j<=10002;j++)
    {
    fscanf(fp,"%f",&b);
    }

    Dosomethingwith(float a, 500);
    }
    ---------------------------------------------------------------------------------------------
    Tom, Sep 30, 2007
    #1
    1. Advertising

  2. Tom

    Thad Smith Guest

    Tom wrote:
    > I have a datafile which has a form of
    >
    > 10.10 10.20 10.30 ...
    > 20.10 20.20 20.30 ...
    > 30.10 30.20 30.30 ...
    > . . .
    > . . .
    > . . .
    >
    > it has 500 rows and 10002 colums.
    >
    > I tried to read the whole data in a single variable.
    > My pc shows error. (Windows XP, notebook)
    >
    > All I need is to read each column data and do some integration.
    > and so on for every colum.
    >
    > my naive code is following but, it is too slow.
    > (it is also limited to read first column, but have to expand to read
    > every column)
    >
    > ---------------------------------------------------------------------------------------------
    > FILE *fp;
    > float b;
    > int i,j;
    > float a[500];
    >
    > fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");
    >
    > for(i=0;i<=499;i++)
    > {
    > fscanf(fp,"%f",&(a));
    >
    > for(j=1;j<=10002;j++)
    > {
    > fscanf(fp,"%f",&b);
    > }
    >
    > Dosomethingwith(float a, 500);
    > }
    > ---------------------------------------------------------------------------------------------


    If the data is in strict character columns you may be able to seek to
    the exact positions of the file where each datum is. I believe it is
    not guaranteed for text files, but usually works.

    Another approach is to transpose the original data file before processing.

    --
    Thad
    Thad Smith, Oct 1, 2007
    #2
    1. Advertising

  3. "Tom" <> a écrit dans le message de news:
    ...
    >I have a datafile which has a form of
    >
    > 10.10 10.20 10.30 ...
    > 20.10 20.20 20.30 ...
    > 30.10 30.20 30.30 ...
    > . . .
    > . . .
    > . . .
    >
    > it has 500 rows and 10002 colums.
    >
    > I tried to read the whole data in a single variable.
    > My pc shows error. (Windows XP, notebook)


    It should not: the amount of data is roughly 5 million values, which can be
    loaded in memory as float or double for a total of 20 or 40 megabytes. If
    you are running XP, you must have a lot more RAM than that. It could also
    be a limitation of the C compiler you are using. Try cygwin and gcc.

    > All I need is to read each column data and do some integration.
    > and so on for every colum.
    >
    > my naive code is following but, it is too slow.
    > (it is also limited to read first column, but have to expand to read
    > every column)
    >
    > ---------------------------------------------------------------------------------------------
    > FILE *fp;
    > float b;
    > int i,j;
    > float a[500];
    >
    > fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");
    >
    > for(i=0;i<=499;i++)


    Better use the actual number of elements:

    for (i = 0; i < 500; i++)

    > {
    > fscanf(fp,"%f",&(a));
    >
    > for(j=1;j<=10002;j++)
    > {
    > fscanf(fp,"%f",&b);
    > }


    try removing this second loop and just scan for the '\n':

    {
    int c;
    while ((c = getc(fp)) != EOF && c != '\n')
    continue;
    }

    This should make the program faster.

    >
    > Dosomethingwith(float a, 500);
    > }
    > ---------------------------------------------------------------------------------------------


    --
    Chqrlie.
    Charlie Gordon, Oct 1, 2007
    #3
  4. On Sun, 30 Sep 2007 17:49:44 -0000, Tom <> wrote:

    >I have a datafile which has a form of
    >
    >10.10 10.20 10.30 ...
    >20.10 20.20 20.30 ...
    >30.10 30.20 30.30 ...
    >. . .
    >. . .
    >. . .
    >
    >it has 500 rows and 10002 colums.
    >
    >I tried to read the whole data in a single variable.
    >My pc shows error. (Windows XP, notebook)


    What error? Compile time or run time?

    >
    >All I need is to read each column data and do some integration.
    >and so on for every colum.
    >
    >my naive code is following but, it is too slow.
    >(it is also limited to read first column, but have to expand to read
    >every column)
    >
    >---------------------------------------------------------------------------------------------
    > FILE *fp;
    > float b;
    > int i,j;
    > float a[500];
    >
    > fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");


    You should check that fopen succeeded.

    >
    > for(i=0;i<=499;i++)
    > {
    > fscanf(fp,"%f",&(a));


    You should check that fscanf succeeded. You have now read the first
    column of a record.

    >
    > for(j=1;j<=10002;j++)


    This loop will try to read the next 10002 columns of the record.
    Unfortunately, there are only 10001 columns left to be read. At the
    end of this loop you will have read the first column of the next
    record.

    You might want to try using fgets and strtod to insure you never lose
    synchronization.

    > {
    > fscanf(fp,"%f",&b);
    > }
    >
    > Dosomethingwith(float a, 500);


    It seems like you should be using a and b here.

    > }
    >---------------------------------------------------------------------------------------------



    Remove del for email
    Barry Schwarz, Oct 10, 2007
    #4
  5. Tom

    alex Guest

    On Oct 1, 1:49 am, Tom <> wrote:
    > I have a datafile which has a form of
    >
    > 10.10 10.20 10.30 ...
    > 20.10 20.20 20.30 ...
    > 30.10 30.20 30.30 ...
    > . . .
    > . . .
    > . . .
    >
    > it has 500 rows and 10002 colums.
    >
    > I tried to read the whole data in a single variable.
    > My pc shows error. (Windows XP, notebook)
    >
    > All I need is to read each column data and do some integration.
    > and so on for every colum.
    >
    > my naive code is following but, it is too slow.
    > (it is also limited to read first column, but have to expand to read
    > every column)
    >
    > ---------------------------------------------------------------------------------------------
    > FILE *fp;
    > float b;
    > int i,j;
    > float a[500];
    >
    > fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20posi.dat","r");
    >
    > for(i=0;i<=499;i++)
    > {
    > fscanf(fp,"%f",&(a));
    >
    > for(j=1;j<=10002;j++)
    > {
    > fscanf(fp,"%f",&b);
    > }
    >
    > Dosomethingwith(float a, 500);
    > }
    > ---------------------------------------------------------------------------------------------


    Since only numeric data presents, my suggestion is first convert this
    text file into binary, and then seek to right
    position as needed.

    for example,
    position(i, j) = i * 10002 * sizeof(float) + j * sizeof(float)
    (i, j) is (row, col) index, starting from 0
    alex, Oct 10, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. morc
    Replies:
    5
    Views:
    478
    Roedy Green
    Mar 3, 2006
  2. bahoo
    Replies:
    8
    Views:
    354
    Bruno Desthuilliers
    Apr 4, 2007
  3. Amkcoder

    Need help reading a file.

    Amkcoder, Oct 4, 2008, in forum: C Programming
    Replies:
    6
    Views:
    306
    CBFalconer
    Oct 5, 2008
  4. Stephen Moon

    reading output file data as input data

    Stephen Moon, Feb 28, 2004, in forum: Perl Misc
    Replies:
    5
    Views:
    163
    Tad McClellan
    Feb 29, 2004
  5. Replies:
    5
    Views:
    69
    Chris Angelico
    May 14, 2014
Loading...

Share This Page