Reading large files

Discussion in 'C Programming' started by JS, Sep 19, 2003.

  1. JS

    JS Guest

    Hello all!

    I have come on to a problem for which I am not able to find a solution
    searching the web.

    What I am trying to do is reading a log-file with a size of 1.3 GB.
    When reading it using fread() or the Visual C specific(?) read() they
    return that the number of bytes read equals the number of bytes
    requested, however all of the bytes read turn out to have a value of
    zero (the file contains other values). When using the exact same code
    on a much smaller file (1 MB) it works fine.

    char filename[] = "c:/temp/data2000.dat";
    char buffer[40];
    FILE *file_d;
    struct _stat stat_buf;
    int fd;
    int file_exists;
    long file_size;

    //Returns true:
    file_exists = (_stat(filename,&stat_buf)==0) && (stat_buf.st_mode &
    S_IFREG);
    //Returns 1328000000 (actual filesize):
    file_size = (file_exists) ? stat_buf.st_size : -1;


    //Booth returns values <> NULL and -1:
    //file_d = fopen(filename, "r");
    fd = open(filename, _O_RDONLY);

    //Both returns 40:
    //bytes_read = fread(buffer, 1, sizeof(buffer), file_d);
    bytes_read = read(fd, buffer, 40);

    //fclose(file_d);
    close(fd);

    Greatful for any ideas!
    Best regards
     
    JS, Sep 19, 2003
    #1
    1. Advertising

  2. (JS) wrote:

    >Hello all!
    >
    >I have come on to a problem for which I am not able to find a solution
    >searching the web.
    >
    >What I am trying to do is reading a log-file with a size of 1.3 GB.
    >When reading it using fread() or the Visual C specific(?) read() they
    >return that the number of bytes read equals the number of bytes
    >requested, however all of the bytes read turn out to have a value of
    >zero (the file contains other values). When using the exact same code
    >on a much smaller file (1 MB) it works fine.


    <CODE SNIPPED>

    Did you make sure your file contains pure text data? If it does not,
    you should open it in binary mode.

    Note that _stat(), open(), read(), and close() are not standard C
    functions (Ah, I see from your remark above you already know ...).

    Also note that both of us swapped the second and third argument of
    fread, though this is useful to get the number of characters read in the
    final call to fread, which will result most certainly in a partial
    filled buffer. (It doesn't really matter anyway in this case, because
    there's not a big difference between fread performing 40*1 or 1*40 calls
    to fgetc respectively.)

    The program below works fine for me (tested with a 2.8 GB file):

    #include <stdio.h>
    #include <stdlib.h>

    #define BUFELEM 40
    #define FILENAME "test.dat"

    int main( void )
    {
    size_t num;
    unsigned long count;
    unsigned char buf[ BUFELEM ];
    FILE *fp;

    if ( ( fp = fopen( FILENAME, "rb" ) ) != NULL )
    {
    count = 0;
    do
    {
    num = fread( buf, 1, BUFELEM, fp );
    ++count;
    /* do sth. with buffer contents here [1] */
    }
    while ( num == BUFELEM );

    /* Error checking snipped for brevity */

    fclose( fp );
    printf( "calls to fread: %ld\n", count );
    }
    else
    {
    fprintf( stderr, "Error opening file %s\n", FILENAME );
    return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
    }

    [1] You may write the buffer contents to another file and compare
    it to the original to make sure valid data had ben read.


    Regards

    Irrwahn
    --
    The generation of random numbers is too important
    to be left to chance.
     
    Irrwahn Grausewitz, Sep 19, 2003
    #2
    1. Advertising

  3. JS

    Severian Guest

    On 19 Sep 2003 00:13:37 -0700, (JS) wrote:

    >Hello all!
    >
    >I have come on to a problem for which I am not able to find a solution
    >searching the web.
    >
    >What I am trying to do is reading a log-file with a size of 1.3 GB.
    >When reading it using fread() or the Visual C specific(?) read() they
    >return that the number of bytes read equals the number of bytes
    >requested, however all of the bytes read turn out to have a value of
    >zero (the file contains other values). When using the exact same code
    >on a much smaller file (1 MB) it works fine.


    <code snipped>

    Have you verified that the beginning of the file is not a bunch of
    null characters, by using a binary dump facility or something?

    - Sev


    --
    I am just a thought of mine, an egotisticality. (P. Crews)
     
    Severian, Sep 19, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?SHV0dHk=?=

    Reading large text files

    =?Utf-8?B?SHV0dHk=?=, Sep 24, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    725
    =?Utf-8?B?SHV0dHk=?=
    Sep 27, 2005
  2. Davidski
    Replies:
    0
    Views:
    3,911
    Davidski
    Nov 5, 2004
  3. freesoft_2000

    Reading Large Files

    freesoft_2000, Aug 11, 2005, in forum: Java
    Replies:
    18
    Views:
    1,028
    Andrew Thompson
    Aug 13, 2005
  4. Replies:
    4
    Views:
    408
    Paul Duffy
    Mar 4, 2007
  5. Matthew Crema

    Reading a large number of text files into an array

    Matthew Crema, Apr 27, 2005, in forum: C Programming
    Replies:
    4
    Views:
    717
    Matthew Crema
    Apr 27, 2005
Loading...

Share This Page