Re: Storing/processing binary file input help needed

Discussion in 'C++' started by Gianni Mariani, Jan 6, 2004.

  1. Arnold wrote:
    > I need to read a binary file and store it into a buffer in memory (system
    > has large amount of RAM, 2GB+) then pass it to a function. The function
    > accepts input as 32 bit unsigned longs (DWORD). I can pass a max of 512
    > words to it at a time. So I would pass them in chunks of 512 words until the
    > whole file has been processed. I haven't worked with binary files before so
    > I'm confused with how to store the binary file into memory. What sort of
    > array do I use? Does C allow char only? Can I declare a DWORD buffer since
    > that's what the function is taking as input? Or do I need to know the format
    > of the original data that binary file is encoding and store it in that?
    > That's the part that is really confusing me.
    > I believe I'll need to used fread to copy the file to that array. I plan on
    > getting the size of file, then determining how many DWORD are present in it
    > (for example 9000) and use that my number of object parameter in fread. So
    > in this case:
    > fread(buffer, 4,9000,fp); //each DWORD is 4 bytes, 900 DWORDs in my binary
    > file
    > Is that right?
    > Once I get the file into the buffer, I can then do a loop where I pass 512
    > elements of the array to a function until all 9000 elements are processed. I
    > hope that's right. Any other tips on improving speed and efficiency would be
    > appreciated. Thanks.

    The fastest way to access large binary disk files is to use mapped
    files. This is off-topic in this NG. Posix systems use mmap() and
    win32 systems use CreateFile/CreateFileMapping/MapViewOfFile.

    I've written countless layers for these, it's pretty easy to code an
    interface that works the same way for Posix and Windows.

    Once that is done, you simply have a pointer and a length and you can
    cast the pointer to whatever the data really is and viola.

    There are numerous cross platform gotchas but if you create the file on
    the same machine you read the file you're safe - but that's true for any
    binary file anyway. These are - endianness, alignment, pointer translation.

    Memory mapping does have the disadvantage that the size of the mapped
    segment of the file is limited to your address space. On 64 bit systems
    this limit is unlikely to be a problem, however it seems you're
    encroaching on that limit here.
    Gianni Mariani, Jan 6, 2004
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Arnold
    Martijn Lievaart
    Jan 7, 2004
  2. Arnold
    Martijn Lievaart
    Jan 7, 2004
  3. Jacek Dziedzic
  4. toton
    Oct 13, 2006
  5. Jonathan Wood
    Jonathan Wood
    Jun 2, 2008

Share This Page