I/O ofa large number of files

Discussion in 'C Programming' started by David, Aug 1, 2005.

  1. David

    David Guest

    Hi all:

    I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
    each).

    If I want read large amount of radom data from this series, how could I
    buffer the file to get optimized performance? With WinXP pro/512MB memories
    and no other big programmes running at the same time.

    Cheers

    David
     
    David, Aug 1, 2005
    #1
    1. Advertising

  2. David

    Eric Sosman Guest

    David wrote:
    > Hi all:
    >
    > I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
    > each).
    >
    > If I want read large amount of radom data from this series, how could I
    > buffer the file to get optimized performance?


    If the access pattern is "radom" there isn't much you
    can do. Load as much as you can manage into memory and hope
    your next "radom" access is for something already loaded; if
    it isn't, throw something away and load the piece you're
    trying to get at. The selection of what to pre-load, what
    to throw away, and how much to load for an out-of-memory
    experience depends on the access patterns. Some patterns
    will lend themselves to exploitation, others won't.

    The C pieces you'll need will probably be fopen (with
    "rb" mode, most likely), fseek, fread, and fclose, along
    with the usual memory-management stuff. I mention fclose
    because your system may not permit you to have 1024 file
    streams open simultaneously; you may need to "multiplex"
    the large number of files across a smaller number of FILE*
    streams by closing and re-opening as necessary. The
    FOPEN_MAX macro in <stdio.h> gives an approximation to the
    number of streams you can open simultaneously, but the
    value should be treated only as an approximation.

    > With WinXP pro/512MB memories
    > and no other big programmes running at the same time.


    Irrelevant detail. Well, highly relevant in some ways,
    but not to your question (if there is one) about the C
    programming language. Windows-oriented newsgroups may have
    suggestions that go beyond what C itself can provide.

    --
     
    Eric Sosman, Aug 1, 2005
    #2
    1. Advertising

  3. David

    jacob navia Guest

    David wrote:
    > Hi all:
    >
    > I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
    > each).
    >
    > If I want read large amount of radom data from this series, how could I
    > buffer the file to get optimized performance? With WinXP pro/512MB memories
    > and no other big programmes running at the same time.
    >
    > Cheers
    >
    > David
    >
    >

    Map the whole file into memory (backed up by the OS).
    Read the docs for MapViewOfFile API. This will not use
    all the RAM of course, but the system will do the paging
    for you, what is far more efficient than what you can do
    yourself.

    This is not a standard C function, and in another operating
    systems you may have to use a different approach.

    Using standard C functions you can do the same (as Eric
    Sossman replied), but it will be less efficient and is not
    trivial to develop.

    A simpler approach using just fopen would be to allocate 1MB of virtual
    memory with malloc(), then read the whole file into it. The system will
    do the paging for you in that case. Frequently used pages will be kept
    in memory, less frequently used will be eventually be paged out.

    jacob
     
    jacob navia, Aug 1, 2005
    #3
  4. In article <dclchh$1ku$>,
    David <> wrote:
    >
    > I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
    > each).
    >
    > If I want read large amount of radom data from this series, how could I
    > buffer the file to get optimized performance? With WinXP pro/512MB memories
    > and no other big programmes running at the same time.


    I'll take a wild guess and say that whatever process is generating
    these 3D bitmaps is pretty high-tech and not cheap...perhaps there is
    room in the budget for another gig of RAM? Then you could load the
    whole 3D image into RAM and still have the original 512 for the OS and
    application. Next step would be to upgrade to a processor with large
    amounts of cache.

    Sure, it would be cool to optimize some graphics-intensive C code to
    minimize thrashing but I hope you are being paid well enough that
    1G RAM is cheaper than a couple days of your valuable time.
    --
    7842++
     
    Anonymous 7843, Aug 1, 2005
    #4
  5. David

    David Guest

    Thank you all for thsoe informatvie and inspiring replies!

    Maybe for my purpose, I may opt to buffer a number of files and use a
    counter to record the times of individual files being accessed. Once an
    unbuffered file is required, it will replace the least accessed file.
    I would prefer this easy and platform independent method... coz I know
    little APIs...
    :( ...

    Hope the random statistics would do the jod themselves.

    "David" <> wrote in message
    news:dclchh$1ku$...
    > Hi all:
    >
    > I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of
    > 1MB each).
    >
    > If I want read large amount of radom data from this series, how could I
    > buffer the file to get optimized performance? With WinXP pro/512MB
    > memories and no other big programmes running at the same time.
    >
    > Cheers
    >
    > David
    >
    >
     
    David, Aug 2, 2005
    #5
  6. David

    Jasen Betts Guest

    In article <dclchh$1ku$>, David wrote:
    > Hi all:
    >
    > I am processing a 3D bitmaps(essentially ~1024 2D bitmaps with a size of 1MB
    > each).
    >
    > If I want read large amount of radom data from this series, how could I
    > buffer the file to get optimized performance? With WinXP pro/512MB memories
    > and no other big programmes running at the same time.


    the answer is process specific - it depends what you're doing with the bitmap.

    the easiest solution is probably to put more ram in your PC - if the OS can
    handle it.

    otherwise you may need to break the bitmap up into chunks which can be
    processed independantly.


    Bye.
    Jasen
     
    Jasen Betts, Aug 4, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Lionel

    Large number of files...

    Lionel, Feb 19, 2005, in forum: ASP .Net
    Replies:
    3
    Views:
    382
    Kevin Spencer
    Feb 21, 2005
  2. Replies:
    13
    Views:
    10,917
  3. Donald Firesmith
    Replies:
    1
    Views:
    485
    Peter Flynn
    Jan 8, 2005
  4. Replies:
    2
    Views:
    544
  5. bart plessers

    dbase: how to get fieldNAME ofa table

    bart plessers, Oct 2, 2003, in forum: ASP General
    Replies:
    4
    Views:
    159
    Bob Barrows
    Oct 2, 2003
Loading...

Share This Page