storing data in file instead of memory

Discussion in 'C Programming' started by pereges, Jun 3, 2008.

  1. pereges

    pereges Guest

    I've to store an array of structures:

    typedef struct
    {
    double origin[3];
    double e_field_at_origin_real, e_field_at_origin_imag;
    double direction[3];
    double pathlength;
    int depth;
    }ray;

    A single instance occupies 80 bytes of memory on my machine. I need
    to store an array of more than 1,000,000 structures. My program is
    failing if the array size exceeds 400,000 because I other big data
    structures to store as well. Do you think it will be a better idea to
    store the list on a file and use the file as an array ?
    pereges, Jun 3, 2008
    #1
    1. Advertising

  2. pereges

    Guest

    On Jun 3, 8:57 am, pereges <> wrote:
    > I've to store an array of structures:
    >
    > typedef struct
    > {
    > double origin[3];
    > double e_field_at_origin_real, e_field_at_origin_imag;
    > double direction[3];
    > double pathlength;
    > int depth;
    >
    > }ray;
    >
    > A single instance occupies 80 bytes of memory on my machine. I need
    > to store an array of more than 1,000,000 structures. My program is
    > failing if the array size exceeds 400,000 because I other big data
    > structures to store as well. Do you think it will be a better idea to
    > store the list on a file and use the file as an array ?

    Depends. You might not need a 1.000.000 elements array in the first
    place.
    The hard disk is a lot slower than the RAM, but a lot bigger. You can
    store all your data there, and have, say 10.000 structures loaded in
    memory.
    But this is not topical for comp.lang.c; FILE streams may very well be
    implemented entirely in memory.
    , Jun 3, 2008
    #2
    1. Advertising

  3. On 3 Jun 2008 at 5:57, pereges wrote:
    > I've to store an array of structures:
    >
    > A single instance occupies 80 bytes of memory on my machine. I need
    > to store an array of more than 1,000,000 structures. My program is
    > failing if the array size exceeds 400,000 because I other big data
    > structures to store as well. Do you think it will be a better idea to
    > store the list on a file and use the file as an array ?


    Can't you buy more memory? I mean, most people probably have 80Mb going
    spare in their wristwatches nowadays...

    Failing that, storing to disk should be an absolute last resort. As long
    as there aren't too many writes, it will almost certainly be orders of
    magnitude quicker to keep the data in RAM using some compression scheme,
    and decompress it on-the-fly as needed.
    Antoninus Twink, Jun 3, 2008
    #3
  4. On 3 Jun, 09:25, Antoninus Twink <> wrote:
    > On  3 Jun 2008 at  5:57, pereges wrote:


    > > I've to store an array of structures:

    >
    > > A single instance  occupies 80 bytes of memory on my machine. I need
    > > to store an array of more than 1,000,000 structures. My program is
    > > failing if the array size exceeds 400,000 because I other big data
    > > structures to store as well. Do you think it will be a better idea to
    > > store the list on a file and use the file as an array ?

    >
    > Can't you buy more memory? I mean, most people probably have 80Mb going
    > spare in their wristwatches nowadays...
    >
    > Failing that, storing to disk should be an absolute last resort. As long
    > as there aren't too many writes, it will almost certainly be orders of
    > magnitude quicker to keep the data in RAM using some compression scheme,
    > and decompress it on-the-fly as needed.


    what a sensible post!

    to the OP: the values in your struct mostly seem to be doubles.

    typedef struct
    {
    double origin[3];
    double e_field_at_origin_real, e_field_at_origin_imag;
    double direction[3];
    double pathlength;
    int depth;
    }ray;

    Do you really use the full range of double for all the values?
    Is depth really a 32-bit value (as int is usually 32-bit on modern
    hardware. I'm not saying change your struct but consider Twink's idea
    of compression and where it might be applied.


    --
    Nick Keighley
    Nick Keighley, Jun 3, 2008
    #4
  5. In article <>,
    Antoninus Twink <> wrote:

    >> A single instance occupies 80 bytes of memory on my machine. I need
    >> to store an array of more than 1,000,000 structures. My program is
    >> failing if the array size exceeds 400,000 because I other big data
    >> structures to store as well. Do you think it will be a better idea to
    >> store the list on a file and use the file as an array ?


    >Can't you buy more memory? I mean, most people probably have 80Mb going
    >spare in their wristwatches nowadays...


    He probably already has more memory. The first thing to check is
    that he hasn't got some limit set on his memory use that he can
    change. That'll be operating-system-specific: in unix with bash
    you would use the "ulimit" command. Also note that there may be
    separate limits for stack and heap memory.

    -- Richard
    --
    In the selection of the two characters immediately succeeding the numeral 9,
    consideration shall be given to their replacement by the graphics 10 and 11 to
    facilitate the adoption of the code in the sterling monetary area. (X3.4-1963)
    Richard Tobin, Jun 3, 2008
    #5
  6. pereges <> wrote:
    > I've to store an array of structures:


    > typedef struct
    > {
    > double origin[3];
    > double e_field_at_origin_real, e_field_at_origin_imag;
    > double direction[3];
    > double pathlength;
    > int depth;
    > }ray;


    > A single instance occupies 80 bytes of memory on my machine. I need
    > to store an array of more than 1,000,000 structures. My program is
    > failing if the array size exceeds 400,000 because I other big data
    > structures to store as well. Do you think it will be a better idea to
    > store the list on a file and use the file as an array ?


    It's not clear if you create an array like this

    ray all_may_rays[ 1000000 ];

    or if you use dynamically allocated memory, e.g. using

    ray * all_may_rays;
    all_may_rays = malloc( 1000000 * sizeof *all_may_rays );
    if ( all_may_rays == NULL ) {
    fprintf( stderr, "Not enough memory for rays\n" );
    exit( EXIT_FAILURE );
    }

    If you use the first method then I would consider using dynamically
    allocated memory. While the amount of memory available for arrays
    as created with the first method can be rather limited, the second
    method should only get you in trouble if you need about the same
    amount of memory as is available on your machine (and it may even
    do some storing on the disk automatically for you on many modern
    systems that have swap space).
    Regards, Jens
    --
    \ Jens Thoms Toerring ___
    \__________________________ http://toerring.de
    Jens Thoms Toerring, Jun 3, 2008
    #6
  7. pereges

    santosh Guest

    wrote:

    > FILE streams may very well be implemented entirely in memory.


    How would you be able to access disk files (or named files) with fopen
    if this were so? I think C streams need some kind persistent storage.
    Maybe I'm not thinking this out though.
    santosh, Jun 3, 2008
    #7
  8. pereges

    Bartc Guest

    "pereges" <> wrote in message
    news:...
    > I've to store an array of structures:
    >
    > typedef struct
    > {
    > double origin[3];
    > double e_field_at_origin_real, e_field_at_origin_imag;
    > double direction[3];
    > double pathlength;
    > int depth;
    > }ray;
    >
    > A single instance occupies 80 bytes of memory on my machine. I need
    > to store an array of more than 1,000,000 structures. My program is
    > failing if the array size exceeds 400,000 because I other big data
    > structures to store as well. Do you think it will be a better idea to
    > store the list on a file and use the file as an array ?


    I first read this as 80GB. 80MB doesn't sound that much and you're failing
    at 32MB anyway; what do all the other tables add up to?

    How you thought at all of using float instead of double? That'll pretty much
    halve the requirement to 40MB (but will still fail it seems).

    Where do the fields origin and direction come from? If they are simply
    copies of points in other data structures, you might be able to store
    pointers or indices to those points instead (so 24 bytes becomes 4 bytes).

    Or possibly the same origin or direction is shared amongst many different
    rays; then it may be possible to store one copy only of each xyz value and
    again use indices or pointers to it. This will require some analysis.

    When some years ago I had to store large numbers of xyz points in memory
    (together with transformation data and other stuff), I created a packed (not
    compressed) record format. This worked well when a large proportion of
    values were 0.0 or 1.0, or had many trailing zero bits. But access is then
    much more complex and slower.

    It might also be possible (depending on exactly what your appl does), to
    partition the task, say into regions. And it could be that you can do a
    region at a time and the number of rays for each region is much smaller than
    for the entire task.

    As for storing the file on disk: your virtual memory system should easily be
    able to cope with an extra 80MB or so. If not then you should see what else
    is using the memory. Or maybe just install more ram as someone suggested...

    --
    Bartc
    Bartc, Jun 4, 2008
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    427
    Jon Paal
    Jan 17, 2006
  2. toton
    Replies:
    11
    Views:
    696
    toton
    Oct 13, 2006
  3. Jonathan Wood
    Replies:
    1
    Views:
    498
    Jonathan Wood
    Jun 2, 2008
  4. mk
    Replies:
    3
    Views:
    539
  5. mk
    Replies:
    10
    Views:
    367
    castironpi
    Jul 18, 2008
Loading...

Share This Page