Creating large arrays..

Discussion in 'C++' started by hamishd, Jan 28, 2008.

  1. hamishd

    hamishd Guest

    What is the best way to store large arrays of numbers (eg, 4-byte
    integers)?

    Say I want to store an array of 1billion 4-byte integers.. If my
    computer has > 4gB memory, then is this possible?

    int Large_Array[1000000000];

    This will cause a "stack overflow" when i try and execute. How can I
    do this?
     
    hamishd, Jan 28, 2008
    #1
    1. Advertising

  2. * hamishd:
    > What is the best way to store large arrays of numbers (eg, 4-byte
    > integers)?
    >
    > Say I want to store an array of 1billion 4-byte integers.. If my
    > computer has > 4gB memory, then is this possible?


    Depends on the computer and C++ implementation.


    > int Large_Array[1000000000];
    >
    > This will cause a "stack overflow" when i try and execute. How can I
    > do this?


    You might try to allocate it dynamically, by using

    std::vector<int> Large_Array( 1000000000 );

    However, if you're skirting the limits of your computer's memory
    capacity, then it's not a good idea even if it seems to work.

    You'd be better off using some disk-based structure, processing only
    parts of it at a time.


    Cheers, & hth.,

    - Alf

    --
    A: Because it messes up the order in which people normally read text.
    Q: Why is it such a bad thing?
    A: Top-posting.
    Q: What is the most annoying thing on usenet and in e-mail?
     
    Alf P. Steinbach, Jan 28, 2008
    #2
    1. Advertising

  3. hamishd

    hamishd Guest

    On Jan 28, 11:58 am, "Alf P. Steinbach" <> wrote:
    > * hamishd:
    >
    > > What is the best way to store large arrays of numbers (eg, 4-byte
    > > integers)?

    >
    > > Say I want to store an array of 1billion 4-byte integers.. If my
    > > computer has > 4gB memory, then is this possible?

    >
    > Depends on the computer and C++ implementation.
    >
    > > int Large_Array[1000000000];

    >
    > > This will cause a "stack overflow" when i try and execute. How can I
    > > do this?

    >
    > You might try to allocate it dynamically, by using
    >
    >    std::vector<int> Large_Array( 1000000000 );
    >
    > However, if you're skirting the limits of your computer's memory
    > capacity, then it's not a good idea even if it seems to work.


    thanks.

    > You'd be better off using some disk-based structure, processing only
    > parts of it at a time.


    What are the quicker ways of doing this?
     
    hamishd, Jan 28, 2008
    #3
  4. * hamishd:
    > On Jan 28, 11:58 am, "Alf P. Steinbach" <> wrote:
    >> * hamishd:
    >>
    >>> What is the best way to store large arrays of numbers (eg, 4-byte
    >>> integers)?
    >>> Say I want to store an array of 1billion 4-byte integers.. If my
    >>> computer has > 4gB memory, then is this possible?

    >> Depends on the computer and C++ implementation.
    >>
    >>> int Large_Array[1000000000];
    >>> This will cause a "stack overflow" when i try and execute. How can I
    >>> do this?

    >> You might try to allocate it dynamically, by using
    >>
    >> std::vector<int> Large_Array( 1000000000 );
    >>
    >> However, if you're skirting the limits of your computer's memory
    >> capacity, then it's not a good idea even if it seems to work.

    >
    > thanks.
    >
    >> You'd be better off using some disk-based structure, processing only
    >> parts of it at a time.

    >
    > What are the quicker ways of doing this?


    Depends on what you're doing.

    E.g. it might be that what you're doing best fits some kind of tree
    structure (note: a B-tree isn't a binary tree :)).

    Or it might be that what you're really doing is just some kind of
    sequential processing, in which case read in suitably large chunks of a
    file at a time, process them, and write them back or to some other file.

    If it weren't for the size problem a memory mapped file would be ideal,
    and IIRC the Boost library provides some fairly portable support for that.

    However, the size problem means resorting to old-fashioned reads and
    writes, and the clue there for efficiency is to do large enough chunks,
    and perhaps also preferentially going down to at least the C file i/o
    level instead of C++ iostreams (but it might be good idea to measure!).


    Cheers, & hth.,

    - Alf

    --
    A: Because it messes up the order in which people normally read text.
    Q: Why is it such a bad thing?
    A: Top-posting.
    Q: What is the most annoying thing on usenet and in e-mail?
     
    Alf P. Steinbach, Jan 28, 2008
    #4
  5. hamishd

    tOmMy Guest

    why not use pointer instead of array?


    On 1ÔÂ28ÈÕ, ÏÂÎç7ʱ38·Ö, hamishd <> wrote:
    > What is the best way to store large arrays of numbers (eg, 4-byte
    > integers)?
    >
    > Say I want to store an array of 1billion 4-byte integers.. If my
    > computer has > 4gB memory, then is this possible?
    >
    > int Large_Array[1000000000];
    >
    > This will cause a "stack overflow" when i try and execute. How can I
    > do this?
     
    tOmMy, Jan 28, 2008
    #5
  6. hamishd

    hamishd Guest

    On Jan 28, 3:13 pm, "Alf P. Steinbach" <> wrote:
    > * hamishd:
    >
    >
    >
    >
    >
    > > On Jan 28, 11:58 am, "Alf P. Steinbach" <> wrote:
    > >> * hamishd:

    >
    > >>> What is the best way to store large arrays of numbers (eg, 4-byte
    > >>> integers)?
    > >>> Say I want to store an array of 1billion 4-byte integers.. If my
    > >>> computer has > 4gB memory, then is this possible?
    > >> Depends on the computer and C++ implementation.

    >
    > >>> int Large_Array[1000000000];
    > >>> This will cause a "stack overflow" when i try and execute. How can I
    > >>> do this?
    > >> You might try to allocate it dynamically, by using

    >
    > >>    std::vector<int> Large_Array( 1000000000 );

    >
    > >> However, if you're skirting the limits of your computer's memory
    > >> capacity, then it's not a good idea even if it seems to work.

    >
    > > thanks.

    >
    > >> You'd be better off using some disk-based structure, processing only
    > >> parts of it at a time.

    >
    > > What are the quicker ways of doing this?

    >
    > Depends on what you're doing.


    I'm storing the frequency that a value occurs (value ranging from 1..
    a billion).

    However.. I suspect the array maybe quite sparce. That is, not all
    values will occur at all, so there may be a large number of zeroes.
     
    hamishd, Jan 28, 2008
    #6
  7. hamishd

    Daniel Kraft Guest

    hamishd wrote:
    >>>>> What is the best way to store large arrays of numbers (eg, 4-byte
    >>>>> integers)?
    >>>>> Say I want to store an array of 1billion 4-byte integers.. If my
    >>>>> computer has > 4gB memory, then is this possible?
    >>>> Depends on the computer and C++ implementation.
    >>>>> int Large_Array[1000000000];
    >>>>> This will cause a "stack overflow" when i try and execute. How can I
    >>>>> do this?
    >>>> You might try to allocate it dynamically, by using
    >>>> std::vector<int> Large_Array( 1000000000 );
    >>>> However, if you're skirting the limits of your computer's memory
    >>>> capacity, then it's not a good idea even if it seems to work.
    >>> thanks.
    >>>> You'd be better off using some disk-based structure, processing only
    >>>> parts of it at a time.
    >>> What are the quicker ways of doing this?

    >> Depends on what you're doing.

    >
    > I'm storing the frequency that a value occurs (value ranging from 1..
    > a billion).
    >
    > However.. I suspect the array maybe quite sparce. That is, not all
    > values will occur at all, so there may be a large number of zeroes.


    You could use either a hash-table or even std::map<int, int> for this
    job, I think.

    Daniel

    --
    Got two Dear-Daniel-Instant Messages
    by MSN, associate ICQ with stress--so
    please use good, old E-MAIL!
     
    Daniel Kraft, Jan 28, 2008
    #7
  8. hamishd

    Grizlyk Guest

    hamishd <> wrote:
    >
    > What is the best way to store large arrays of numbers
    > ...
    > If my computer has > 4gB memory
    > ...
    > int Large_Array[1000000000];
    > This will cause a "stack overflow" when i try and execute.


    Do you want to write portable application? C++ knows nothing about
    installed memory of your machine. There are some layers between C++
    and hardware, you need to refer to the layers to make portable
    implementation of the array.

    Do you want to write concrete hardware application? You need to refer
    to the concrete hardware. For some executable environment more than 32
    bit address RAM can be allowed for you and you will not get "stack
    overflow" there.

    For example, you can process the array with own wrapper (user defined
    class) even for very low-memory executable environment or with limited
    or poor virtual memory management, implementing the large array
    storage as binary file with random access to manage memory explicitly.
    The own wrapper also can use own types to represent indexes of the
    array (to remove limits of xxx_MAX).

    I think, long arrays are known for many years, so must be well-known
    wrappers for the work.

    Maksim A. Polyanin
    http://grizlyk1.narod.ru/cpp_new
     
    Grizlyk, Jan 29, 2008
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    0
    Views:
    1,976
  2. Maxim

    Large Arrays of Strings

    Maxim, Jun 29, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    507
    Natty Gur
    Jun 30, 2003
  3. Philipp
    Replies:
    21
    Views:
    1,157
    Philipp
    Jan 20, 2009
  4. Ketchup
    Replies:
    1
    Views:
    262
    Jan Tielens
    May 25, 2004
  5. Replies:
    5
    Views:
    922
    Xho Jingleheimerschmidt
    Apr 2, 2009
Loading...

Share This Page