handling large data

Discussion in 'C Programming' started by a, Nov 24, 2007.

  1. a

    a Guest

    Would anybody please tell me how to handle array data of int 2000 * 2000?
    Even make file passes by using a supercomputer, it seems calloc fails and
    segmentation fault results during a trial run.
     
    a, Nov 24, 2007
    #1
    1. Advertising

  2. a

    a Guest

    "a" <> wrote in message news:fi80g5$d8e$...
    > Would anybody please tell me how to handle array data of int 2000 * 2000?
    > Even make file passes by using a supercomputer, it seems calloc fails and
    > segmentation fault results during a trial run.
    >


    Make error:

    gcc -O3 -Wall -I. vote.c datamanager.c -o vote -L. -lhungarian
    gcc: Internal error: Segmentation fault (program cc1)
    Please submit a full bug report.
    See <URL:http://bugzilla.redhat.com/bugzilla/> for instructions.
    make: *** [vote] Error 1


    Run-time (made in a supercomputer) error:
    Segmentation fault


    The program snippet:
    int r[2000*2000]={
    123,188,158,197,184,180,146,129,117,192,120,148,175,198,153,190,135,175,158,155,157,192,108,187,148,187,199,139,158,120,174,168,184,162,132,191,166,113,167,148,131,159,107,120,164,115,115,110,117,105,131,1
    ....};
     
    a, Nov 24, 2007
    #2
    1. Advertising

  3. a

    Ian Collins Guest

    a wrote:
    > "a" <> wrote in message news:fi80g5$d8e$...
    >> Would anybody please tell me how to handle array data of int 2000 * 2000?
    >> Even make file passes by using a supercomputer, it seems calloc fails and
    >> segmentation fault results during a trial run.
    >>

    >
    > Make error:
    >
    > gcc -O3 -Wall -I. vote.c datamanager.c -o vote -L. -lhungarian
    > gcc: Internal error: Segmentation fault (program cc1)
    > Please submit a full bug report.
    > See <URL:http://bugzilla.redhat.com/bugzilla/> for instructions.
    > make: *** [vote] Error 1
    >

    Maybe it ran out of space?
    >
    > The program snippet:
    > int r[2000*2000]={
    > 123,188,158,197,184,180,146,129,117,192,120,148,175,198,153,190,135,175,158,155,157,192,108,187,148,187,199,139,158,120,174,168,184,162,132,191,166,113,167,148,131,159,107,120,164,115,115,110,117,105,131,1
    > ....};
    >

    Put the data in a files and read it in at runtime instead of using
    static initialisation.

    --
    Ian Collins.
     
    Ian Collins, Nov 24, 2007
    #3
  4. a

    CBFalconer Guest

    a wrote:
    >
    > Would anybody please tell me how to handle array data of int 2000
    > * 2000? Even make file passes by using a supercomputer, it seems
    > calloc fails and segmentation fault results during a trial run.


    Assuming you have a fairly normal system with sizeof int == 4, that
    array requires 4,000,000 * 4, or 16 gigabytes of memory. Note that
    the guaranteed available size is 65k (for C99) and 32k (for C90).
    Many systems will give you 2 gigabytes, but only some 64 bit
    systems will allow 16 gigs.

    Look for another mechanism.

    --
    Chuck F (cbfalconer at maineline dot net)
    <http://cbfalconer.home.att.net>
    Try the download section.



    --
    Posted via a free Usenet account from http://www.teranews.com
     
    CBFalconer, Nov 24, 2007
    #4
  5. a

    Ian Collins Guest

    CBFalconer wrote:
    > a wrote:
    >> Would anybody please tell me how to handle array data of int 2000
    >> * 2000? Even make file passes by using a supercomputer, it seems
    >> calloc fails and segmentation fault results during a trial run.

    >
    > Assuming you have a fairly normal system with sizeof int == 4, that
    > array requires 4,000,000 * 4, or 16 gigabytes of memory.


    s/giga/mega/ which leave a manageable answer, but maybe not for static
    initialisation.

    --
    Ian Collins.
     
    Ian Collins, Nov 24, 2007
    #5
  6. CBFalconer said:

    > a wrote:
    >>
    >> Would anybody please tell me how to handle array data of int 2000
    >> * 2000? Even make file passes by using a supercomputer, it seems
    >> calloc fails and segmentation fault results during a trial run.

    >
    > Assuming you have a fairly normal system with sizeof int == 4, that
    > array requires 4,000,000 * 4, or 16 gigabytes of memory.


    ITYM "megabytes", which isn't all *that* big really (although it's pretty
    hefty for a local array).

    One malloc for the row pointers, and one malloc per row, should do it
    nicely on modern desktop systems with not too much else running.

    --
    Richard Heathfield <http://www.cpax.org.uk>
    Email: -http://www. +rjh@
    Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
    "Usenet is a strange place" - dmr 29 July 1999
     
    Richard Heathfield, Nov 24, 2007
    #6
  7. a

    CBFalconer Guest

    Richard Heathfield wrote:
    > CBFalconer said:
    >> a wrote:
    >>>
    >>> Would anybody please tell me how to handle array data of int 2000
    >>> * 2000? Even make file passes by using a supercomputer, it seems
    >>> calloc fails and segmentation fault results during a trial run.

    >>
    >> Assuming you have a fairly normal system with sizeof int == 4, that
    >> array requires 4,000,000 * 4, or 16 gigabytes of memory.

    >
    > ITYM "megabytes", which isn't all *that* big really (although it's
    > pretty hefty for a local array).
    >
    > One malloc for the row pointers, and one malloc per row, should do
    > it nicely on modern desktop systems with not too much else running.


    I keep doing that - missing a factor of 1000, or 10 cubed. Maybe I
    need three less fingers. However, I did assume that he wanted a
    single array, which is way over the minimum guarantee.

    --
    Chuck F (cbfalconer at maineline dot net)
    <http://cbfalconer.home.att.net>
    Try the download section.



    --
    Posted via a free Usenet account from http://www.teranews.com
     
    CBFalconer, Nov 24, 2007
    #7
  8. CBFalconer wrote:
    > a wrote:
    >> Would anybody please tell me how to handle array data of int 2000
    >> * 2000? Even make file passes by using a supercomputer, it seems
    >> calloc fails and segmentation fault results during a trial run.

    >
    > Assuming you have a fairly normal system with sizeof int == 4, that
    > array requires 4,000,000 * 4, or 16 gigabytes of memory. Note that
    > the guaranteed available size is 65k (for C99) and 32k (for C90).


    Er, you mean megabytes. As such, the argument below is irrelevant.

    > Many systems will give you 2 gigabytes, but only some 64 bit
    > systems will allow 16 gigs.
    >
    > Look for another mechanism.
    >
     
    Philip Potter, Nov 24, 2007
    #8
  9. Ian Collins wrote:
    > a wrote:
    >> "a" <> wrote in message news:fi80g5$d8e$...
    >>> Would anybody please tell me how to handle array data of int 2000 * 2000?
    >>> Even make file passes by using a supercomputer, it seems calloc fails and
    >>> segmentation fault results during a trial run.
    >>>

    >> Make error:
    >>
    >> gcc -O3 -Wall -I. vote.c datamanager.c -o vote -L. -lhungarian
    >> gcc: Internal error: Segmentation fault (program cc1)
    >> Please submit a full bug report.
    >> See <URL:http://bugzilla.redhat.com/bugzilla/> for instructions.
    >> make: *** [vote] Error 1
    >>

    > Maybe it ran out of space?


    Even if it does, it should fail cleanly rather than segfault.
     
    Philip Potter, Nov 24, 2007
    #9
  10. a

    santosh Guest

    In article <fi80g5$d8e$>, a <> wrote on
    Saturday 24 Nov 2007 7:28 am:

    > Would anybody please tell me how to handle array data of int 2000 *
    > 2000? Even make file passes by using a supercomputer, it seems calloc
    > fails and segmentation fault results during a trial run.


    If you do not have sufficient primary storage the only option is to
    operate on smaller chunks of the data. From the standpoint of Standard
    C, you have no recourse upon a *alloc() failure, but to reduce the
    requirements and try again. However a specific system may provide a
    considerable amount of additional strategies and methods to cope with
    large data.

    With modern operating systems the system will do a lot of
    behind-the-scenes work for you, so that often, you can simply do the
    allocation and proceed. If however this does not work, you might try
    memory-mapped files.
     
    santosh, Nov 24, 2007
    #10
  11. "Richard Heathfield" <> wrote in message
    >

    2000 * 2000 * int
    > ITYM "megabytes", which isn't all *that* big really (although it's pretty
    > hefty for a local array).
    >
    > One malloc for the row pointers, and one malloc per row, should do it
    > nicely on modern desktop systems with not too much else running.
    >

    That's just a fairly normal sized image. How did we use to manage with
    little 64k systems?

    --
    Free games and programming goodies.
    http://www.personal.leeds.ac.uk/~bgy1mm
     
    Malcolm McLean, Nov 24, 2007
    #11
  12. In article <fi80g5$d8e$>, a <> wrote:
    >Would anybody please tell me how to handle array data of int 2000 * 2000?
    >Even make file passes by using a supercomputer, it seems calloc fails and
    >segmentation fault results during a trial run.


    If calloc() fails trying to allocate 2000*2000 ints, you almost
    certainly have a system limit set too low. Ask your system
    administrator how to change it (presumably you have a system
    administrator if you have a supercomputer). You should of course be
    checking for calloc() failing so that you don't get a segmentation
    fault.

    Or else there's some other bug in your program and it's nothing to do
    with the amount of memory. To check for that possibility use a
    program such as valgrind.

    -- Richard

    --
    "Consideration shall be given to the need for as many as 32 characters
    in some alphabets" - X3.4, 1963.
     
    Richard Tobin, Nov 24, 2007
    #12
  13. a

    a Guest

    it seems that big static *2D* array creates the problem, e.g.


    a[1024][1024] but not a[1024*1024] makes the problem.

    i try gcc -static to see if this helps but of no hope.

    there will be too many modifications to take care when I switch to malloc or
    related solutions. Is there any workaround?
     
    a, Nov 24, 2007
    #13
  14. a

    a Guest

    Which file is to be changed? I also remember there should be a config file
    to set many parameters...


    "Richard Tobin" <> wrote in message
    news:fi8vgd$2thk$...
    > In article <fi80g5$d8e$>, a <> wrote:
    >>Would anybody please tell me how to handle array data of int 2000 * 2000?
    >>Even make file passes by using a supercomputer, it seems calloc fails and
    >>segmentation fault results during a trial run.

    >
    > If calloc() fails trying to allocate 2000*2000 ints, you almost
    > certainly have a system limit set too low. Ask your system
    > administrator how to change it (presumably you have a system
    > administrator if you have a supercomputer). You should of course be
    > checking for calloc() failing so that you don't get a segmentation
    > fault.
    >
    > Or else there's some other bug in your program and it's nothing to do
    > with the amount of memory. To check for that possibility use a
    > program such as valgrind.
    >
    > -- Richard
    >
    > --
    > "Consideration shall be given to the need for as many as 32 characters
    > in some alphabets" - X3.4, 1963.
     
    a, Nov 24, 2007
    #14
  15. a

    Vitor Py Guest

    On 23 nov, 23:58, "a" <> wrote:
    > Would anybody please tell me how to handle array data of int 2000 * 2000?
    > Even make file passes by using a supercomputer, it seems calloc fails and
    > segmentation fault results during a trial run.


    If you're handling matrices, you may want study them use more
    apropriate storage. Let's say, if you have a three diagonal matrix,
    use 3 arrays for storage (it's going to make your code uglier, and
    require more work - but it's gonna be really faster). Diagonal
    matrices are fairly common from problems arising from finite
    differences or finite elements. Allocating them as full matrices
    usually is not worth it's price.

    Vitor.
    PS: I'm not a native speaker of english, so, pardon for my errors.
     
    Vitor Py, Nov 24, 2007
    #15
  16. On Sat, 24 Nov 2007 22:32:37 +0800, "a" <> wrote:

    >it seems that big static *2D* array creates the problem, e.g.
    >
    >
    >a[1024][1024] but not a[1024*1024] makes the problem.
    >
    >i try gcc -static to see if this helps but of no hope.
    >
    >there will be too many modifications to take care when I switch to malloc or
    >related solutions. Is there any workaround?


    What is the actual type that size_t is a typedef for? What is the
    maximum value for that type?

    Since both a[1024][1024] and a[1024*1024] require the exact same
    space, if one succeeds and the other fails then something else is
    going on. A likely candidate is that SIZE_MAX is less than one
    million and the multiplication in the latter declaration is wrapping
    to some tolerable value. You should print the size of a in this case.

    In any event, changing from a defined array to an allocated one
    requires replacing only the one line declaration
    int a[1024][1024];
    with the small loop
    int *a[1024];
    for (i = 0; i < 1024; i++)
    a = malloc(1024 * sizeof *a);
    From then on, all references in your code to a[j] should remain
    valid without modification.



    Remove del for email
     
    Barry Schwarz, Nov 25, 2007
    #16
  17. a

    Gordon Guest

    On Nov 24, 9:58 am, "a" <> wrote:
    > Would anybody please tell me how to handle array data of int 2000 * 2000?
    > Even make file passes by using a supercomputer, it seems calloc fails and
    > segmentation fault results during a trial run.
     
    Gordon, Dec 8, 2007
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. nish
    Replies:
    1
    Views:
    344
  2. Wayne Marsh

    Handling large amounts of data

    Wayne Marsh, Nov 20, 2004, in forum: C Programming
    Replies:
    3
    Views:
    453
    Malcolm
    Nov 20, 2004
  3. Ketchup
    Replies:
    1
    Views:
    281
    Jan Tielens
    May 25, 2004
  4. Martin Pirker

    handling large data sets

    Martin Pirker, Dec 8, 2003, in forum: Ruby
    Replies:
    10
    Views:
    256
    Ian Hobson
    Dec 9, 2003
  5. Replies:
    5
    Views:
    992
    Xho Jingleheimerschmidt
    Apr 2, 2009
Loading...

Share This Page