dynamic memory for large arrays

Discussion in 'C Programming' started by Richard Bos, Feb 23, 2004.

  1. Richard Bos

    Richard Bos Guest

    Akyl Tulegenov <> wrote:

    > I've got a question which is rather system specific case.


    Quite. So why post it here? We can't answer your question.

    > Suppose I am dealing with a large array of ~4Gig of entities.
    > The question is : what is faster , to address it through a binary file
    > created on disk or using dynamic memory which is extended through a large
    > swap file (again on disk)? Note that swap file is created at the
    > point of the disk partition.


    Who knows? Measure, measure, measure. This depends not only on the OS,
    the compiler, and the technique used to map the dynamic memory, but
    probably also on the relative speeds of the swap disk and the normal FS,
    and several others. Measure, and then assume your measurements are
    correct for you and nobody else.

    > I would be grateful if in addition to your comments you could point me to
    > the relevant source (URL or else).


    "The" relevant source? I don't think you quite understand this "system
    specific" idea. There must be hundreds of relevant sources. Do a web
    search; _you_ know what's inside your system, we don't.

    Richard
     
    Richard Bos, Feb 23, 2004
    #1
    1. Advertising

  2. Dear All!

    I've got a question which is rather system specific case.
    Suppose I am dealing with a large array of ~4Gig of entities.
    The question is : what is faster , to address it through a binary file
    created on disk or using dynamic memory which is extended through a large
    swap file (again on disk)? Note that swap file is created at the
    point of the disk partition.
    I would be grateful if in addition to your comments you could point me to
    the relevant source (URL or else).

    Thanks,
    Akyl.
     
    Akyl Tulegenov, Feb 23, 2004
    #2
    1. Advertising

  3. Richard Bos

    Guillaume Guest

    > I've got a question which is rather system specific case.

    You're right, it's off-topic and risks getting flamed because of that.

    > Suppose I am dealing with a large array of ~4Gig of entities.
    > The question is : what is faster , to address it through a binary file
    > created on disk or using dynamic memory which is extended through a large
    > swap file (again on disk)? Note that swap file is created at the
    > point of the disk partition.
    > I would be grateful if in addition to your comments you could point me to


    I'd suggest using memory-mapped I/O if you have access to this.
    You'll get the best of both worlds.

    If you have access to libc/glibc, mmap() is your friend.
     
    Guillaume, Feb 23, 2004
    #3
  4. Akyl Tulegenov wrote:
    > Dear All!
    >
    > I've got a question which is rather system specific case.
    > Suppose I am dealing with a large array of ~4Gig of entities.
    > The question is : what is faster , to address it through a binary file
    > created on disk or using dynamic memory which is extended through a large
    > swap file (again on disk)? Note that swap file is created at the
    > point of the disk partition.
    > I would be grateful if in addition to your comments you could point me to
    > the relevant source (URL or else).
    >
    > Thanks,
    > Akyl.
    >


    A great topic for news:comp.programming (follups set).

    As other people have stated, measure.

    With memory access, you may get a "page fault" which tells
    the operating system to swap in memory from a "virtual page".
    This would be your worst case timing.

    When reading from a file, your worst case is that you
    have to read from the file without any caching. Best
    case is that the data is in a cache somewhere.

    Another alternative is for you to allocate some memory and
    read in a chunk of data. You can set the chunk size yourself.

    As for references, search the web for:
    "Operating Systems" "page fault"
    or "double buffering"
    or "spooling".

    --
    Thomas Matthews

    C++ newsgroup welcome message:
    http://www.slack.net/~shiva/welcome.txt
    C++ Faq: http://www.parashift.com/c -faq-lite
    C Faq: http://www.eskimo.com/~scs/c-faq/top.html
    alt.comp.lang.learn.c-c++ faq:
    http://www.raos.demon.uk/acllc-c /faq.html
    Other sites:
    http://www.josuttis.com -- C++ STL Library book
     
    Thomas Matthews, Feb 23, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    5
    Views:
    372
    -berlin.de
    Dec 24, 2004
  2. s.subbarayan

    Dynamic memory allocation and memory leak...

    s.subbarayan, Mar 18, 2005, in forum: C Programming
    Replies:
    10
    Views:
    707
    Eric Sosman
    Mar 22, 2005
  3. Mark Healey
    Replies:
    4
    Views:
    298
    Ben C
    Apr 16, 2006
  4. Atemporal
    Replies:
    4
    Views:
    375
    Lionel B
    Jun 9, 2008
  5. Philipp
    Replies:
    21
    Views:
    1,134
    Philipp
    Jan 20, 2009
Loading...

Share This Page