dynamic memory for large arrays

R

Richard Bos

Akyl Tulegenov said:
I've got a question which is rather system specific case.

Quite. So why post it here? We can't answer your question.
Suppose I am dealing with a large array of ~4Gig of entities.
The question is : what is faster , to address it through a binary file
created on disk or using dynamic memory which is extended through a large
swap file (again on disk)? Note that swap file is created at the
point of the disk partition.

Who knows? Measure, measure, measure. This depends not only on the OS,
the compiler, and the technique used to map the dynamic memory, but
probably also on the relative speeds of the swap disk and the normal FS,
and several others. Measure, and then assume your measurements are
correct for you and nobody else.
I would be grateful if in addition to your comments you could point me to
the relevant source (URL or else).

"The" relevant source? I don't think you quite understand this "system
specific" idea. There must be hundreds of relevant sources. Do a web
search; _you_ know what's inside your system, we don't.

Richard
 
A

Akyl Tulegenov

Dear All!

I've got a question which is rather system specific case.
Suppose I am dealing with a large array of ~4Gig of entities.
The question is : what is faster , to address it through a binary file
created on disk or using dynamic memory which is extended through a large
swap file (again on disk)? Note that swap file is created at the
point of the disk partition.
I would be grateful if in addition to your comments you could point me to
the relevant source (URL or else).

Thanks,
Akyl.
 
G

Guillaume

I've got a question which is rather system specific case.

You're right, it's off-topic and risks getting flamed because of that.
Suppose I am dealing with a large array of ~4Gig of entities.
The question is : what is faster , to address it through a binary file
created on disk or using dynamic memory which is extended through a large
swap file (again on disk)? Note that swap file is created at the
point of the disk partition.
I would be grateful if in addition to your comments you could point me to

I'd suggest using memory-mapped I/O if you have access to this.
You'll get the best of both worlds.

If you have access to libc/glibc, mmap() is your friend.
 
T

Thomas Matthews

Akyl said:
Dear All!

I've got a question which is rather system specific case.
Suppose I am dealing with a large array of ~4Gig of entities.
The question is : what is faster , to address it through a binary file
created on disk or using dynamic memory which is extended through a large
swap file (again on disk)? Note that swap file is created at the
point of the disk partition.
I would be grateful if in addition to your comments you could point me to
the relevant source (URL or else).

Thanks,
Akyl.

A great topic for (follups set).

As other people have stated, measure.

With memory access, you may get a "page fault" which tells
the operating system to swap in memory from a "virtual page".
This would be your worst case timing.

When reading from a file, your worst case is that you
have to read from the file without any caching. Best
case is that the data is in a cache somewhere.

Another alternative is for you to allocate some memory and
read in a chunk of data. You can set the chunk size yourself.

As for references, search the web for:
"Operating Systems" "page fault"
or "double buffering"
or "spooling".

--
Thomas Matthews

C++ newsgroup welcome message:
http://www.slack.net/~shiva/welcome.txt
C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.learn.c-c++ faq:
http://www.raos.demon.uk/acllc-c++/faq.html
Other sites:
http://www.josuttis.com -- C++ STL Library book
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top