mmap and large files

F

Fishtank

Hello,
On linux, suppose I have a 64 bit system with 32GB ram. If I were to
call mmap
mmap(void *start, size_t length, int prot, int flags, int fd, off_t
offset);

offset equal to 0 and length the length of a 10 gb file, will mmap map
the *entire* file into memory?

Or will it (behind the scenes) load in chunks?
So e.g were the file to be 35GB, would a call to mmap fail? And the
way around would be to load say X mb chunks by shifting the offset.

Thank you in advance
Sapsi
 
N

Nick

Fishtank said:
Hello,
On linux, suppose I have a 64 bit system with 32GB ram. If I were to
call mmap
mmap(void *start, size_t length, int prot, int flags, int fd, off_t
offset);

offset equal to 0 and length the length of a 10 gb file, will mmap map
the *entire* file into memory?

Or will it (behind the scenes) load in chunks?
So e.g were the file to be 35GB, would a call to mmap fail? And the
way around would be to load say X mb chunks by shifting the offset.

That really is a Linux question (Unix at the very best) and not a C
one. I program in C on Linux and haven't a clue of this. So this isn't
the best place to ask.
 
L

Lew Pitcher

Hello,
On linux, suppose I have a 64 bit system with 32GB ram. If I were to
call mmap
mmap(void *start, size_t length, int prot, int flags, int fd, off_t
offset);

offset equal to 0 and length the length of a 10 gb file, will mmap map
the *entire* file into memory?

Or will it (behind the scenes) load in chunks?
So e.g were the file to be 35GB, would a call to mmap fail? And the
way around would be to load say X mb chunks by shifting the offset.

comp.os.linux.development.aps is a better forum for this question. F'ups set
to there.

<remark status="off-topic in comp.lang.c">
AFAICT, mmap will, if it /can/, map the entire file into process memory.
That /does not/ mean that it /loads/ the file into memory, either wholly,
or in chunks. What it does mean is that directed access into that memory
region will cause the kernel to retrieve and manipulate pages of the file,
such that the file /appears/ to be mapped contiguously.

However, that behaviour is contingent on mmap() returning a success value.
It /could/ return ((char *) -1) and set errno to EINVAL, indicating that
the requested length was too big.
</remark>

HTH
 
G

Gene

Hello,
On linux, suppose I have a 64 bit system with 32GB ram. If I were to
call mmap
mmap(void *start, size_t length, int prot, int flags,  int fd, off_t
offset);

offset equal to 0 and length the length of a 10 gb file, will mmap map
the *entire* file into memory?

Or will it (behind the scenes) load in chunks?
So e.g were the file to be 35GB, would a call to mmap fail? And the
way around would be to load say X mb chunks by shifting the offset.

As other have said, this isn't a C problem. mmap() is really a way of
touching the virtual memory system through the file abstraction. What
happens behind the scenes is up to the OS. However, because you are
working with _virtual_ memory, there is no reason a priori that the
size of an mmap()ed file should be limited to _physical_ memory size.
While I don't know how mmap() is implemented in Linux, I'd guess that
it merely sets up your file as something like a process-local swap
file, with pages loaded in physical memory on demand and kicked out by
the same replacement mechanisms used for swapping (clock algorithm or
whatever...).

I suggest you jut try it. If it works, smile. If it doesn't, take it
to the Linux or other relevant group.
 
N

Nobody

On linux, suppose I have a 64 bit system with 32GB ram. If I were to
call mmap
mmap(void *start, size_t length, int prot, int flags, int fd, off_t
offset);

offset equal to 0 and length the length of a 10 gb file, will mmap map
the *entire* file into memory?
Yes.

Or will it (behind the scenes) load in chunks?

Yes.

mmap() *maps* the file; it doesn't *load* it.
So e.g were the file to be 35GB, would a call to mmap fail?

No. You only need sufficient free *address space*; the amount of physical
RAM available isn't relevant.
 
A

Antoninus Twink

No. You only need sufficient free *address space*; the amount of
physical RAM available isn't relevant.

mmap on Linux accepts a MAP_NORESERVE flag, meaning

Do not reserve swap space for this mapping. When swap space is
reserved, one has the guarantee that it is possible to modify the
mapping. When swap space is not reserved one might get SIGSEGV
upon a write if no physical memory is available.

This suggests that by default (i.e. if this flag is not supplied) then
trying to mmap a huge file is likely to fail even with a 64-bit address
space.
 
N

Nobody

No. You only need sufficient free *address space*; the amount of
physical RAM available isn't relevant.

mmap on Linux accepts a MAP_NORESERVE flag, meaning

Do not reserve swap space for this mapping. When swap space is
reserved, one has the guarantee that it is possible to modify the
mapping. When swap space is not reserved one might get SIGSEGV
upon a write if no physical memory is available.

This suggests that by default (i.e. if this flag is not supplied) then
trying to mmap a huge file is likely to fail even with a 64-bit address
space.

Only if you also use both PROT_WRITE and MAP_PRIVATE (or MAP_ANONYMOUS).

If it's read-only, you *will* get SIGSEGV upon write regardless of the
availability of swap or physical memory. If you use MAP_SHARED, writes
will go to the underlying file, not swap.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,480
Members
44,900
Latest member
Nell636132

Latest Threads

Top