W
Wolfgang Maier
Antoine said:
The Linux version of "man posix_fadvise" probably holds the answer:
"In kernels before 2.6.18, POSIX_FADV_NOREUSE had the same semantics
as POSIX_FADV_WILLNEED. This was probably a bug; since kernel
2.6.18, this flag is a no-op."
"POSIX_FADV_DONTNEED attempts to free cached pages associated with the
specified region. This is useful, for example, while streaming large
files. A program may periodically request the kernel to free cached
data that has already been used, so that more useful cached pages are
not discarded instead."
So, in summary:
- POSIX_FADV_NOREUSE doesn't do anything on (modern) Linux kernels
- POSIX_FADV_DONTNEED must be called *after* you are done with a range of
data, not before you read it (note that I haven't tested to confirm it >)
Regards
Antoine.
Hi Antoine,
you're right and thanks a lot for this great piece of information.
The following quick check works like a charm now:
.... d = fo.read(chunk_size)
.... pos += chunk_size
.... if pos > 2000000000:
.... print ('another 2GB read, flushing')
.... os.posix_fadvise(fo.fileno(), last_flush, last_flush+pos,
os.POSIX_FADV_DONTNEED)
.... last_flush += pos
.... pos = 0
With this page caching for my huge file (30 GB in that case) still occurs,
of course, but it never occupies more than 2 GB of memory. This way it
should interfere less with cached data of other applications.
Have to test carefully how much that improves overall performance of the
system, but for the moment I'm more than happy!
Best wishes,
Wolfgang
P.S.: Maybe these new os module features could use a bit more documentation?