Re: Flushing buffer on file copy on linux

Discussion in 'Python' started by Antoine Pitrou, Aug 15, 2012.

  1. J <dreadpiratejeff <at> gmail.com> writes:
    >
    > Now, the problem I have is that linux tends to buffer data writes to a
    > device, and I want to work around that. When run in normal non-stress
    > mode, the program is slow enough that the linux buffers flush and put
    > the file on disk before the hash occurs. However, when run in stress
    > mode, what I'm finding is that it appears that the files are possibly
    > being hashed while still in the buffer, before being flushed to disk.


    Your analysis is partly wrong. It is right that the files can be hashed from
    in-memory buffers; but even if you flush the buffers to disk using standard
    techniques (such as fsync()), those buffers still exist in memory, and
    therefore the file will still be hashed from memory (for obvious efficiency
    reasons).

    I don't think there's a portable solution to get away entirely with the
    in-memory buffers, but under Linux you can write "1" to the special file
    /proc/sys/vm/drop_caches:

    $ sudo sh -c "echo 1 > /proc/sys/vm/drop_caches"

    Or, to quote the /proc man page:

    /proc/sys/vm/drop_caches (since Linux 2.6.16)
    Writing to this file causes the kernel to drop clean
    caches, dentries and inodes from memory, causing that memâ€
    ory to become free.

    To free pagecache, use echo 1 > /proc/sys/vm/drop_caches;
    to free dentries and inodes, use echo 2 >
    /proc/sys/vm/drop_caches; to free pagecache, dentries and
    inodes, use echo 3 > /proc/sys/vm/drop_caches.

    Because this is a nondestructive operation and dirty
    objects are not freeable, the user should run sync(8)
    first.


    Regards

    Antoine.


    --
    Software development and contracting: http://pro.pitrou.net
    Antoine Pitrou, Aug 15, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Eric

    Flushing the Buffer in .Net

    Eric, Apr 25, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    602
  2. arnuld
    Replies:
    5
    Views:
    790
    =?ISO-8859-1?Q?Erik_Wikstr=F6m?=
    Mar 18, 2007
  3. Ethan Metsger

    Decorators and buffer flushing

    Ethan Metsger, Feb 28, 2008, in forum: Python
    Replies:
    0
    Views:
    274
    Ethan Metsger
    Feb 28, 2008
  4. J
    Replies:
    0
    Views:
    152
  5. Cameron Simpson

    Re: Flushing buffer on file copy on linux

    Cameron Simpson, Aug 15, 2012, in forum: Python
    Replies:
    0
    Views:
    159
    Cameron Simpson
    Aug 15, 2012
Loading...

Share This Page