Re: Flushing buffer on file copy on linux

Discussion in 'Python' started by Cameron Simpson, Aug 15, 2012.

  1. On 14Aug2012 22:55, J <> wrote:
    | Now, the problem I have is that linux tends to buffer data writes to a
    | device, and I want to work around that.

    To what _specific_ purpose? Benchmarking? Ensuring the device can be
    pulled? Ensuring another program can see the data? (The last should be
    taken care of regardless of the Linux OS level buffering.)

    | When run in normal non-stress
    | mode, the program is slow enough that the linux buffers flush and put
    | the file on disk before the hash occurs. However, when run in stress
    | mode, what I'm finding is that it appears that the files are possibly
    | being hashed while still in the buffer, before being flushed to disk.

    You're probably right, but how are you inferring this?

    | Generate the parent data file
    | hash parent
    | instead of copy, open parent and write to a new file object on disk
    | with a 0 size buffer
    | or flush() before close()
    | hash the copy.
    |
    | Does that seem reasonable? or is there a cleaner way to copy a file
    | from one place to another and ensure the buffers are properly flushed
    | (maybe something in os or sys that forces file buffers to be flushed?)

    For OS buffers you either want to call sync() (flushes _all_ OS buffers
    to disc before returning) or fsync(open-file-handle), which flushes at
    least the blocks for that file (in practice, typically everything
    outstanding for the same filesystem, alas).

    So look at os.fsync in python.

    You will need to do this after a python-level data flush but before file
    close:

    with open("output-file", "w") as fp:
    fp.write(lots of stuff)...
    fp.flush()
    os.fsync(fp.fileno)

    But be clear about your purpose: why do you care that the disc writes
    themselve are complete? There are legitimate reasons for this, but
    unless it is benchmarking or utterly mad (eg database) data integrity,
    they generally aren't:)

    Cheers,
    --
    Cameron Simpson <>

    English is a living language, but simple illiteracy is no basis for
    linguistic evolution. - Dwight MacDonald
    Cameron Simpson, Aug 15, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Eric

    Flushing the Buffer in .Net

    Eric, Apr 25, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    605
  2. arnuld
    Replies:
    5
    Views:
    790
    =?ISO-8859-1?Q?Erik_Wikstr=F6m?=
    Mar 18, 2007
  3. Ethan Metsger

    Decorators and buffer flushing

    Ethan Metsger, Feb 28, 2008, in forum: Python
    Replies:
    0
    Views:
    274
    Ethan Metsger
    Feb 28, 2008
  4. J
    Replies:
    0
    Views:
    153
  5. Antoine Pitrou

    Re: Flushing buffer on file copy on linux

    Antoine Pitrou, Aug 15, 2012, in forum: Python
    Replies:
    0
    Views:
    214
    Antoine Pitrou
    Aug 15, 2012
Loading...

Share This Page