fsync() doesn't work as advertised?

[Brian D]

If I'm running a process in a loop that runs for a long time, I
occasionally would like to look at a log to see how it's going.

I know about the logging module, and may yet decide to use that.

Still, I'm troubled by how fsync() doesn't seem to work as advertised:

http://docs.python.org/library/os.html

"If you’re starting with a Python file object f, first do f.flush(),
and then do os.fsync(f.fileno())"

Has anyone else experienced and overcome this issue?

What I've seen is that flush() alone produces a complete log when the
loop finishes. When I used fsync(), I lost all of the write entries
except the first, along with odd error trap and the last entry.

 

[Antoine Pitrou]

Le Mon, 04 Jan 2010 08:09:56 -0800, Brian D a écrit :

What I've seen is that flush() alone produces a complete log when the
loop finishes. When I used fsync(), I lost all of the write entries
except the first, along with odd error trap and the last entry.

Perhaps you are writing to the file from several threads or processes at
once?

By the way, you shouldn't need fsync() if you merely want to look at the
log files, because your OS will have an up-to-date view of the file
contents anyway. fsync() is useful if you want to be sure the data has
been written to the hard disk drive, rather than just kept in the
operating system's filesystem cache.

 

[Brian D]

Le Mon, 04 Jan 2010 08:09:56 -0800, Brian D a écrit :




Perhaps you are writing to the file from several threads or processes at
once?

By the way, you shouldn't need fsync() if you merely want to look at the
log files, because your OS will have an up-to-date view of the file
contents anyway. fsync() is useful if you want to be sure the data has
been written to the hard disk drive, rather than just kept in the
operating system's filesystem cache.

Sure -- I hadn't considered how threads might affect the write
process. That's a good lead to perhaps fixing the problem.

Thanks for your help, Antoine.

 

[Nobody]

If I'm running a process in a loop that runs for a long time, I
occasionally would like to look at a log to see how it's going.

I know about the logging module, and may yet decide to use that.

Still, I'm troubled by how fsync() doesn't seem to work as advertised:

http://docs.python.org/library/os.html

"If you’re starting with a Python file object f, first do f.flush(),
and then do os.fsync(f.fileno())"

The .flush() method (and the C fflush() function) causes the
contents of application buffers to be sent to the OS, which basically
copies the data into the OS-level buffers.

fsync() causes the OS-level buffers to be written to the physical drive.

File operations normally use the OS-level buffers; e.g. if one process
write()s to a file and another process read()s it, the latter will see
what the former has written regardless of whether the data has been
written to the drive.

The main reason for using fsync() is to prevent important data from being
lost in the event of an unexpected reboot or power-cycle (an expected
reboot via the "shutdown" or "halt" commands will flush all OS-level
buffers to the drive first). Other than that, fsync() is almost invisible
(I say "almost", as there are mechanisms to bypass the OS-level buffers,
e.g. the O_DIRECT open() flag).

 

[Brian D]

The .flush() method (and the C fflush() function) causes the
contents of application buffers to be sent to the OS, which basically
copies the data into the OS-level buffers.

fsync() causes the OS-level buffers to be written to the physical drive.

File operations normally use the OS-level buffers; e.g. if one process
write()s to a file and another process read()s it, the latter will see
what the former has written regardless of whether the data has been
written to the drive.

The main reason for usingfsync() is to prevent important data from being
lost in the event of an unexpected reboot or power-cycle (an expected
reboot via the "shutdown" or "halt" commands will flush all OS-level
buffers to the drive first). Other than that,fsync() is almost invisible
(I say "almost", as there are mechanisms to bypass the OS-level buffers,
e.g. the O_DIRECT open() flag).

An excellent explanation of the process. I've seen this in other
programming environments, so I could visualize something to that
effect, but couldn't have verbalized it. I certainly have a better
idea what's happening. Thanks for the contribution.