Help me, a friend of mine wrote a program in C# and I wrote the sameprogram in C..His is faster than

[Tor Rustad]

fclose() just closes the FILE* (and related fd); it does _not_ guarantee
that the data is actually physically on the disk. There may be some
OS-specific function that will give you the indication you're wrongly
assuming you're getting, but it's not on by default.


It's completely normal. That's why one should always shut down machines
cleanly instead of pulling the plug -- and why data (and filesystems)
tend to get corrupted when the power goes out. Even the disks lie to
the OS about when data is written; as soon as the data is in the drive's
cache, it tells the OS it's done writing so that the OS can reuse the
buffer(s). Only top-of-the-line controllers with battery-backed caches
are immune from these sorts of problems (and even then, you have to boot
the machine again and let it finish writing before removing the disk
from the system).

Some years ago, I was told that lazy commit, was the reason we didn't
run *any* production systems on Linux, now it's allowed in *some* cases
to use this OS. If an OS don't flush system buffers when a file is
closed, or is doing a fflush() in an async manner, then it is impossible
to code a bullet-proof C program with file I/O, in that environment.

The C standard is silent here, data has only to be transfered to the
host environment, depending on QoI, this data may written to persistent
storage, before fclose() return, or program exit().

Just consider this, a DB program start a transaction with "begin work",
then perform lots of updates, before "commit work"... if you can't trust
that the data has been saved on disk, then you risk that the transaction
can be lost, in case of e.g. power outage or HW failures (e.g. disk
controller failure).

If a disk controller lie, the flaw is limited to that unit. If an OS
lie, that is an important thing to know about, and I don't think I would
like to use such an OS, for work related production systems.

It's not so amazing when you realize you're measuring the speed of the
OS's I/O system, not the hardware's.

Well, in the benchmark I ran, the result couldn't be explained by disk
cache alone, since this is somewhere in the range of 8 Mb - 32 Mb, while
the measured IO peek, was >150 Mb/s too high.

Also, since I did an explicit call to fsync() in the last test-case, the
data should have been committed to disk, before the stop timer was set.
A clear sign that something was wrong, was that fsync()'ing, boosted the
performance by another 100 Mb/s!

I think the simple answer here, is that the clock() implementation on
Linux, measured processor time used in *user* space only, ignoring the
(significant) time spent in system calls.

 

[Willem]

Tor wrote:
) Some years ago, I was told that lazy commit, was the reason we didn't
) run *any* production systems on Linux, now it's allowed in *some* cases
) to use this OS. If an OS don't flush system buffers when a file is
) closed, or is doing a fflush() in an async manner, then it is impossible
) to code a bullet-proof C program with file I/O, in that environment.

You can actually turn that off in Linux, you know.

) I think the simple answer here, is that the clock() implementation on
) Linux, measured processor time used in *user* space only, ignoring the
) (significant) time spent in system calls.

Hmm, not quite. The clock() function is supposed to only measure processor
time used, so time spent waiting for I/O to finish will not be counted.
(After all, during that time, other tasks can get their share of CPU time.)
It does count CPU time spent in the system call though, AFAIK.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

 

[Tor Rustad]

Tor wrote:
) Some years ago, I was told that lazy commit, was the reason we didn't
) run *any* production systems on Linux, now it's allowed in *some* cases
) to use this OS. If an OS don't flush system buffers when a file is
) closed, or is doing a fflush() in an async manner, then it is impossible
) to code a bullet-proof C program with file I/O, in that environment.

You can actually turn that off in Linux, you know.

It appears, that write-back cache is turned off by default on e.g.
Windows and Solaris, but not on my Linux distro. I see that InnoDB MySQL
people suggest doing e.g.

hdparm -W 0 /dev/sda

to turn it off on Linux. BUT to my mind, this must be a work around,
since fsync() should do the job. If fsync() cannot be trusted to do the
right thing, that's a nasty Linux bug, and write-back cache must/should
be turned off manually.

) I think the simple answer here, is that the clock() implementation on
) Linux, measured processor time used in *user* space only, ignoring the
) (significant) time spent in system calls.

Hmm, not quite. The clock() function is supposed to only measure processor
time used, so time spent waiting for I/O to finish will not be counted.
(After all, during that time, other tasks can get their share of CPU time.)
It does count CPU time spent in the system call though, AFAIK.

You might be right, the point is anyway that the actual I/O was not
counted by clock(), as such, the clock() function is rather useless for
measuring performance of I/O bound programs.

 

[Richard]

It appears, that write-back cache is turned off by default on
e.g. Windows and Solaris, but not on my Linux distro. I see that
InnoDB MySQL people suggest doing e.g.

hdparm -W 0 /dev/sda

to turn it off on Linux. BUT to my mind, this must be a work around,
since fsync() should do the job. If fsync() cannot be trusted to do
the right thing, that's a nasty Linux bug, and write-back cache
must/should be turned off manually.

Assuming journal markers are stored on the same device then I don't see
what the issue is. Lazy on or off, there is still the opportunity for
data loss and any restarting program (application or os) must examine
journal marks to determine the last safe write.