strange file.write() behavior on windows: $ConvertToNonresident, $ReplaceAttribute2

W

welch

while taking some rough disk performance measures on windows machines,
and snooping with FileMon, i've noticed some odd behavior

here's the little nul-writer i'm running:
def writeTest(nBlocks, blockSize):
"""write nBlocks*blockSize nuls to a file"""
f = open("writeTest.out", "wb")
nulBlock = '\0'*blockSize
while nBlocks:
f.write(nulBlock)
nBlocks -= 1
f.close()

when i invoke this with a blockSize of 64K or less, no surprises.
here's a row of FileMon output for the 64K block size:
2:44:48 PM python.exe:3064 WRITE C:\writeTest.out SUCCESS Offset:
214106112 Length: 65536

when i invoke this with larger blocksizes (say, 655360), FileMon output
looks like this:
2:37:46 PM python.exe:3064 WRITE C:\writeTest.out Offset: 140902400
Length: 655360
2:37:46 PM python.exe:3064 WRITE C:\$ReplaceAttribute2 SUCCESS Offset:
140902400 Length: 65536
....(the $ReplaceAttribute2 line repeats several times, writing 65536
and advancing the offset by 65536 each time)

sometimes, depending on the phase of the moon, instead of
$ReplaceAttribute2 the write goes to $ConvertToNonresident. within a
single run of writeTest the write always goes to the same one,
whichever one that is. these runs are always slower (sometimes greatly)
than writing the same with 64K blocks, even though many fewer
file.write()'s are being issued because of the larger block size.

finally, where you'd expect an even 10 of the WRITE
C:\$ReplaceAttribute2 lines per WRITE C:\writeTest.out lines in the
example above, instead FileMon reports 8 lines for the first, 6 for the
second, 8 for the third, 6 for the fourth, etc... i've no idea if this
means FileMon is missing messages, but this pattern is absolutely
consistent across every run i've made, on both xp and win2k3 server
machines.

a look at the python fileobject.c source shed no light for me, anyone
know what could be going on? the "equivalent" c version of writeTest
using fwrite() shows a succession of 1K blocks writing to the named
file (even when fwrite is given 64K blocks), and no mysterious
$ReplaceAttribute2 lines

-- w
 
J

jepler

This is really just a snide joke at the expense of Microsoft, but have you
checked the MSDN documentation about ConvertToNonresident or ReplaceAttribute2?

Your search - site:msdn.microsoft.com ConvertToNonresident OR
ReplaceAttribute2 - did not match any documents.
-- google.com

Internally, Python also simply calls fwrite(). Are you using the same C
runtime for both sets of tests?

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDe8alJd01MZaTXX0RAjpDAKCPpv7J3sJ21YszxDjknKG0dEvjRwCfQfP5
oU/o2FKjl0cDqqr3dDufeyU=
=C/YG
-----END PGP SIGNATURE-----
 
F

Fredrik Lundh

welch said:
while taking some rough disk performance measures on windows machines,
and snooping with FileMon, i've noticed some odd behavior
sometimes, depending on the phase of the moon, instead of
$ReplaceAttribute2 the write goes to $ConvertToNonresident. within a
single run of writeTest the write always goes to the same one,
whichever one that is. these runs are always slower (sometimes greatly)
than writing the same with 64K blocks, even though many fewer
file.write()'s are being issued because of the larger block size.

finally, where you'd expect an even 10 of the WRITE
C:\$ReplaceAttribute2 lines per WRITE C:\writeTest.out lines in the
example above, instead FileMon reports 8 lines for the first, 6 for the
second, 8 for the third, 6 for the fourth, etc... i've no idea if this
means FileMon is missing messages, but this pattern is absolutely
consistent across every run i've made, on both xp and win2k3 server
machines.

a look at the python fileobject.c source shed no light for me, anyone
know what could be going on? the "equivalent" c version of writeTest
using fwrite() shows a succession of 1K blocks writing to the named
file (even when fwrite is given 64K blocks), and no mysterious
$ReplaceAttribute2 lines

you're seeing activities by the Windows caching system (c:\$ stuff is various
NTFS data streams). unless you're writing low-level drivers, you don't really
have to care about what it does, and when it does what...

</F>
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,900
Latest member
Nell636132

Latest Threads

Top