Strange performance issue

[Dan Stromberg]

I'm rewriting 3 programs as one program - from Python with Tkinter to
Python with pygtk, both on Windows XP.

My new version formats an SD card and preallocates some file space in
about 3 minutes with "Optimize Performance" selected, and in about 30
minutes with "Optimize for Quick Removal" selected. Needless to say, I
don't like the 27 minute penalty much.

But the old version of the program formats and preallocates in 3 minutes
irrespective of whether the drive is in "optimize performance" or
"optimize for quick removal".

I'm baffled. I've gone over the old code, and found nothing relevant
looking, and I've never seen anything this puzzling in my years of work
on *ix.

Can someone please elucidate?

(I've also written a pair of minimal programs in Python and C that
exhibit the same behavior on this XP system - so it's not just my GUI app)

Thanks!

PS: Here's the minimal python test program I mentioned. It's dog slow
unless I Optimize for Performance the drive.:

#!/cygdrive/c/Python26/python

import os
import time

one_meg = 'a'*2**20

t0 = time.time()

i = 0

file = open('remove-me', 'w')
while True:
i += 1
file.write(one_meg)
t1 = time.time()
bits_written = i*2**20*8
duration = t1-t0
print i, str(duration)+'s', bits_written/(duration*1024*1024),
'Mbps'
if duration > 120:
break
file.close()

os.unlink('remove-me')

 

[Steven D'Aprano]

I'm rewriting 3 programs as one program - from Python with Tkinter to
Python with pygtk, both on Windows XP.

My new version formats an SD card and preallocates some file space in
about 3 minutes with "Optimize Performance" selected, and in about 30
minutes with "Optimize for Quick Removal" selected. Needless to say, I
don't like the 27 minute penalty much.

I don't understand what that means. How are you formatting the SD card?

I'm guessing that Optimize for Quick Removal means that every write is
immediately synced to disk. That will probably be slow, no matter what.

But the old version of the program formats and preallocates in 3 minutes
irrespective of whether the drive is in "optimize performance" or
"optimize for quick removal".

I suspect that if there was no performance penalty in the old version, it
was because you inadvertently were always using "Optimize Performance" no
matter what.

BTW, if you want to pre-allocate some space, and you don't care what is
in the file, you *may* be able to use file.truncate() to do so quickly.
Despite the name, it doesn't just truncate files, it can also be used to
extend them. (At least on POSIX systems.)

 

[Ulrich Eckhardt]

My new version formats an SD card and preallocates some file space in
about 3 minutes with "Optimize Performance" selected, and in about 30
minutes with "Optimize for Quick Removal" selected. Needless to say, I
don't like the 27 minute penalty much.

For performance, the driver will probably use delayed write operations,
buffering etc. For quick removal, it will probably make all operations
atomic, i.e. perform a write operation and not return until the data has
really reached the SD card.

But the old version of the program formats and preallocates in 3 minutes
irrespective of whether the drive is in "optimize performance" or
"optimize for quick removal".

Somehow, I guess the new version does many small writes while the old one
doesn't. When optimizing for quick removal, those operations add up,
otherwise their overhead is negligible.

one_meg = 'a'*2**20

That's one mib, not one meg.

file = open('remove-me', 'w')

Try adding the 'b' flag, too. I wouldn't expect it to affect the IO speed,
but it is one factor that adds up. Otherwise, I looked over your program
and couldn't find anything that would explain a problem. Just wondering,
what transfer speed do you get with the two versions? What is the
theoretical maximum for the SD card?

Uli

 

[Dan Stromberg]

I don't understand what that means. How are you formatting the SD card?

The FormatEx function via ctypes.

I'm guessing that Optimize for Quick Removal means that every write is
immediately synced to disk. That will probably be slow, no matter what.

Yes, presumably Optimize for Quick Removal is a write-through cache
(synchronous) and "Optimize for Performance" is a write-back cache
(asynchronous).

I suspect that if there was no performance penalty in the old version, it
was because you inadvertently were always using "Optimize Performance" no
matter what.

I'm pretty confident that unless Windows was lying to me, that I did
some tests with Optimize for Performance and some tests without.

BTW, if you want to pre-allocate some space, and you don't care what is
in the file, you *may* be able to use file.truncate() to do so quickly.
Despite the name, it doesn't just truncate files, it can also be used to
extend them. (At least on POSIX systems.)

On a POSIX system, I'd normally just seek and write a single null to get
a file with holes. But this is not only Windows, but FAT32 - and the
data will later be read without going through the filesystem. It seems
prudent to actually write the blocks this time.


Thanks, and see my other message (to be sent momentarily) for an
apparent solution.

 

[Dan Stromberg]

For performance, the driver will probably use delayed write operations,
buffering etc. For quick removal, it will probably make all operations
atomic, i.e. perform a write operation and not return until the data has
really reached the SD card.

The driver may delay writes, but I'd think it more likely that the
filesystem or buffer cache would be doing so.

Somehow, I guess the new version does many small writes while the old one
doesn't. When optimizing for quick removal, those operations add up,
otherwise their overhead is negligible.
Nope, same block size.

That's one mib, not one meg.

You're aware that a lot of people are ignoring the new units?

Try adding the 'b' flag, too. I wouldn't expect it to affect the IO speed,
but it is one factor that adds up.

This appears to have wholly accounted for the problem! Thank you very much.

Adding a "b" to my open sped up the writes by a factor of about 15.

Otherwise, I looked over your program
and couldn't find anything that would explain a problem. Just wondering,
what transfer speed do you get with the two versions? What is the
theoretical maximum for the SD card?

I don't know what the theoretical max write speeds of the USB bus, card
reader and card are, but I was getting over 10x faster using the same
card and card reader on an ancient Thinkpad running Ubuntu. With the
"b" specified, the XP system is now faster than the Thinkpad.

Interesting that the "b" would make such a performance difference.
You'd think it'd just be, in C, looking for newlines and adding a
carriage return after them into a buffer of potentially double the size,
then writing as usual. The data I've been writing so far -has- no newlines.

Thanks!