HDD Burn In with Python

[hokiegal99]

Hi Again,

I have Macs, Windows and Linux PCs. I'd like to have a generic HDD
burn in program to test new drives on all platforms. I know Python is
portable enough to handle the platform differences, but would it be a
good language for this type of application? The process would involve
lots of reads and writes all over the drives for an extended period of
time (several hours). Anybody else done something similar to this? Any
suggestions?

Thanks !!!

 

[Anand Pillai]

I dont know of any application that currently does this.

I think for applications that perform a lot of I/O, Python
might not be a good choice. The file I/O in Python is written
in optimized C, but still you would be better off writing
a C program for this, that is portable. For example, if you
are targeting only Unix/Linux platforms there wont be much
difficulty writing a portable C app optimized, say for the
gcc compiler.

Since you have Windows also mentioned, I recommend Perl.
Perl is a bit faster than Python in system related tasks and
it is portable also. Python, though my favorite language comes
only second here. And of course, forget about Jython ;-)

If you do write your app in Python, perform benchmarks to
time the Python app itself. A good way to do this is by
using the 'Profile' class (of 'profile' module) by its 'calibrate'
method. This will make sure that the timings are done accurately.
This will be the major difference between a C app and a Python app,
since the C system calls are fast enough that their times can be
ignored.

It will be a tough job to get it portable and just right
on all platforms. So let me know if you do come up with an app
like that in Python.

-Anand

 

[Irmen de Jong]

Perl is a bit faster than Python in system related tasks and
it is portable also.
[...]

If you do write your app in Python, perform benchmarks to
time the Python app itself.

Hokiegal99 didn't ask for a disk benchmark tool that has to
be high-performant, he (she?) wanted a harddisk IO stress tool.
Just letting your harddisk work hard for a long time doesn't
require optimized C code. Python is perfectly suitable for that.

The only problem is, that for a true harddisk stress test, you
will want to access sectors all over the disk. I'm not aware of
any mechanism to do that with portable Python...
(although you could format the disk in one huge partition and
then write random files to it until the disk is full).

--Irmen

 

[hokieghal99]

I dont know of any application that currently does this.

I think for applications that perform a lot of I/O, Python
might not be a good choice. The file I/O in Python is written
in optimized C, but still you would be better off writing
a C program for this, that is portable. For example, if you
are targeting only Unix/Linux platforms there wont be much
difficulty writing a portable C app optimized, say for the
gcc compiler.

Since you have Windows also mentioned, I recommend Perl.
Perl is a bit faster than Python in system related tasks and
it is portable also. Python, though my favorite language comes
only second here. And of course, forget about Jython ;-)

Speed isn't a huge concern for this test. Heck, it will take many hours
or possibly a day or two depending on the size of the drive. I like Perl
as well, but it's too convoluted. I sometimes look back at one month old
Perl scripts and wonder what the heck they do. To me, that's scary. I
have a close friend who heads up a large NOC at a big US university and
they have banned Perl for this very reason. I only use Perl when there
is no other option, and I think Python is a valid choice for this task.

If you do write your app in Python, perform benchmarks to
time the Python app itself. A good way to do this is by
using the 'Profile' class (of 'profile' module) by its 'calibrate'
method. This will make sure that the timings are done accurately.
This will be the major difference between a C app and a Python app,
since the C system calls are fast enough that their times can be
ignored.

This is a good tip. Thanks, I'll try it.

 

[hokieghal99]

Just letting your harddisk work hard for a long time doesn't
require optimized C code. Python is perfectly suitable for that.

Yes, I agree! Now, if I only knew how to conduct a proper stress test.
Read, write, verify reads & writes... would that be sufficient?

The only problem is, that for a true harddisk stress test, you
will want to access sectors all over the disk. I'm not aware of
any mechanism to do that with portable Python...
(although you could format the disk in one huge partition and
then write random files to it until the disk is full).

That's an idea. But what approach might one take if there were already
files and folders on the volume that should not be destroyed? Some type
of non-destructive test?

 

[Terry Reedy]

Irmen de Jong wrote:


That's an idea. But what approach might one take if there were already
files and folders on the volume that should not be destroyed? Some type
of non-destructive test?

You can start start with the root directory and use os.path.walk to read
everything already the disk.

You could also get the assembly coded SpinRite5 (<$100) which will
systematically read and test every block, including boot sectors, on the
disk (moving each block of data temporarily to another place while write
testing). It also fixes bad formatting and has twice saved a disk for me
that ScanDisk could not.

Terry J. Reedy

 

[hokieghal99]

You could also get the assembly coded SpinRite5 (<$100) which will
systematically read and test every block, including boot sectors, on the
disk (moving each block of data temporarily to another place while write
testing). It also fixes bad formatting and has twice saved a disk for me
that ScanDisk could not.

Terry J. Reedy

I've used SpinRite before. An excellent piece of software, but it has a
few short falls:

Only works on x86.
Only works on fat or fat32 filesyatems.
Very, Very slow because of its throughness.
Has trouble with bigish (> 60GB) HDDs.

If you use Windows 98 with a fat32 filesystem on a smallish IDE based
HDD... it's a great program, otherwise it's not very useful.

 

[David M. Wilson]

(e-mail address removed) (hokiegal99) wrote...

I have Macs, Windows and Linux PCs. I'd like to have a generic HDD
burn in program to test new drives on all platforms. I know Python is
portable enough to handle the platform differences, but would it be a
good language for this type of application? The process would involve
lots of reads and writes all over the drives for an extended period of
time (several hours). Anybody else done something similar to this? Any
suggestions?

Properly burn-in testing a hard drive via lots of seeks, reads, and
writes etc. is something I wouldn't have thought Python was
particularly applicable to. The problems I can see here are:

- Python can't provide uniform low-level access to your disks. You can
use the os module, or write your own uniform disk module, which would
possibly be more effort than it was worth.

- Using the os module to create a massive file, then doing seeks,
reads, and writes around in it is not my idea of a good solution to
the problem. You are also then including the underlying operating
system's filesystem layer which will almost certainly be doing all
manner of caching and additional journalling which you can't control.

- Once you've solved the problem of uniform low level access in a
'disk' module, you also have the problem of ensuring that data
actually hits the disk. The disk has buffers, and the os has buffers.


If your targets are only *NIX/x86 and OS/X, then you are at least
confined to a unix-style environment, which makes getting low-level
disk access easy and the only problem is working out device file
names. I'm not sure how fsync() works on file descriptors open on
device files, but at a guess, I think you're going to have to try
harder to make sure the data actually hits the disk (ioctls?).

I don't think the problem is even confined to making sure writes hit
the disk. You may have problems with the operating system caching
reads too.

Are the disks IDE-based? 99.9%+ of all modern drives manufactured have
SMART capability, which has a lot of disk tests built into it,
although I don't think it has the ability to do thorough write tests.

On top of that, if you are expecting to use the burn-in tests for
benchmarking too, the results you'll get will probably not represent
the true performance of the drive in a platform-neutral way.


HTH,


David.

 

[Dennis Lee Bieber]

Irmen de Jong fed this fish to the penguins on Tuesday 30 December 2003
03:11 am:

then write random files to it until the disk is full).

Probably using multiple threads to do overlapping I/O operations, and
maybe insert some reads into the mess too?


--

 

[Neil Hodgson]

David M. Wilson:

If your targets are only *NIX/x86 and OS/X, then you are at least
confined to a unix-style environment, which makes getting low-level
disk access easy and the only problem is working out device file
names.

On Windows NT or XP the drives have names like \\.\PhysicalDrive0 so the
following Python code can do some reading from a particular position on the
first physical disk:

f = open("\\\\.\\PhysicalDrive0", "rb")
f.seek(1800*1024*1024)
for i in range(8000000):
x = f.read(1024)

Neil