"Virtual" file system mock object - replacing stuff in __builtins__

[Remy Blank]

Hi everybody,

I find myself writing lots of small utilities these days that need to
iterate over a file system tree and perform operations on every file,
e.g. setting ID3 tags or generating playlists or <plug> archiving files
(sync2cd: an incremental archiving tool to CD/DVD at
http://www.calins.ch/software/sync2cd.html) </plug>.

As a big fan of Python, that's the language of choice for these tools.
By the way, this is my first posting, and therefore also my first
opportunity to thank all the people who help providing such a great
language!

As a fan of TDD (test-driven development), I use the unittest module to
write the test cases for these tools. However, to make the test cases
independent of the file system environment, I simulate a file system
tree with a mock object using the following technique (sorry for the
long posting):

TestFiles = {
"/lost": ((16832, 1227166L, 774L, 51, 500, 600, 4096L,
1067342814, 1067336294, 1067336294), ),
"/lost/file1.mp3": ((33188, 884738L, 774L, 1, 0, 0, 1463L,
1054331223, 991209287, 1054333751), "File content"),
"/lost/file2.mp3": ((33188, 884738L, 774L, 1, 0, 0, 1463L,
1054331223, 991209287, 1054333751), "Other file content")
}

class VfsTestCase(unittest.TestCase):
"""Setup test case for 'virtual' filesystem."""
def setUp(self):
self.OldOpen = __builtins__.open
self.OldStat = os.stat
__builtins__.open = self.open
os.stat = self.stat
# Same technique for other calls

def tearDown(self):
__builtins__.open = self.OldOpen
os.stat = self.OldStat

def open(self, Path, Mode):
Path = os.path.abspath(Path)
try:
return StringIO.StringIO(TestFiles[Path][1])
except KeyError:
raise OSError, "[Errno 2] No such file or
directory: '" + Path + "'"

def stat(self, Path):
Path = os.path.abspath(Path)
try:
Data = TestFiles[Path]
Mode = Data[0][stat.ST_MODE]
RDev = 0
if stat.S_ISBLK(Mode) or stat.S_ISCHR(Mode):
RDev = Data[1]
return os.stat_result(Data[0], {"st_rdev": RDev})

except KeyError:
raise OSError, "[Errno 2] No such file or
directory: '" + Path + "'"


Now my questions:
- Am I doing something fundamentally forbidden (replacing stuff in
existing modules like __builtins__ or os)?
- Is there a better way of achieving the same result?
- If the technique is generally useful, would there be interest in
having a full-featured "virtual" file system mock object? I couldn't
find anything similar on Google.

Thanks.
-- Remy


Remove underscore and anti-spam suffix in reply address for a timely
response.

 

[Skip Montanaro]

Remy> - If the technique is generally useful, would there be interest in
Remy> having a full-featured "virtual" file system mock object? I
Remy> couldn't find anything similar on Google.

Here's a link:

http://mail.python.org/pipermail/python-list/2002-June/thread.html

Search for "virtual file system". It talks about the abstraction necessary
to eventually plug in virtual file systems, not VFS's themselves.

Skip

 

[Peter Hansen]

As a fan of TDD (test-driven development), I use the unittest module to
write the test cases for these tools. However, to make the test cases
independent of the file system environment, I simulate a file system
tree with a mock object using the following technique (sorry for the
long posting):
[snip code]
Now my questions:
- Am I doing something fundamentally forbidden (replacing stuff in
existing modules like __builtins__ or os)?
- Is there a better way of achieving the same result?
- If the technique is generally useful, would there be interest in
having a full-featured "virtual" file system mock object? I couldn't
find anything similar on Google.

It's not fundamentally forbidden to stick things in __builtins__, or
even other standard modules, but it's rarely necessary and might be
subject to a future ban if those people who periodically discuss
disallow assignments to __builtins__ have their way and forget about
those of us who need the technique for testing.

Nevertheless, if you are really doing things in a test-driven fashion,
it should not be necessary to use __builtins__, if the open() calls and
such are in your own code. In that case just create a reference to your
mock file system methods in the module under test, using the names
"open" and "stat" and so forth, and they will shadow the __builtin__
names. (Locals are found first, then globals, then builtins.)

I don't know of a better way of doing this, and in fact we've built our
own mock file system in several different projects, in a limited fashion.

The idea of a standardized or at least widely used "full-featured"
virtual file system is an excellent one! Writing tests that rely on the
real filesystem can be awkward and is probably one of the many small
things stopping/slowing some people from adopting TDD more.

-Peter

 

[Lothar Scholz]

As a fan of TDD (test-driven development), I use the unittest module to
write the test cases for these tools. However, to make the test cases
independent of the file system environment, I simulate a file system
tree with a mock object using the following technique (sorry for the
long posting):

Setup a test directory with data files and in the setUp method copy
the items from the directory to a temporary directory. Do the tests
with the temporary directory and remove it in the tearDown method.

This is the simplest case to avoid strange effects.