[ANN] FileSystem 0.1.0: Beta for me, Alpha for you

[Francis Hwang]

Greetings!

Next down my non-stop pipeline of absolutely necessary libraries that
only I seem to need: FileSystem, a library that will mock out File,
FileUtils, Dir, and other file-dependent built-in classes. It aims to
simulate an entire file-system in memory for the purposes of testing.

http://rubyforge.org/projects/filesystem/

== HOW DOES IT WORK? ==

To use it in live code, you call get methods on the FileSystem module:

FileSystem.get_dir => Dir
FileSystem.get_file => File
FileSystem.get_file_utils => FileUtils
FileSystem.get_dir.entries( '.' ) => [ '.', '..', 'file1', ... ]

(I'm thinking about adding a voodoo.rb file that actually re-assigns
the constants Dir, File, and FileUtils, though I suspect this will not
be that usable in many cases.)

Then, to mock out that activity for a test code, simply call
FileSystem.mock= :

FileSystem.mock = true
FileSystem.get_dir => FileSystem:irAdapter
FileSystem.get_file => FileSystem::FileAdapter
FileSystem.get_file_utils => FileSystem::FileUtilsAdapter
FileSystem.get_dir.entries( '.' ) => [ '.', '..', ... ]

== HOW COMPLETE IS IT? ==
Oh dear, not at all. Hence the release name: "Beta for me, Alpha for
you". I am now using it day-to-day, both at Rhizome[1] and my own
blog-publishing software, Dauxite[2]. So I know it works for me. It
currently handles basic issues of looking up paths, writing and reading
files, modification times, etc. It doesn't know anything about
symlinks, creation times, permissions. I suspect that if you tried to
mock a Windows system with it, that it would squeal like a 2-year-old.

I'm putting out today's release in hopes of getting bug reports. Please
file bug reports! Lots and lots of bug reports! Please be specific,
though. If you file a bug titled "Support Windows, you l4m3r" I won't
know where to start.

Thanks,
Francis "I'd mock out the air itself if my lungs were hot-swappable"
Hwang
http://fhwang.net/

[1] Rhizome is the premier online resource in the field of new media
arts: It gets more than 1 million page views a month and contains the
world's largest online archive of new media art, at almost 1500 works.
[2] Dauxite is my own idiosyncratic, unreleased software. So it's not
as heavily tested as Rhizome, but I use it at least once a week, so
it's a good way to test something like FileSystem.

 

[gabriele renzi]

Francis Hwang ha scritto:

Greetings!

Next down my non-stop pipeline of absolutely necessary libraries that
only I seem to need: FileSystem, a library that will mock out File,
FileUtils, Dir, and other file-dependent built-in classes. It aims to
simulate an entire file-system in memory for the purposes of testing.

http://rubyforge.org/projects/filesystem/

first: this seem incredibly cool.
second: would'nt something like "FS::Mock" or "test/filesystem" be a
better name? when I first saw this on rubyforge I thought it was some
kind of VFS

 

[Francis Hwang]

Francis Hwang ha scritto:

first: this seem incredibly cool.

Thanks! It's already been pretty helpful for me, hopefully it'll be
useful to others too.

second: would'nt something like "FS::Mock" or "test/filesystem" be a
better name? when I first saw this on rubyforge I thought it was some
kind of VFS

I think it's related to how you're supposed to access the lib in your
live code. The name "FileSystem" to me seems less distracting in
production code than something like "FS::Mock" ... It's only being
mocked out in your tests.

I'm not 100% happy with the current form of access either: Typing
"FileSystem.get_file" when you really just want "File" is sort of
cumbersome. But it just seemed like the least broken thing I could go
forward with while I figured out something else. When we talked about
this at Ruby-NYC a few weeks ago, Patrick was saying I could write a
file to change the definitions of File, FileUtils, Dir, etc. Maybe
that'll be the way to go.

Anyway, I'm very very open to suggestions regarding the name and/or how
to access it.

Francis Hwang
http://fhwang.net/

 

[Daniel Berger]

Greetings!

Next down my non-stop pipeline of absolutely necessary libraries that

only I seem to need: FileSystem, a library that will mock out File,
FileUtils, Dir, and other file-dependent built-in classes. It aims to

simulate an entire file-system in memory for the purposes of testing.

Oh, dear. I believe we have our first significant name collision -
your package and Mike Hall's filesystem package, and they do different
things.

Caveat Programmor.

Regards,

Dan

 

[Anders Bengtsson]

Next down my non-stop pipeline of absolutely necessary libraries that
only I seem to need: FileSystem, a library that will mock out File,
FileUtils, Dir, and other file-dependent built-in classes. It aims to
simulate an entire file-system in memory for the purposes of testing.

Very cool! I was looking for something similar to this some time ago,
but figured it would be too much work to implement it. I'm glad you made
a different decision!

Did you examine the possibility to override the methods in the standard
IO and File classes, so that it could be more transparent?

/Anders

--

 

[Ben Giddings]

To use it in live code, you call get methods on the FileSystem module:

FileSystem.get_dir => Dir
FileSystem.get_file => File
FileSystem.get_file_utils => FileUtils
FileSystem.get_dir.entries( '.' ) => [ '.', '..', 'file1', ... ]

(I'm thinking about adding a voodoo.rb file that actually re-assigns
the constants Dir, File, and FileUtils, though I suspect this will not
be that usable in many cases.)

Then, to mock out that activity for a test code, simply call
FileSystem.mock= :

FileSystem.mock = true
FileSystem.get_dir => FileSystem:irAdapter
FileSystem.get_file => FileSystem::FileAdapter
FileSystem.get_file_utils => FileSystem::FileUtilsAdapter
FileSystem.get_dir.entries( '.' ) => [ '.', '..', ... ]

Hi Francis,

I like the idea, and the idea of mocking out things like filesystems is
very important. Just a few comments on the naming of things:

If the class is mainly just a test construct and/or a proxy for actual
Filesystem calls, maybe "Test" or "Proxy" should be part of the class
name. Secondly, method names that start with "get_" seem pretty
un-rubyish. They feel more like Java getFoo() and setFoo() type
functions. Maybe it would make sense to rename them just dir, file,
file_utils and dir_entries?

Ben

P.S. If you ever decide to mock out a human being, let me know. This
morning in the shower I was wondering how I can mock out the person who
has to push a button on an embedded device I'm working on. Sure, I can
easily mock out the register that pushing the button supposedly
changes, but that kinda defeats the purpose of the test.

 

[Ilmari Heikkinen]

P.S. If you ever decide to mock out a human being, let me know. This
morning in the shower I was wondering how I can mock out the person
who has to push a button on an embedded device I'm working on. Sure,
I can easily mock out the register that pushing the button supposedly
changes, but that kinda defeats the purpose of the test.

Robots, man, robots. Preferably small dancing ones.

 

[Daniel Berger]

P.S. If you ever decide to mock out a human being, let me know.

You smell and your momma dresses you funny. How's that?

Oh...mock "out".

Regards,

Dan

 

[Francis Hwang]

Oh, dear. I believe we have our first significant name collision -
your package and Mike Hall's filesystem package, and they do different
things.

Aw, for Pete's sake. I guess that's what I guess for not checking RAA
...

So I guess I better rename my FileSystem to something else. Suggestions?

Francis Hwang
http://fhwang.net/

 

[Francis Hwang]

Very cool! I was looking for something similar to this some time ago,
but figured it would be too much work to implement it. I'm glad you
made
a different decision!

I've actually done this, in half-hearted ways, three separate times in
three separate apps ... So this lib is my attempt to do it once, and
correct ...

Did you examine the possibility to override the methods in the standard
IO and File classes, so that it could be more transparent?

More transparency is important, yeah, but maybe I won't want to
redefine methods on already existing classes & modules. Method coverage
is currently really spotty, so if I redefined, say, File.mtime but not
File.ctime, your test code would be mixing up its calls against the
MockFileSystem with its calls to the real file system ... pretty
confusing, I think.

Might be better to simply undefine and redefine those constants:

Object.send( :remove_const, :File )
class File
...
end

Though it's quite possible that that could be really nasty, too. I'll
have to poke around for a while to see.

Francis Hwang
http://fhwang.net/

 

[Francis Hwang]

If the class is mainly just a test construct and/or a proxy for actual
Filesystem calls, maybe "Test" or "Proxy" should be part of the class
name. Secondly, method names that start with "get_" seem pretty
un-rubyish. They feel more like Java getFoo() and setFoo() type
functions. Maybe it would make sense to rename them just dir, file,
file_utils and dir_entries?

Yeah, good points. This may all be mooted if I decide to just replace
Dir, File, FileUtils, etc with some strange magicks. We'll see.

P.S. If you ever decide to mock out a human being, let me know. This
morning in the shower I was wondering how I can mock out the person
who has to push a button on an embedded device I'm working on. Sure,
I can easily mock out the register that pushing the button supposedly
changes, but that kinda defeats the purpose of the test.

Well, this won't directly solve your problem, but for an idea of how
you might solve this, you could look at EasyPrompt, which does the same
for a command-line user: http://easyprompt.rubyforge.org/ In
particular, check out the MockCommandLineUser.

Funny thing is, code that uses EasyPrompt (and is thus more mockable)
looks a lot like the example code people use when talking about
continuations in web apps ... leading me to believe that
continuation-based web apps would be tons easier to test across
multi-step processes. Not that I've ever used continuations, just
saying.

Francis Hwang
http://fhwang.net/

 

[Alexander Kellett]

(sorry for breaking threading)

does it already, or would it be possible, for this to act as an overlay
file system?
as in, in a test, i can create a number of virtual files, any lookups
on real files
succeed - thusly meaning that e.g loading of new libraries doesn't
screw up - but
any access to the virtual directories / files would return pseudo
objects.

whats the feasibility of this? it would vastly simplify many testcases
i imagine.

Alex

 

[Pit Capitain]

I'm not 100% happy with the current form of access either: Typing
"FileSystem.get_file" when you really just want "File" is sort of
cumbersome. But it just seemed like the least broken thing I could go
forward with while I figured out something else. When we talked about
this at Ruby-NYC a few weeks ago, Patrick was saying I could write a
file to change the definitions of File, FileUtils, Dir, etc. Maybe
that'll be the way to go.

It would be nice if you could implement it in this way. Then the code under test
would be completely independent from your library. I once used this technique
for a Time mock [1] and I liked it very much.

Regards,
Pit

[1] In my tests I had a lot of calls to Kernel#sleep, but I wanted them to run
as fast as possible. So I implemented a Time mock where you could change the
current time. In the tests, a call to sleep simply adjusted the current time
accordingly.

 

[Alexander Kellett]

MockFS / MockFiles / FileMocker / MocktasticFileCreationWizardry??

(personally i like MockFiles)

 

[Daniel Berger]

MockFS / MockFiles / FileMocker / MocktasticFileCreationWizardry??

(personally i like MockFiles)

I like MockFS.

Dan

 

[gabriele renzi]

Francis Hwang ha scritto:

I think it's related to how you're supposed to access the lib in your
live code. The name "FileSystem" to me seems less distracting in
production code than something like "FS::Mock" ... It's only being
mocked out in your tests.

mh.. but I have the feeling that it /should/ be distracting.
Or, better put, that it should make clear that we're working with a test
facility. Anyway I understand your reason and you're the author

I'm not 100% happy with the current form of access either: Typing
"FileSystem.get_file" when you really just want "File" is sort of
cumbersome. But it just seemed like the least broken thing I could go
forward with while I figured out something else. When we talked about
this at Ruby-NYC a few weeks ago, Patrick was saying I could write a
file to change the definitions of File, FileUtils, Dir, etc. Maybe
that'll be the way to go.

Anyway, I'm very very open to suggestions regarding the name and/or how
to access it.

just a suggestion, maybe you can take a look at how libvfs (on raa) does
this.

 

[Shalev NessAiver]

I'll second that.

-Shalev

I like MockFS.

Dan

 

[Francis Hwang]

does it already, or would it be possible, for this to act as an
overlay file system?
as in, in a test, i can create a number of virtual files, any lookups
on real files
succeed - thusly meaning that e.g loading of new libraries doesn't
screw up - but
any access to the virtual directories / files would return pseudo
objects.

whats the feasibility of this? it would vastly simplify many testcases
i imagine.

Maybe this is what you're talking about:

At Rhizome, we send out lots of customized emails, with templates. So
maybe you'll have a template file in
/var/www/includes/email/welcome.tmpl . But if you want to play around
with the codebase on, say, another server, this is annoying, because
then you'd have to build up that directory path just to put one file in
it.

So instead, I save a representative copy of that template somewhere in
the test code directory, say test/test_data/email. Then, if we've got
an email class that uses that template:

class WelcomeEmail < RhizMail::SimpleTemplateMessage
def initialize( user )
template = FileSystem.get_file.open(
'/var/www/includes/email/welcome.tmpl' ) { |f| f.gets nil }
super( template )
end
end

the setup for the test code might look like this:

class TestWelcomeEmail < Test::Unit::TestCase
def setup
FileSystem.mock = true
contents = File.open( 'test/test_data/email/welcome.tmpl' ) { |f|
f.gets nil
}
FileSystem.mock_file_system.fill_path( '/var/www/includes/email' )
FileSystem.get_file.open( '/var/www/includes/email/welcome.tmpl',
File::CREAT | File::WRONLY ) { |f|
f << contents
}
end
end

So that when the WelcomeEmail#initialize reads the contents of
/var/www/includes/email/welcome.tmpl, it's actually reading contents
from the virtual file system that was set up in your test case.

Is that what you had in mind?

Francis Hwang
http://fhwang.net/

 

[Francis Hwang]

mh.. but I have the feeling that it /should/ be distracting.
Or, better put, that it should make clear that we're working with a
test facility. Anyway I understand your reason and you're the author

Well, it's interesting, I think some of us would like something like
this to be really distracting, and others would like it to be
completely transparent. The two are pretty opposite in my mind, but I
suppose there's no reason I couldn't support both, with an option to
use one or the other depending on what files get included. Maybe by
default we'd use, say MockFS.file and MockFS.file_utils to get the
classes, but then if you wanted you could have your code simply use
File and FileUtils, your test code could require 'mockfs/voodoo' or
some such file to redefine those constants. I'll have to look into the
details.

just a suggestion, maybe you can take a look at how libvfs (on raa)
does this.

Thanks for the tip; I'll definitely take a look.

Francis Hwang
http://fhwang.net/

 

[Florian Gross]

[1] In my tests I had a lot of calls to Kernel#sleep, but I wanted them
to run as fast as possible. So I implemented a Time mock where you could
change the current time. In the tests, a call to sleep simply adjusted
the current time accordingly.

Heh, I implemented that on the real Time class once as well as setting
methods for setting the Time delta. Fun stuff!