Testing - Network Layer Fault Injection in Ruby

[James Tucker]

Hi all,

I've yet to find something that fills this hole in my test cases.

Does anyone do this, and if so, what tools do they use?

Thanks,

James.

 

[Eric Hodel]

I've yet to find something that fills this hole in my test cases.
Does anyone do this, and if so, what tools do they use?

I do some of this with custom stubs in memcache-client, see below.
I've also simulated call failures (Errno::whatever) with slightly
more powerful stubs that can take a block to raise the exception.

Here's a fake socket:

class FakeSocket

attr_reader :written, :data

def initialize
@written = StringIO.new
@data = StringIO.new
end

def write(data)
@written.write data
end

def gets
@data.gets
end

def read(arg)
@data.read arg
end

end

And here's a test using it:

def test_flush_all_failure
socket = FakeSocket.new
socket.data.write "ERROR\r\n"
socket.data.rewind
server = FakeServer.new socket
def server.host() "localhost"; end
def server.port() 11211; end

@cache.servers = []
@cache.servers << server

assert_raise MemCache::MemCacheError do
@cache.flush_all
end

assert_equal "flush_all\r\n", socket.written.string
end

 

[James Tucker]

Eric,

Thanks, that's somewhat the route I had started down for some of the units. I probably shouldn't have omitted from my orignal question, the fact that I'm also using OpenSSL.

Some of the specific issues I have been dealing with are caused by a harsh environment (trans-atlantic links and poor ADSL connections), and in some cases this causes irrecoverable blocking. I think I've managed to cover all of the blocking cases, but local tests are hard to implement, as you have to fake things like a full TCP connection timeout. (Which requires not sending and timing out packet data which normally the OS doesn't want you to do).

It's also difficult to find out where the blocks are, as the state renders the app essentially useless, and happens in a different environment than that of the testing / development stage.

As some of this connection state stuff is actually outside of pure ruby, I was somewhat hoping for an external solution, such as a fault injection proxy socket or similar. Do you know of any?

Regards,

James.

I've yet to find something that fills this hole in my test cases.
Does anyone do this, and if so, what tools do they use?

I do some of this with custom stubs in memcache-client, see below. I've
also simulated call failures (Errno::whatever) with slightly more
powerful stubs that can take a block to raise the exception.

Here's a fake socket:

class FakeSocket

attr_reader :written, :data

def initialize
@written = StringIO.new
@data = StringIO.new
end

def write(data)
@written.write data
end

def gets
@data.gets
end

def read(arg)
@data.read arg
end

end

And here's a test using it:

def test_flush_all_failure
socket = FakeSocket.new
socket.data.write "ERROR\r\n"
socket.data.rewind
server = FakeServer.new socket
def server.host() "localhost"; end
def server.port() 11211; end

@cache.servers = []
@cache.servers << server

assert_raise MemCache::MemCacheError do
@cache.flush_all
end

assert_equal "flush_all\r\n", socket.written.string
end

 

[Eric Hodel]

Thanks, that's somewhat the route I had started down for some of
the units. I probably shouldn't have omitted from my orignal
question, the fact that I'm also using OpenSSL.

Some of the specific issues I have been dealing with are caused by
a harsh environment (trans-atlantic links and poor ADSL
connections), and in some cases this causes irrecoverable blocking.
I think I've managed to cover all of the blocking cases, but local
tests are hard to implement, as you have to fake things like a full
TCP connection timeout. (Which requires not sending and timing out
packet data which normally the OS doesn't want you to do).

It's also difficult to find out where the blocks are, as the state
renders the app essentially useless, and happens in a different
environment than that of the testing / development stage.

As some of this connection state stuff is actually outside of pure
ruby, I was somewhat hoping for an external solution, such as a
fault injection proxy socket or similar. Do you know of any?

I don't I've not needed that level of fault tolerance.