StringIO affecting Benchmark results

[Daniel Berger]

Hi all,

Ruby 1.8.x (any version will do)

I was working updating my benchmark suite to generate a nicely
formatted HTML table using a combination of StringIO and HTML::Table
for various Ruby method benchmarks. Unfortunately, I discovered that
using StringIO seems to alter the results of the benchmark itself. I
can repeat this consistently.

There are two things to keep in mind initially. First, the
BMSuite::StringSplitBench#start_benchmark (in the code below) finishes
before any data munging happens, so that's not the source of the
difference. Second, most of the time the version that uses StringIO
is *faster* than the non-stringio version.

My guess is that it has to do with the Benchmark module and STDOUT.
Also, does this mean then that the mere act of printing to STDOUT (or
not) is included in the benchmark itself?

Here is a sample class:

module BMSuite
class StringSplitBench
VERSION = "0.2.0"
DEF_STRING = "The quick brown fox jumped over the lazy dog's
back"

attr_accessor :max, :string, :directive
def initialize(max = 40000, string = DEF_STRING)
@max = max
@string = string
@directive = "a" * @string.length
end

def start_benchmark
if $DEBUG
puts "***************************"
puts "* STRING SPLIT BENCHMARKS *"
puts "***************************"
end
bm do |x|
x.report("each_byte:"){
@max.times{
@string.each_byte{|c| c.chr}
}
}
x.report("split:"){
@max.times{
@string.split('')
}
}
x.report("unpack:"){
@max.times{
@string.unpack(@directive)
}
}
end
end

end
end

# Without using StringIO or HTML::Table
if $0 == __FILE__
$:.unshift Dir.pwd

ssb = BMSuite::StringSplitBench.new
ssb.start_benchmark
end

# Using StringIO + HTML::Table
if $0 == __FILE__
$:.unshift Dir.pwd

require "stringio"
require "html/table"
include HTML

t1 = Table.new{ |t| t.border = 1 }

s = StringIO.new
$stdout = s

STDOUT.print "\n\nStarting benchmarks...\n\n"
ssb = BMSuite::StringSplitBench.new
ssb.start_benchmark

s.rewind

s.each_with_index{ |line,n|
STDOUT.puts line
line.gsub!(/\(|\)/,"")
line.strip!
t1 << Table::Row.new{ |r|
if n == 0
# Bump column headers over 1
content = [""] + line.split + ["Iterations"]
content.map!{ |e| e = "<font color='green'>" +
e.capitalize }
r.content = content
else
r.content = line.split + [ssb.max]
end
}
}

File.open("stringsplit_results.html","w+"){ |fh| fh.puts t1.html }
s.close
end

Regards,

Dan