R
Ruby Quiz
Does this library have any practical value? Probably not. It's been suggested
in the Perl community that hacks like this are a good minor deterrent to those
trying to read source code you would rather keep hidden, but it must be stressed
that this is no form of serious security. Regardless, it's a fun little toy to
play with.
It was mentioned in the discussion that Perl, where ACME::Bleach comes from,
includes a framework for source filtering. It can be used to make modules that
modify source code much as we are doing in this quiz. Perl's Switch.pm is a
good example of this, but ironically ACME::Bleach is not.
That naturally leads to the question, can you build source filters in Ruby?
Clearly we can build ACME::Bleach, but not all source filters are as simple I'm
afraid. Consider this:
#!/usr/local/bin/ruby -w
require "fix_my_broken_syntax"
invalid++
Now the thought here is that fix_my_broken_syntax.rb will read my source, change
it so that it does something valid, eval() it, and exit() before the invalid
code is an issue. Here's a trivial example of fix_my_broken_syntax.rb:
#!/usr/local/bin/ruby -w
puts "Fixed!"
exit
Does that work? Unfortunately, no:
$ ruby invalid.rb
invalid.rb:5: syntax error
invalid++
^
Ruby never gets to loading the library, because it's not happy with the syntax
of the first file. That makes writing a source filter for anything that isn't
valid Ruby syntax complicated and if it is valid Ruby syntax, you can probably
just code it up in Ruby to begin with.
Except for whiteout.rb, our version of ACME::Bleach.
You can't build Ruby constructs out of whitespace alone, so some form of source
filtering is required. Luckily, we can get away with the approach described
above for this source filter, because a bunch of whitespace (with no code) is
valid Ruby syntax. It just doesn't do anything. Ruby will skip right over our
whitespace and load the library that restores and runs the code.
Most people took this approach. Let's examine one such example by Robin
Stocker:
#!/usr/bin/ruby
#
# This is my solution for Ruby Quiz #34, Whiteout.
# Author:: Robin Stocker
#
#
# The Whiteout module includes all functionality like:
# - whiten
# - run
# - encode
# - decode
#
module Whiteout
@@bit_to_code = { '0' => " ", '1' => "\t" }
@@code_to_bit = @@bit_to_code.invert
@@chars_to_ignore = [ "\n", "\r" ]
#
# Whitens the content of a file specified by _filename_.
# It leaves the shebang intact, if there is one.
# At the beginning of the file it inserts the require 'whiteout'.
# See #encode for details about how the whitening works.
#
def Whiteout.whiten( filename )
code = ''
File.open( filename, 'r' ) do |file|
file.each_line do |line|
if code.empty?
# Add shebang if there is one.
code << line if line =~ /#!\s*.+/
code << "#{$/}require 'whiteout'#{$/}"
else
code << encode( line )
end
end
end
File.open( filename, 'w' ) do |file|
file.write( code )
end
end
# ...
First, we can see that the module defines some module variables, which are
really used as constants here. Their contents hint at the encoding algorithm
we'll see later.
Then we have a method for managing the transformation of the source into
whitespace. It starts by opening the passed file and reading the code
line-by-line. If the first line is a shebang line, it's saved in the variable
code. Next, a "require 'whiteout'" line is added to code. Finally, all other
lines from the file are appended to code after being passed through an encode()
method we'll examine shortly. With the contents read and transformed, the
method then reopens the source for writing and dumps the modifications into it.
The next method is the reverse process:
# ...
#
# Reads the file _filename_, decodes and runs it through eval.
#
def Whiteout.run( filename )
text = ''
File.open( filename, 'r' ) do |file|
decode = false
file.each_line do |line|
if not decode
# We don't want to decode the "require 'whiteout'",
# so start decoding not before we passed it.
decode = true if line =~ /require 'whiteout'/
else
text << decode( line )
end
end
end
# Run the code!
eval text
end
# ...
This method again reads the passed file. It skips over the "require 'whiteout'"
line, then copies the rest of the file into the variable text, after passing it
through decode() line-by-line. The final line of the method calls eval() on
text, which should now contain the restored program.
On to encode() and decode():
#
# Encodes text to "whitecode". It works like this:
# - Chars in @@char_to_ignore are ignored
# - Each byte is converted to its bit representation,
# so that we have something like 01100001
# - Then, it is converted to whitespace according to @@bit_to_code
# - 0 results in a " " (space)
# - 1 results in a "\t" (tab)
#
def Whiteout.encode( text )
white = ''
text.scan(/./m) do |char|
if @@chars_to_ignore.include?( char )
white << char
else
char.unpack('B8').first.scan(/./) do |bit|
code = @@bit_to_code[bit]
white << code
end
end
end
return white
end
#
# Does the inverse of #encode, it takes "white"
# and returns the decoded text.
#
def Whiteout.decode( white )
text = ''
char = ''
white.scan(/./m) do |code|
if @@chars_to_ignore.include?( code )
text << code
else
char << @@code_to_bit
in the Perl community that hacks like this are a good minor deterrent to those
trying to read source code you would rather keep hidden, but it must be stressed
that this is no form of serious security. Regardless, it's a fun little toy to
play with.
It was mentioned in the discussion that Perl, where ACME::Bleach comes from,
includes a framework for source filtering. It can be used to make modules that
modify source code much as we are doing in this quiz. Perl's Switch.pm is a
good example of this, but ironically ACME::Bleach is not.
That naturally leads to the question, can you build source filters in Ruby?
Clearly we can build ACME::Bleach, but not all source filters are as simple I'm
afraid. Consider this:
#!/usr/local/bin/ruby -w
require "fix_my_broken_syntax"
invalid++
Now the thought here is that fix_my_broken_syntax.rb will read my source, change
it so that it does something valid, eval() it, and exit() before the invalid
code is an issue. Here's a trivial example of fix_my_broken_syntax.rb:
#!/usr/local/bin/ruby -w
puts "Fixed!"
exit
Does that work? Unfortunately, no:
$ ruby invalid.rb
invalid.rb:5: syntax error
invalid++
^
Ruby never gets to loading the library, because it's not happy with the syntax
of the first file. That makes writing a source filter for anything that isn't
valid Ruby syntax complicated and if it is valid Ruby syntax, you can probably
just code it up in Ruby to begin with.
Except for whiteout.rb, our version of ACME::Bleach.
You can't build Ruby constructs out of whitespace alone, so some form of source
filtering is required. Luckily, we can get away with the approach described
above for this source filter, because a bunch of whitespace (with no code) is
valid Ruby syntax. It just doesn't do anything. Ruby will skip right over our
whitespace and load the library that restores and runs the code.
Most people took this approach. Let's examine one such example by Robin
Stocker:
#!/usr/bin/ruby
#
# This is my solution for Ruby Quiz #34, Whiteout.
# Author:: Robin Stocker
#
#
# The Whiteout module includes all functionality like:
# - whiten
# - run
# - encode
# - decode
#
module Whiteout
@@bit_to_code = { '0' => " ", '1' => "\t" }
@@code_to_bit = @@bit_to_code.invert
@@chars_to_ignore = [ "\n", "\r" ]
#
# Whitens the content of a file specified by _filename_.
# It leaves the shebang intact, if there is one.
# At the beginning of the file it inserts the require 'whiteout'.
# See #encode for details about how the whitening works.
#
def Whiteout.whiten( filename )
code = ''
File.open( filename, 'r' ) do |file|
file.each_line do |line|
if code.empty?
# Add shebang if there is one.
code << line if line =~ /#!\s*.+/
code << "#{$/}require 'whiteout'#{$/}"
else
code << encode( line )
end
end
end
File.open( filename, 'w' ) do |file|
file.write( code )
end
end
# ...
First, we can see that the module defines some module variables, which are
really used as constants here. Their contents hint at the encoding algorithm
we'll see later.
Then we have a method for managing the transformation of the source into
whitespace. It starts by opening the passed file and reading the code
line-by-line. If the first line is a shebang line, it's saved in the variable
code. Next, a "require 'whiteout'" line is added to code. Finally, all other
lines from the file are appended to code after being passed through an encode()
method we'll examine shortly. With the contents read and transformed, the
method then reopens the source for writing and dumps the modifications into it.
The next method is the reverse process:
# ...
#
# Reads the file _filename_, decodes and runs it through eval.
#
def Whiteout.run( filename )
text = ''
File.open( filename, 'r' ) do |file|
decode = false
file.each_line do |line|
if not decode
# We don't want to decode the "require 'whiteout'",
# so start decoding not before we passed it.
decode = true if line =~ /require 'whiteout'/
else
text << decode( line )
end
end
end
# Run the code!
eval text
end
# ...
This method again reads the passed file. It skips over the "require 'whiteout'"
line, then copies the rest of the file into the variable text, after passing it
through decode() line-by-line. The final line of the method calls eval() on
text, which should now contain the restored program.
On to encode() and decode():
#
# Encodes text to "whitecode". It works like this:
# - Chars in @@char_to_ignore are ignored
# - Each byte is converted to its bit representation,
# so that we have something like 01100001
# - Then, it is converted to whitespace according to @@bit_to_code
# - 0 results in a " " (space)
# - 1 results in a "\t" (tab)
#
def Whiteout.encode( text )
white = ''
text.scan(/./m) do |char|
if @@chars_to_ignore.include?( char )
white << char
else
char.unpack('B8').first.scan(/./) do |bit|
code = @@bit_to_code[bit]
white << code
end
end
end
return white
end
#
# Does the inverse of #encode, it takes "white"
# and returns the decoded text.
#
def Whiteout.decode( white )
text = ''
char = ''
white.scan(/./m) do |code|
if @@chars_to_ignore.include?( code )
text << code
else
char << @@code_to_bit
Code:
if char.length == 8
text << [char].pack("B8")
char = ''
end
end
end
return text
end
end
# ...
The comments in there detail the exact process we're looking at here, so I'm not
going to repeat them.
Note that @@char_to_ignore contains "\n" and "\r" so they are not translated.
The effect of that is that line-endings are untouched by this conversion. Some
solutions used such characters in their encoding algorithm. The gotcha there is
that any line-ending translation done to the modified source (say FTP through
ASCII mode) will break the hidden code. Robin's solution doesn't have that
problem.
Here's the code that ties all those methods into a solution:
# ...
#
# And here's the logic part of whiteout.
# If it was run directly, whites out the files in ARGV.
# And if it was required, decodes the whitecode and runs it.
#
if __FILE__ == $0
ARGV.each do |filename|
Whiteout.whiten( filename )
end
else
Whiteout.run( $0 )
end
Again, the comment saves me some explaining.
That was Robin's first solution to a Ruby Quiz, but I never would have known
that from looking at the code. Thanks for sharing Robin!
Obviously, a conversion of this type grossly inflates the size of the source.
Around eight times the size, to be exact. A couple of solutions used zlib to
control the expansion, which I thought was clever. By compressing the source
and then encoding() (and using a base three conversion) Dominik Bathom got
results around three times the inflation instead.
Ara.T.Howard took a different approach, using whiteout.rb as a database to store
the trimmed files. That was a very interesting process, demonstrated well in
the submission email. The advantages to this approach would be no inflation
penalty and the code stays readable (just not in the original location). The
disadvantage I see is that it requires the exact same library to be present both
at encoding and decoding, which probably makes sharing the altered code
impractical.
As always, my thanks to all who gave this little diversion an attempt. I'm sure
we'll see tons of whitespace only code on RubyForge in the future, thanks to our
efforts.
Tomorrow begins part one of our first two-part Ruby Quiz. Stay tuned...