I am new to Ruby and I could use some expert advice as to how I can make this code run faster.

[Ruby Maniac]

I am new to Ruby and I could use some expert advice as to how I can
make this code run faster.

def scramble(fname)
f = File.new(fname, "rb")
_fname = fname + ".scrambled"
begin
File.exist?(_fname) if File.delete(_fname)
rescue
end
ff = File.new(_fname, "wb+")
for l in f
l.each_byte{|c| ff.write((c | 0x80).chr) }
end
f.close()
ff.close()
end

 

[Brian Adkins]

I am new to Ruby and I could use some expert advice as to how I can
make this code run faster.

Write a C extension. Seriously, Ruby is pretty darn slow at iterating
over each byte of a file. The I/O is reasonable, but the byte
iteration is terrible. You might get a small speedup by using
File#read instead of File#each_byte:

while (buffer = f.read(BUF_SIZE))
buffer.each_byte {|b| ... }
end

I got about a 5% speedup using an 8K buffer.

You also may want to consider the idiom:

File.open(filename) do |f|
...
end

This will automatically close the file at the end of the block.

def scramble(fname)
f = File.new(fname, "rb")
_fname = fname + ".scrambled"
begin
File.exist?(_fname) if File.delete(_fname)

You may want to double check the previous line.

rescue
end
ff = File.new(_fname, "wb+")
for l in f
l.each_byte{|c| ff.write((c | 0x80).chr) }

So you don't need to get the original file data back, right? If you
do, you may want to use xor instead, and if you don't, then why not
just write zeros?

 

[John Joyce]

Write a C extension. Seriously, Ruby is pretty darn slow at iterating
over each byte of a file. The I/O is reasonable, but the byte
iteration is terrible. You might get a small speedup by using
File#read instead of File#each_byte:

while (buffer = f.read(BUF_SIZE))
buffer.each_byte {|b| ... }
end

I got about a 5% speedup using an 8K buffer.

You also may want to consider the idiom:

File.open(filename) do |f|
...
end

This will automatically close the file at the end of the block.


You may want to double check the previous line.


So you don't need to get the original file data back, right? If you
do, you may want to use xor instead, and if you don't, then why not
just write zeros?

Benchmark it so you can find the bottlenecks.

 

[Ruby Maniac]

Benchmark it so you can find the bottlenecks.- Hide quoted text -

- Show quoted text -

Here is the fastest Ruby code I have been able to develop so far.

Note the use of the Hash caused the Ruby code to run 14% slower than
when the Hash was not used.

Also it is worth noting that I was able to whip-up some Python code
that runs this same problem in 2.278 secs using Python 2.5.1 and this
makes Python almost 30x faster than Ruby for this problem.

$h = Hash.new
(0..127).select {|x| $h[x.chr] = ((x & 0x7f) | 0x80).chr}

def scramble(fname)
f = File.new(fname, "rb")
_fname = fname + ".scrambled"
begin
File.exist?(_fname) if File.delete(_fname)
rescue
end
ff = File.new(_fname, "wb+")
for l in f
l.each_byte{|c| ff.write((c | 0x80).chr) }
end
f.close()
ff.close()
end

def _scramble(fname)
f = File.new(fname, "rb")
_fname = fname + ".scrambled"
begin
File.exist?(_fname) if File.delete(_fname)
rescue
end
ff = File.new(_fname, "wb+")
for l in f
l.each_byte{|c| ff.write($h[c]) }
end
f.close()
ff.close()
end

The slowest section of code is iteration of each_byte where Ruby
spends 50% of its time. Python spends 67% of its time translating
characters.

Also the Python community was very quick to help me out given the same
amount of time with this problem.

After this exorcise I can easily see why Ruby is relegated to the
kinds of problems people are willing to use Ruby for versus the far
more intense set of problem domains in which Python can be more easily
used.

Don't take me wrong, I neither love nor hate any languages I encounter
- I simply perform benchmarks and choose the fastest of the breed and
then I tend to use whichever languages or tools perform better. I find
it easier to meet customer expectations when I begin with a set of
tools that already runs reasonably fast. For some this means Ruby
runs reasonably fast however for me this means Ruby comes in dead-last
and therefore may not be worth using versus other languages or tools
that benchmark faster than Ruby does.

On the other hand, I am also waiting for Ruby 1.9.0 and YARV to see if
Ruby can be made to perform better than Ruby 1.8.6. Having said this
Ruby 1.9.0 has been in the works for about 18 months by now with no
real date other than Nov 2007 but Ruby 1.9.0 was supposed to be
release sometime during 2006 also. Sadly for me, or happily depending
on where you stand on this issue - even if Ruby were made to run
faster than Python but it lacked a byte-code compiler so I don't have
to ship source code I would still spend my time coding Python along
with the D Language just so I don't have to ship source code.

Ruby is ideal for those who don't care about performance or for Open
Source Projects where source code will be shipped - all other uses
Python will be faster.

 

[Todd Benson]

Don't take me wrong, I neither love nor hate any languages I encounter
- I simply perform benchmarks and choose the fastest of the breed and
then I tend to use whichever languages or tools perform better.

Yes, if you get a chance, please compare with other languages and
their communities, too!

Obviously, everyone has some aspect of any language that excites them
or fits their project. I don't speak for rubyists when I say this,
but, not being a CS guy, and instead an engineer, I sort of prefer
elegance combined with efficiency, and not pure speed.

I think Ruby will eventually surpass Python in
usability/speed/libraries/put-whatever-you-want-here-to-love-your-language
It's just going to take a while, that's all. Who knows really,
though? Maybe it will just fall to the wayside. Or, maybe it will
end up just filling a certain niche of a greater whole. No language
is perfect for everything, after all!

Todd

 

[7stud --]

I think Ruby will eventually surpass Python in
usability/speed/libraries/put-whatever-you-want-here-to-love-your-language
It's just going to take a while, that's all. Who knows really,
though? Maybe it will just fall to the wayside. Or, maybe it will
end up just filling a certain niche of a greater whole. No language
is perfect for everything, after all!

Todd

I too am having a hard time understanding why anyone would use Ruby.
What is Ruby perfect for?

Also, why do you think Ruby will eventually surpass Python. Do you
think Python has halted all development and is standing still waiting
for Ruby to catch up? What makes you think that Python isn't putting
more distance between itself and Ruby as we speak? After all, Python
3.0 alpha is already out.

 

[Todd Benson]

I too am having a hard time understanding why anyone would use Ruby.
What is Ruby perfect for?

Also, why do you think Ruby will eventually surpass Python. Do you
think Python has halted all development and is standing still waiting
for Ruby to catch up? What makes you think that Python isn't putting
more distance between itself and Ruby as we speak? After all, Python
3.0 alpha is already out.

Hmm. Good point. I'm not the type of person that picks the right
tool for the job. I like making a square peg fit a round hole most of
the time, so take me with a grain of salt. (BTW, don't take that to
mean that Ruby is the square peg). I'm still researching a small
handful of languages, and after I get further into it, I'll be able to
ascertain, possibly, whether Ruby has a chance of surpassing Python in
features.

I guess, though, I kind of don't want it to, simply, because then, it
will lose its charm, and then, it may lose its powerful "fun".

Python just doesn't seem "fun" to me, but I'm no expert on the
language (I can read most of it, though). So, after I write a couple
of programs/applications in Python, I'll get back to you.

It's funny though that I have been translating a couple apps/libraries
written in Python to Ruby, and I can't quite figure out why I just
don't feel like doing it the other way around.

Whatever floats your boat, I guess.

Todd

 

[Phrogz]

I too am having a hard time understanding why anyone would use Ruby.

I personally love the syntax. It gives me great power with little
typing, in a format that is easy for me to understand when I read my
program a month later.

What is Ruby perfect for?

"Perfect" might be an overstatement. I find it fantastic for quickly
generating one-off programs and automating tasks. I use it for complex
or repeated text manipulation, such as performing the same 10 regex
replacements on 20 different files. I use it for code generation
(creating C++ headers and code based on XML files). I use it for
pulling large text information from a database, transforming it, and
pushing it back up. (Automated updating of some pages, bypassing the
web interface, if you must know.) I use it for generating help and
example HTML files based on XML and other sources.

Also, why do you think Ruby will eventually surpass Python.

I certainly have no such thoughts, one way or the other. I *hope* it
will, because a larger library of contributors and libraries is always
nice. But mostly I don't care about Python. I tried it, I have to
program in it occasionally at work, and I personally don't like it
nearly as much as Ruby.


Programming languages are a personal choice. There's no need to argue
about which is 'better' or 'worse'. You must decide for yourself what
you need to get done, what language(s) allow you to accomplish those
goals, and which you personally prefer.

 

[7stud --]

Programming languages are a personal choice. There's no need to argue
about which is 'better' or 'worse'. You must decide for yourself what
you need to get done, what language(s) allow you to accomplish those
goals, and which you personally prefer.

Agreed. However, I learn a new languages because I think it will help
me write better programs. Ruby gets a lot of hype as part of ROR, but
everything I read leads me to believe that Ruby is just too slow to be
practical.

 

[Phil]

-----Original Message-----
From: (e-mail address removed) [mailto:[email protected]] On
Behalf Of 7stud --
Sent: Saturday, September 22, 2007 1:02 AM
To: ruby-talk ML
Subject: Re: I am new to Ruby and I could use some expert advice as t
=20
=20
Agreed. However, I learn a new languages because I think it will = help
me write better programs. Ruby gets a lot of hype as part of ROR, but
everything I read leads me to believe that Ruby is just too slow to be
practical.

Tell that the guys at Twitter or 37signals.

For a given problem-domain (i.e. nothing number-crunching), Ruby is Fast =
Enough. If you want speed, Ruby is not for you (although that differs on =
the interpreter you use. JRuby is faster in some situation than Matz's =
implementation, and YARV (Ruby 1.9) seems to have speed improvements, =
too).

Considering how fast today's computers are, and how much memory is =
available on average, neither Ruby's (lack of) speed, nor it's memory =
requirements are a big issue. And where it *is* an issue, it is a matter =
of picking the right tools (C for device drivers and embedded jobs, for =
example, .NET for the Windows world).

Ruby has it's interesting points, especially in duck-typing and =
reflection/meta-programming. Not to mention, that it has a friendly and =
helpful community, and a rather short learning curve (the way to =
mastery, though, is long, much as it is with Go ).

Of course, Ruby provides at least some productivity gains, and =
Programmer time is costlier than CPU time.



In short: Look at what you need, and be pragmatic: If Ruby doesn't fit =
your problem domain, don't use it.

 

[James Edward Gray II]

Ruby gets a lot of hype as part of ROR, but everything I read leads
me to believe that Ruby is just too slow to be practical.

I never understand this statement. It really doesn't mean anything
to me.

Too slow for what? Have you switched algorithms? It's so dang rare
that move alone doesn't eliminate all speed issues. Failing that,
have you moved the bottleneck code into a C extension? Ruby makes
this pretty painless for the very rare times that a good algorithm
won't get you all the way to what you need.

If it's just that you don't like Ruby, I would prefer you say that.
There are a couple of languages that just feel wrong to me, so I
would completely understand. But your excuse doesn't make sense, in
my opinion.

James Edward Gray II

 

[Phrogz]

Agreed. However, I learn a new languages because I think it will help
me write better programs. Ruby gets a lot of hype as part of ROR, but
everything I read leads me to believe that Ruby is just too slow to be
practical.

Too slow for what? As I said, I use it frequently at work, for a lot
of different tasks. Yes, the code generation system that I wrote takes
15 seconds to run, and if you rewrote it in C it might (I'm totally
guessing) only take 1.5 seconds to run.

But it only took me 30 minutes to write something that I would guess
(guessing) would take my very experienced C++ coworkers several hours
to write.

Using these numbers (which admittedly are totally wild-ass guesses), I
would need to run the program 600 times, and have totally lost that
productivity each time, before I would even break even. (In reality,
1.5 seconds versus 15 means very little to me for something that is
run once a week or so.) Ruby has benefited my employer - it was way
more practical to use Ruby than another language.

 

[Phrogz]

Using these numbers (which admittedly are totally wild-ass guesses), I
would need to run the program 600 times, and have totally lost that
productivity each time, before I would even break even. (In reality,
1.5 seconds versus 15 means very little to me for something that is
run once a week or so.) Ruby has benefited my employer - it was way
more practical to use Ruby than another language.

To keep the world in harmony, let me give the other side:

My company DOES employee C++ programmers. Lots of them. We work on
real-time 3D applications that need to support interactive mesh
deformation with vertex weighting using joints, in multiple windows
simultaneously. We develop high speed runtimes for 3D data on next-gen
consoles, where you can't just trade memory for speed, because cache
misses due to using more memory also lose you speed.

In this world, bit packing and high high high speed data structures
and algorithms are paramount. When a racing game is using 98% of the
limited console CPU for its own game engine and only allotting 2% for
the full updating of the 3D HUD graphics and logic, Ruby can't be used
for the runtime. It can't even be used for the minor scripting that
artists use to enhance the logic. (We use Lua for that.)

As I said originally: you need to decide what your needs are, and then
what programs can be used to accomplish those needs. Ruby cannot be
used on a PSP or Wii as a high speed runtime. If that's your goal, you
can just walk away from Ruby.

But there are gobs of other very real world problems for which Ruby is
wholly suitable, where development time is way more important than CPU
or memory efficiency. If those are your goal, I encourage you to
ignore benchmarks and comparisons with other languages. (If Ruby is
100x slower to run than language YYY, and it means that it takes 10
seconds for your entire program versus 0.1 second...do you really
care? If it saves you 20 minutes of coding? If it makes you happier
while you are doing your job?) Instead, see if it works for you.

 

[Mohit Sindhwani]

Agreed. However, I learn a new languages because I think it will help
me write better programs. Ruby gets a lot of hype as part of ROR, but
everything I read leads me to believe that Ruby is just too slow to be
practical.

I think "slow to be practical" is very relative, again. It helps me get
my program done extremely quickly (I don't know Python - so I can't
compare with it, but it is certainly quicker than getting it done in
assembly, C, C++ or Java) and it stays out of my way. Ruby scripts are
not part of my main job but it helps a lot in things like data
manipulation where I would use a combination of tools in the past.

For a lot of the work I do, Ruby is used for internal consumption.
Every minute saved in programming is a bonus there since internal
consumption is very technically a cost. At the same time, we are happy
to let a program run for an extra 30 seconds or even 2 minutes if it's a
one-off. If it's part of a pipeline, we can usually optimize it to meet
the needs of the pipeline. For example, we found that using the base
csv classes were much slower than using line.split(',') and since out
files were extremely predictable, we could easily switch. In other
cases, we do a C extension or just hook it up to call a C executable to
preprocess the data, etc. I think Ruby is a powerful tool in a
heterogeneous environment.

It's not the answer for every problem - but it does play really well
with others. Again, I believe we should use the best tools for the job
and there are many things that I would not/ cannot do in Ruby.

Cheers,
Mohit.
9/22/2007 | 1:59 PM.

 

[William James]

File.exist?(_fname) if File.delete(_fname)

This is equivalent to

if File.delete(_fname)
File.exist?(_fname)
end

That's not what you want. Furthermore, there's no need
to delete the file. If you open it in mode "wb", the
existing contents will be obliterated.

 

[William James]

I am new to Ruby and I could use some expert advice as to how I can
make this code run faster.

def scramble(fname)
f = File.new(fname, "rb")
_fname = fname + ".scrambled"
begin
File.exist?(_fname) if File.delete(_fname)
rescue
end
ff = File.new(_fname, "wb+")
for l in f
l.each_byte{|c| ff.write((c | 0x80).chr) }
end
f.close()
ff.close()
end

About 2.8 times as fast.

def scramble( fname )
_fname = fname + ".scrambled"
mask = 0x8080808080808080
data = File.open( fname, "rb" ){|f| f.read }
pad = (8 - (data.size % 8)) % 8
data << "p" * pad
File.open( _fname, "wb"){|ff|
array = data.unpack('Q*')
array.map!{|x| x | mask }
ff.write( array.pack('Q*')[0 ... -pad] )
}
end

 

[Nobuyoshi Nakada]

Hi,

At Fri, 21 Sep 2007 07:05:10 +0900,
Ruby Maniac wrote in [ruby-talk:270110]:

def scramble(fname)
_fname = fname + ".scrambled"

File.new(fname, "rb") do |f|
File.new(_fname, "wb+") do |ff|
ff.write(f.read.tr("\0-\177", "\200-\377"))
end
end

 

[Nobuyoshi Nakada]

Hi,

Sat, 22 Sep 2007 17:38:51 +0900,
Nobuyoshi Nakada wrote in [ruby-talk:270305]:

At Fri, 21 Sep 2007 07:05:10 +0900,
Ruby Maniac wrote in [ruby-talk:270110]:

def scramble(fname)
_fname = fname + ".scrambled"

File.new(fname, "rb") do |f|
File.new(_fname, "wb+") do |ff|

Sorry, these are not "new" but "open".

 

[William James]

I am new to Ruby and I could use some expert advice as to how I can
make this code run faster.

def scramble(fname)
f = File.new(fname, "rb")
_fname = fname + ".scrambled"
begin
File.exist?(_fname) if File.delete(_fname)
rescue
end
ff = File.new(_fname, "wb+")
for l in f
l.each_byte{|c| ff.write((c | 0x80).chr) }
end
f.close()
ff.close()
end

If you actually want speed, use a compiled language.
This FreeBasic code is about 160 times as fast and
demonstrates that using Python for any reason
whatsoever is unjustified.

Use Ruby when ease of programming is paramount;
use something like FreeBasic or LuaJIT when speed
is paramount. Never submit to Python perversion.

sub scramble( filename as string )
dim outname as string
dim as integer in_handle, out_handle, i, size
dim bytes(1 to 4096) as byte

outname = filename & ".scrambled"
in_handle = freefile
open filename for binary access read as in_handle
out_handle = freefile
open outname for binary access write as out_handle
while not eof( in_handle )
size = seek( in_handle )
get #in_handle, , bytes()
size = seek( in_handle ) - size
for i = 1 to size
bytes(i) = bytes(i) or &h80
next
put #out_handle, , bytes(1), size
wend
close
end sub

 

[William James]

If you actually want speed, use a compiled language.
This FreeBasic code is about 160 times as fast and
demonstrates that using Python for any reason
whatsoever is unjustified.

Use Ruby when ease of programming is paramount;
use something like FreeBasic or LuaJIT when speed
is paramount. Never submit to Python perversion.

sub scramble( filename as string )
dim outname as string
dim as integer in_handle, out_handle, i, size
dim bytes(1 to 4096) as byte

outname = filename & ".scrambled"
in_handle = freefile
open filename for binary access read as in_handle
out_handle = freefile
open outname for binary access write as out_handle
while not eof( in_handle )
size = seek( in_handle )
get #in_handle, , bytes()
size = seek( in_handle ) - size
for i = 1 to size
bytes(i) = bytes(i) or &h80
next
put #out_handle, , bytes(1), size
wend
close
end sub

If that prolixity bothered you, look at a
version in the J programming language.

read =: 1!:1
write =: 1!:2
scramble =: verb define
((128 (23 b.) a. i. read <y) { a. ) write <y,'.scrambled'
)

scramble 'myfile'