Ruby optimization - re-implement in compiled language?

[Ant Sims]

Hi all,

I'm a relative Ruby newbie looking for some advice on optimisation ...

I've been using ruby to implement a make-like build system for my personal
use. The end result was quick to produce, has nice code and does things
the way I like it. Full marks to Ruby

.... Sadly it's also very slow.

I've read the advice in the Programmatic Programmers Guide, removed all
my obvious newbie errors, but it's still too slow.

The profiler identifies all the major bottlenecks as low level ruby
methods (.each etc), so I'm considering re-implemented the guts of the
dependency analysis in C++. I'm sure people here have done similar things,
so I was wondering how beneficial it was for them?

For what it's worth:
The algorithms involve quite deep recursive calls (by necessity)
The scripts appear to pause for several seconds at arbitrary points
- I've tried disabling the GC
I'm using Ruby 1.8.2
Most data is stored in DBM style tables

Thanks in advance ...

 

[Nikolai Weibull]

Ant Sims, April 4:

The algorithms involve quite deep recursive calls (by necessity)

Are you sure? Perhaps you need to rethink your algorithm. Before you
go rewriting your application in another language, a simpler solution is
to rethink your problem/implementation.

Other than that, recursion can be sort of expensive, as Ruby doesn't do
tail-call optimizations (yet),
nikolai

 

[Eric Hodel]

--Apple-Mail-2--1015944003
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; format=flowed

Hi all,

I'm a relative Ruby newbie looking for some advice on optimisation ...

I've been using ruby to implement a make-like build system for my
personal
use. The end result was quick to produce, has nice code and does things
the way I like it. Full marks to Ruby

Is there something that Rake doesn't do to fit your needs?

Or make, for that matter?

--
Eric Hodel - (e-mail address removed) - http://segment7.net
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

--Apple-Mail-2--1015944003
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iEYEARECAAYFAkJQqkYACgkQMypVHHlsnwT+igCg03ffAfvWW1oT0PmDWypAAaTf
8+0An14N8RNOe9y5PHOADKFsh2yyW1pw
=0YB+
-----END PGP SIGNATURE-----

--Apple-Mail-2--1015944003--

 

[Aredridel]

I've read the advice in the Programmatic Programmers Guide, removed all

my obvious newbie errors, but it's still too slow.

The profiler identifies all the major bottlenecks as low level ruby
methods (.each etc), so I'm considering re-implemented the guts of the
dependency analysis in C++. I'm sure people here have done similar things,
so I was wondering how beneficial it was for them?

Each shows high in a profiler, because the entire contents of the inner
part of the loop count against it. Just so you're warned.

Ari

 

[Randy Kramer]

Each shows high in a profiler, because the entire contents of the inner
part of the loop count against it. Just so you're warned.

So, iiuc, you're saying that the profiler sort of counts things multiple
times--when you look at any individual part of the inner loop
(wherever/whatever that might be), the entire time spent in that inner loop
is "charged" to each individual part that you might look at?

So, you really can't tell which of those parts is the biggest contributor to
the (presumed) problem?

Randy Kramer

 

[Robert Klemme]

So, iiuc, you're saying that the profiler sort of counts things multiple
times--when you look at any individual part of the inner loop
(wherever/whatever that might be), the entire time spent in that inner loop
is "charged" to each individual part that you might look at?

So, you really can't tell which of those parts is the biggest contributor to
the (presumed) problem?

It's a bit different. First of all there is a column "cumulative seconds"
in the output which actually sums up time through the output table (i.e.
"cumulative seconds" of one row is the value of the previous row plus this
rows "self seconds".

(Note: percent values are garbled because of #sleep)

15:16:01 [source]: ruby -r profile -e '1000.times { sleep 0.01 }'
% cumulative self self total
time seconds seconds calls ms/call ms/call name
206.67 0.03 0.03 1 31.00 31.00
Profiler__.start_profile
100.00 0.05 0.01 1000 0.01 0.01 Kernel.sleep
0.00 0.05 0.00 1 0.00 15.00 Integer#times
0.00 0.05 0.00 1 0.00 15.00 #toplevel

Then if you iterate only once through any Enumerable then #each shows up
as 1 call - which can be confusing. But it's completely logical since
it's actually called only once. Just the block is invoked multiple times.
So, assuming that you have a lot of stuff going on in a block, that time
can be seen nowhere else than within the time of #each - there's no extra
marker for the block. Of course you'll see times of methods that are
called from the block on rows of their own.

Another example with ungarbled percent values:

15:22:48 [source]: ruby -r profile -e '100.times { i=0; i+=1 while i <
1000 }'
% cumulative self self total
time seconds seconds calls ms/call ms/call name
59.89 15.11 15.11 1 15113.00 25234.00 Integer#times
20.83 20.37 5.26 100000 0.05 0.05 Fixnum#+
19.28 25.23 4.87 100100 0.05 0.05 Fixnum#<
0.12 25.27 0.03 1 31.00 31.00
Profiler__.start_profile
0.00 25.27 0.00 1 0.00 25234.00 #toplevel

Kind regards

robert