[JRuby] JRuby perf questions answered

Discussion in 'Ruby' started by Charles Oliver Nutter, Oct 31, 2007.

  1. (Sorry if this is a duplicate post...the original seems to have disappeared)

    Many folks still seem to be in the dark about JRuby performance relative
    to Ruby 1.8.x. I figured a short post with some basic info could start a
    discussion for anyone wanting to know more.

    Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and
    for most execution benchmarks it is the fastest 1.8-compatible Ruby
    implementation available. This is based on the YARV benchmarks, the
    Alioth benchmarks, a number of benchmarks we created for JRuby, and a
    few other external benchmarks we run from time to time. Many (most?) of
    these are microbenchmarks, but a number are nontrivial code.

    For example, M. Ed. Borasky's MatrixBenchmark:

    JRuby:
    Hilbert matrix of dimension 64 times its inverse = identity? true
    18.784000 0.000000 18.784000 ( 18.784000)

    Ruby 1.8.6:
    Hilbert matrix of dimension 64 times its inverse = identity? true
    32.560000 0.110000 32.670000 ( 32.776915)

    Or the YARV "pentomino" benchmark:

    ~/NetBeansProjects/jruby $ time jruby -J-server
    test/bench/yarv/bm_app_pentomino.rb

    real 1m31.770s
    user 1m32.016s
    sys 0m1.278s
    ~/NetBeansProjects/jruby $ time ruby test/bench/yarv/bm_app_pentomino.rb

    real 1m48.100s
    user 1m47.489s
    sys 0m0.287s

    Ola Bini of the JRuby team recently ran all the YARV benchmarks and
    reported the results here:

    http://ola-bini.blogspot.com/2007/10/mystery-expos-on-jruby-performance.html

    And I periodically post about performance progress on my blog here:

    http://headius.blogspot.com

    I expect we'll see Alioth updated soon after the 1.1 or 1.1 beta 1 releases.

    What about Rails?

    One of the biggest uses for JRuby recently has been Rails. Rails
    performance is a more complicated thing to measure; there's a lot of
    non-execution bottlenecks that get in the way. But both Ola and Nick
    Sieger (also of JRuby team) have found that JRuby on Rails performance
    is either very near or exceeding Ruby 1.8.x on Rails:

    http://ola-bini.blogspot.com/2007/10/current-state-of-regular-expressions.html
    (toward the bottom of the post are updated Rails numbers)

    http://blog.nicksieger.com/articles/2007/10/25/jruby-on-rails-fast-enough

    Rails is probably the ultimate real-world benchmark, and we're excited
    to see that JRuby's starting to pass Ruby 1.8.x performance here as well.

    I won't drag this on any longer, but I'd encourage you to try out your
    own Ruby code and let us know how it performs. We've spent a lot of time
    on performance issues for the 1.1 release, and have many future plans to
    continue increasing performance. We want your input.

    Many people believed we'd never be faster than the C implementation, and
    many still think we're slower. Now that I've set that record straight,
    any questions?

    - Charlie
     
    Charles Oliver Nutter, Oct 31, 2007
    #1
    1. Advertising

  2. Charles Oliver Nutter

    Roger Pack Guest

    Re: JRuby perf questions answered

    > any questions?

    Startup time and comparison with 1.9? :)
    --
    Posted via http://www.ruby-forum.com/.
     
    Roger Pack, Oct 31, 2007
    #2
    1. Advertising

  3. Charles Oliver Nutter

    Roger Pack Guest

    Re: JRuby perf questions answered

    Roger Pack wrote:
    >> any questions?

    >
    > Startup time and comparison with 1.9? :)


    Also memory use comparison compared to normal C version. Does the GC
    throw as often, too?
    --
    Posted via http://www.ruby-forum.com/.
     
    Roger Pack, Oct 31, 2007
    #3
  4. Re: JRuby perf questions answered

    Roger Pack wrote:
    > Roger Pack wrote:
    >>> any questions?

    >> Startup time and comparison with 1.9? :)

    >
    > Also memory use comparison compared to normal C version. Does the GC
    > throw as often, too?


    In profiling most apps I've tried, even pretty brutal ones generating a
    lot of garbage, the GC cost is almost invisible. A tiny blip once in a
    while.

    I don't have any really good concrete numbers on overall memory usage.
    Tom's done a comparison of a minimal/bare Rails app here:

    http://www.bloglines.com/blog/ThomasEEnebo?id=38

    The general story is that per JRuby on Rails instance (usually you have
    many in a single JVM), we use less memory than an equivalent Mongrel +
    Ruby + Rails process.

    A quick check of "jirb" compared to "irb" gives me:

    jirb: 27MB
    irb: 2.8MB

    This is on OS X, Java 6 preview version, so it's safe to say that any
    JRuby process will have an initial cost of about 25MB to start up the JVM.

    We've also done a lot in 1.1 to reduce memory consumption, and there's
    more work coming up.

    - Charlie
     
    Charles Oliver Nutter, Nov 1, 2007
    #4
  5. Re: JRuby perf questions answered

    There was a recent thread which asked how fast ruby ran for each user.

    http://www.ruby-forum.com/topic/125344#new

    Josip Gracin wrote:
    > Here's something interesting. The processor is Intel T7200 @ 2.00GHz,
    > Linux Ubuntu.
    >
    > 1. With ruby 1.8.5. which comes packaged with Ubuntu:
    >
    > Ruby 1.8.5 on i486-linux
    > It took 20.346013 seconds to run. 49149 iterations per
    > second.
    >
    > 2. With ruby 1.9.0 (2006-06-08) which comes packaged in Ubuntu:
    >
    > Ruby 1.9.0 on i486-linux
    > It took 23.875774 seconds to run. 41883 iterations per
    > second.
    >
    >
    > 3. With ruby 1.8.6 compiled from sources on the machine:
    >
    > Ruby 1.8.6 on i686-linux
    > It took 7.107875 seconds to run. 140689 iterations per
    > second.
    >
    >
    > Obviously, the target instruction set matters quite a lot. I think I'll
    > be compiling my own ruby from now on. :)


    Greg Halsey wrote:
    > windows xp. intel core2 1.86GHz
    >
    > JRUBY:
    >
    > C:\jruby\jruby-trunk\bin>jruby calculate.rb
    > 55
    >
    > It took 41.125 seconds to run. 24316 iterations per
    > second.
    >
    > Not so good. Let's disable ObjectSpace:
    > C:\jruby\jruby-trunk\bin>jruby -O calculate.rb
    > 55
    >
    > It took 11.109 seconds to run. 90017 iterations per
    > second.
    >
    > That was good. Now let's give it some more juice:
    > C:\jruby\jruby-trunk\bin>jruby -J-server -O calculate.rb
    > 55
    >
    > It took 5.75 seconds to run. 173913 iterations per
    > second.
    > --------------------------------------------
    > MRI:One-Click
    > ruby calculate.rb
    > 55
    >
    > Ruby 1.8.6 patch 0 on i386-mswin32
    > It took 15.828 seconds to run. 63179 iterations per
    > second.


    Now, I am thinking that jruby might work well if we know how to find the
    sweet spot. Why the pre compiled version is so very much slower than a
    home compiled version makes me wonder why they do not compile it better.
    As things can be so radically different in jruby if only you know how to
    tweak it perfectly is another thing that makes me think.

    So, do you find your own benchmarks and work your compiler to make it
    run at its fastest or is performance going to be "roll your own" for
    those with insider knowledge?
    --
    Posted via http://www.ruby-forum.com/.
     
    Lloyd Linklater, Nov 1, 2007
    #5
  6. Re: JRuby perf questions answered

    Roger Pack wrote:
    >> any questions?

    >
    > Startup time and comparison with 1.9? :)


    Startup time varies widely between platforms. OS X is probably one of
    the fastest:

    ~/NetBeansProjects/jruby $ time ruby -e "puts 'hello'"
    hello

    real 0m0.015s
    user 0m0.007s
    sys 0m0.008s
    ~/NetBeansProjects/jruby $ time jruby -e "puts 'hello'"
    hello

    real 0m1.815s
    user 0m1.709s
    sys 0m0.183s

    There's been reports that on server-class machines the startup time can
    be as high as 5-10 seconds. The slow startup time is partially due to
    the JVM and partially our fault (JRuby's complicated, yo).

    As for Ruby 1.9...it's hard to make a comparison right now. Most of the
    benchmarks we use to measure Ruby 1.9 (like those I described
    previously) are usually a bit faster in Ruby 1.9. And of course almost
    all of Ruby 1.9's benchmarks are fastest in Ruby 1.9.

    Part of the difficulty comparing is that Ruby 1.9 contains a bunch of
    changes to Ruby language semantics that we have not yet put in JRuby,
    since we've focused on 1.8 compatibility. Things like:

    - fast math operations, that speed up +, -, etc, but you can't override them
    - constant-time case/when when literal values are used, but you can't
    override === for those literal types

    Here's A few numbers, for your edification. Some things are faster in
    Ruby 1.9:

    JRuby:
    ~/NetBeansProjects/jruby $ jruby -J-server
    test/bench/bench_fib_recursive.rb
    0.869000 0.000000 0.869000 ( 0.869000)
    0.677000 0.000000 0.677000 ( 0.677000)
    0.652000 0.000000 0.652000 ( 0.652000)
    0.648000 0.000000 0.648000 ( 0.648000)

    Ruby 1.9:
    ~/NetBeansProjects/jruby $ ../ruby1.9/ruby -I ../ruby1.9/lib
    test/bench/bench_fib_recursive.rb
    0.400000 0.000000 0.400000 ( 0.405639)
    0.400000 0.000000 0.400000 ( 0.406707)
    0.400000 0.000000 0.400000 ( 0.400150)
    0.400000 0.000000 0.400000 ( 0.398065)

    And some things are slower:

    JRuby:
    1m loops yielding three fixnums 10 times to block splatting and
    accessing them
    3.387000 0.000000 3.387000 ( 3.387000)
    3.314000 0.000000 3.314000 ( 3.315000)
    3.235000 0.000000 3.235000 ( 3.235000)

    Ruby 1.9:
    1m loops yielding three fixnums 10 times to block splatting and
    accessing them
    4.980000 0.020000 5.000000 ( 5.055369)
    4.980000 0.020000 5.000000 ( 5.073821)
    5.010000 0.020000 5.030000 ( 5.088681)

    And so on. However, for Rails, I've only seen one comparison that showed
    Ruby 1.9 was either slightly faster or slower than Ruby 1.8.x:

    http://eigenclass.org/hiki/non-synthetic-benchmarks-for-yarv

    I'm not sure if there's been a more recent comparison. I believe we'll
    be able to match Ruby 1.9 performance once we start making the same
    optimizations.

    - Charlie
     
    Charles Oliver Nutter, Nov 1, 2007
    #6
  7. Re: JRuby perf questions answered

    Lloyd Linklater wrote:
    > There was a recent thread which asked how fast ruby ran for each user.
    >
    > http://www.ruby-forum.com/topic/125344#new
    >
    > Josip Gracin wrote:
    >> Here's something interesting. The processor is Intel T7200 @ 2.00GHz,
    >> Linux Ubuntu.
    >>
    >> 1. With ruby 1.8.5. which comes packaged with Ubuntu:
    >>
    >> Ruby 1.8.5 on i486-linux
    >> It took 20.346013 seconds to run. 49149 iterations per
    >> second.
    >>
    >> 2. With ruby 1.9.0 (2006-06-08) which comes packaged in Ubuntu:
    >>
    >> Ruby 1.9.0 on i486-linux
    >> It took 23.875774 seconds to run. 41883 iterations per
    >> second.
    >>
    >>
    >> 3. With ruby 1.8.6 compiled from sources on the machine:
    >>
    >> Ruby 1.8.6 on i686-linux
    >> It took 7.107875 seconds to run. 140689 iterations per
    >> second.
    >>
    >>
    >> Obviously, the target instruction set matters quite a lot. I think I'll
    >> be compiling my own ruby from now on. :)

    >
    > Greg Halsey wrote:
    >> windows xp. intel core2 1.86GHz
    >>
    >> JRUBY:
    >>
    >> C:\jruby\jruby-trunk\bin>jruby calculate.rb
    >> 55
    >>
    >> It took 41.125 seconds to run. 24316 iterations per
    >> second.
    >>
    >> Not so good. Let's disable ObjectSpace:
    >> C:\jruby\jruby-trunk\bin>jruby -O calculate.rb
    >> 55
    >>
    >> It took 11.109 seconds to run. 90017 iterations per
    >> second.
    >>
    >> That was good. Now let's give it some more juice:
    >> C:\jruby\jruby-trunk\bin>jruby -J-server -O calculate.rb
    >> 55
    >>
    >> It took 5.75 seconds to run. 173913 iterations per
    >> second.
    >> --------------------------------------------
    >> MRI:One-Click
    >> ruby calculate.rb
    >> 55
    >>
    >> Ruby 1.8.6 patch 0 on i386-mswin32
    >> It took 15.828 seconds to run. 63179 iterations per
    >> second.

    >
    > Now, I am thinking that jruby might work well if we know how to find the
    > sweet spot. Why the pre compiled version is so very much slower than a
    > home compiled version makes me wonder why they do not compile it better.
    > As things can be so radically different in jruby if only you know how to
    > tweak it perfectly is another thing that makes me think.
    >
    > So, do you find your own benchmarks and work your compiler to make it
    > run at its fastest or is performance going to be "roll your own" for
    > those with insider knowledge?


    We have been running whatever benchmarks we can, and they're all pretty
    generic. For all the numbers I've reported (and I believe most of the
    numbers others have reported for JRuby), it's been JRuby trunk (soon to
    be 1.1) with the Java 6 "server" VM. There's really no more tweaking
    than that required to get the same performance.

    As for the compiler...there's no additional user tweaking necessary to
    make it run best; it should run its best all the time. The additional
    tweaks you may want to apply would be using a different Java version,
    tweaking the JVM or GC settings, and so on. Normal server tuning most
    people won't need.

    The reality with the JVM, and especially with JRuby, is that short tests
    aren't going to show the real long-term performance. For most execution,
    JRuby starts out interpreted, and the interpreter is slower than Ruby
    1.8.x. But after running for a while, code will eventually compile to
    Java bytecode (or you can precompile to skip that time). Then there's an
    additional delay while the JVM itself eventually compiles that bytecode
    down to platform-native code.

    We expect that for JRuby 1.1, the main sweet spot will be on
    longer-running apps e.g. for server-side apps or for heavy data processing.
     
    Charles Oliver Nutter, Nov 1, 2007
    #7
  8. Re: JRuby perf questions answered

    Charles Oliver Nutter wrote:
    > JRuby 1.1 is now generally faster than Ruby 1.8.6, and
    > for most execution benchmarks it is the fastest 1.8-compatible Ruby
    > implementation available.


    I look forward to getting my hands on JRuby 1.1 and running some Rails
    apps. Keep up the good work!
    --
    Posted via http://www.ruby-forum.com/.
     
    Kevin Williams, Nov 1, 2007
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. kayodeok
    Replies:
    1
    Views:
    368
  2. SithVegeta

    Just need a few questions answered...

    SithVegeta, Nov 15, 2004, in forum: C Programming
    Replies:
    6
    Views:
    391
    SithVegeta
    Nov 15, 2004
  3. Replies:
    10
    Views:
    580
  4. Syam
    Replies:
    25
    Views:
    887
    Kai-Uwe Bux
    Dec 2, 2006
  5. Charles Oliver Nutter

    JRuby performance questions answered

    Charles Oliver Nutter, Oct 31, 2007, in forum: Ruby
    Replies:
    16
    Views:
    209
Loading...

Share This Page