Some benchmarks with MinGW, JRuby, MSVC...

Discussion in 'Ruby' started by Joao Pedrosa, Nov 30, 2008.

  1. Joao Pedrosa

    Joao Pedrosa Guest

    So, I have managed to build several different versions of Ruby to
    compare all
    in some benchmarks. I am going to let the numbers talk for themselves.
    It was
    all tested on an old desktop computer with Windows XP, so I am not
    testing
    anything too impressive.

    Of importance is the seemingly poor performance of Ruby 1.8.7 for
    whatever
    reason. It's as if Ruby 1.8.7 were like Windows ME, a version between
    major
    versions, as an analogy. :)

    Also, I was not able to run the Mongrel gem on Ruby 1.9.1.

    I am going to order the data by fastest to make it easier to compare
    it on text.
    I have collected the data on a Google Spreadsheet:
    http://spreadsheets.google.com/ccc?key=pmtaDyNoButGKeSVdIGi7Fg

    I am going to keep it short here and for more detail on versions and
    whatnot
    please refer to the spreadsheet.

    == WEBrick (req/s)

    22.39 - jruby
    16.95 - jruby --server
    12.82 - ruby oneclick 1.8.6
    12.49 - ruby msvc 1.8.6
    12.45 - ruby rubyinstaller 1.8.6
    12.43 - ruby mingw 1.8.6
    11.94 - ruby msvc 1.9.1
    11.48 - ruby mingw 1.9.1
    3.42 - ruby msvc 1.8.7
    3.40 - ruby mingw 1.8.7


    == Mongrel (req/s)

    357.14 - ruby mingw 1.8.6
    347.45 - ruby rubyinstaller 1.8.6
    326.70 - ruby msvc 1.8.6
    313.57 - ruby oneclick 1.8.6
    274.68 - ruby mingw 1.8.7
    232.39 - ruby msvc 1.8.7
    71.24 - jruby
    62.22 - jruby --server
    (crash when starting up) - ruby mingw 1.9.1
    (error when building gem) - ruby msvc 1.9.1


    == bench_pythag.rb (s)

    19.09 - jruby --server
    24.35 - jruby
    31.72 - ruby mingw 1.9.1
    31.91 - ruby mingw 1.8.6
    32.73 - ruby mingw 1.8.7
    33.94 - ruby msvc 1.9.1
    35.67 - ruby rubyinstaller 1.8.6
    35.83 - ruby msvc 1.8.7
    39.79 - ruby msvc 1.8.6
    48.30 - ruby oneclick 1.8.6


    == bench_fractal.rb (s)

    3.22 - jruby --server
    3.89 - jruby
    5.67 - ruby msvc 1.9.1
    5.75 - ruby mingw 1.9.1
    9.80 - ruby mingw 1.8.6
    9.90 - ruby mingw 1.8.7
    10.11 - ruby rubyinstaller 1.8.6
    10.30 - ruby msvc 1.8.6
    10.64 - ruby msvc 1.8.7
    15.17 - ruby oneclick 1.8.6


    == bench_float_math.rb (s)

    103.44 - jruby --server
    125.89 - jruby
    158.64 - ruby msvc 1.9.1
    175.41 - ruby mingw 1.9.1
    176.61 - ruby mingw 1.8.6
    179.94 - ruby mingw 1.8.7
    184.73 - ruby rubyinstaller 1.8.6
    195.37 - ruby msvc 1.8.7
    197.39 - ruby msvc 1.8.6
    279.62 - ruby oneclick 1.8.6


    Since I collected these data, the Ruby 1.9.1 branch has had a number
    of updates
    for one thing. :)

    Benchmarks can be kind of tough to do right and there are all
    possibilities when it
    comes to doing them.

    If anyone has an algorithm that he would like to test on all those
    Ruby implementations
    I would be glad to test it. With the exception of testing Rails, which
    I have never used
    and don't know enough about it to willingly make myself available when
    testing it. :)

    Cheers,
    Joao
    Joao Pedrosa, Nov 30, 2008
    #1
    1. Advertising

  2. Joao Pedrosa

    Post Tudbc Guest

    Post Tudbc, Nov 30, 2008
    #2
    1. Advertising

  3. Joao Pedrosa wrote:
    > == Mongrel (req/s)
    >
    > 357.14 - ruby mingw 1.8.6
    > 347.45 - ruby rubyinstaller 1.8.6
    > 326.70 - ruby msvc 1.8.6
    > 313.57 - ruby oneclick 1.8.6
    > 274.68 - ruby mingw 1.8.7
    > 232.39 - ruby msvc 1.8.7
    > 71.24 - jruby
    > 62.22 - jruby --server
    > (crash when starting up) - ruby mingw 1.9.1
    > (error when building gem) - ruby msvc 1.9.1


    I'd love to figure out what makes mongrel slow for us, when all other
    cases are faster ;)

    GlassFish gem would be a nice one to test, but of course there's no
    equivalent comparison for the others.

    - Charlie
    Charles Oliver Nutter, Dec 1, 2008
    #3
  4. Benchmarking ruby on windows is fairly useless. MRI 1.8.* is known to
    be very slow on windows and harldy anyone deploys production apps to
    windows because of this. If you want your benches to be taken
    seriously you need to run them on linux, bsd or osx where ruby
    performs much better. Also make sure to compile ruby with --disable-
    pthreads for best perf.


    Cheers-
    Ezra Zygmuntowicz





    On Nov 30, 2008, at 9:14 AM, Joao Pedrosa wrote:

    > So, I have managed to build several different versions of Ruby to
    > compare all
    > in some benchmarks. I am going to let the numbers talk for themselves.
    > It was
    > all tested on an old desktop computer with Windows XP, so I am not
    > testing
    > anything too impressive.
    >
    > Of importance is the seemingly poor performance of Ruby 1.8.7 for
    > whatever
    > reason. It's as if Ruby 1.8.7 were like Windows ME, a version between
    > major
    > versions, as an analogy. :)
    >
    > Also, I was not able to run the Mongrel gem on Ruby 1.9.1.
    >
    > I am going to order the data by fastest to make it easier to compare
    > it on text.
    > I have collected the data on a Google Spreadsheet:
    > http://spreadsheets.google.com/ccc?key=pmtaDyNoButGKeSVdIGi7Fg
    >
    > I am going to keep it short here and for more detail on versions and
    > whatnot
    > please refer to the spreadsheet.
    >
    > == WEBrick (req/s)
    >
    > 22.39 - jruby
    > 16.95 - jruby --server
    > 12.82 - ruby oneclick 1.8.6
    > 12.49 - ruby msvc 1.8.6
    > 12.45 - ruby rubyinstaller 1.8.6
    > 12.43 - ruby mingw 1.8.6
    > 11.94 - ruby msvc 1.9.1
    > 11.48 - ruby mingw 1.9.1
    > 3.42 - ruby msvc 1.8.7
    > 3.40 - ruby mingw 1.8.7
    >
    >
    > == Mongrel (req/s)
    >
    > 357.14 - ruby mingw 1.8.6
    > 347.45 - ruby rubyinstaller 1.8.6
    > 326.70 - ruby msvc 1.8.6
    > 313.57 - ruby oneclick 1.8.6
    > 274.68 - ruby mingw 1.8.7
    > 232.39 - ruby msvc 1.8.7
    > 71.24 - jruby
    > 62.22 - jruby --server
    > (crash when starting up) - ruby mingw 1.9.1
    > (error when building gem) - ruby msvc 1.9.1
    >
    >
    > == bench_pythag.rb (s)
    >
    > 19.09 - jruby --server
    > 24.35 - jruby
    > 31.72 - ruby mingw 1.9.1
    > 31.91 - ruby mingw 1.8.6
    > 32.73 - ruby mingw 1.8.7
    > 33.94 - ruby msvc 1.9.1
    > 35.67 - ruby rubyinstaller 1.8.6
    > 35.83 - ruby msvc 1.8.7
    > 39.79 - ruby msvc 1.8.6
    > 48.30 - ruby oneclick 1.8.6
    >
    >
    > == bench_fractal.rb (s)
    >
    > 3.22 - jruby --server
    > 3.89 - jruby
    > 5.67 - ruby msvc 1.9.1
    > 5.75 - ruby mingw 1.9.1
    > 9.80 - ruby mingw 1.8.6
    > 9.90 - ruby mingw 1.8.7
    > 10.11 - ruby rubyinstaller 1.8.6
    > 10.30 - ruby msvc 1.8.6
    > 10.64 - ruby msvc 1.8.7
    > 15.17 - ruby oneclick 1.8.6
    >
    >
    > == bench_float_math.rb (s)
    >
    > 103.44 - jruby --server
    > 125.89 - jruby
    > 158.64 - ruby msvc 1.9.1
    > 175.41 - ruby mingw 1.9.1
    > 176.61 - ruby mingw 1.8.6
    > 179.94 - ruby mingw 1.8.7
    > 184.73 - ruby rubyinstaller 1.8.6
    > 195.37 - ruby msvc 1.8.7
    > 197.39 - ruby msvc 1.8.6
    > 279.62 - ruby oneclick 1.8.6
    >
    >
    > Since I collected these data, the Ruby 1.9.1 branch has had a number
    > of updates
    > for one thing. :)
    >
    > Benchmarks can be kind of tough to do right and there are all
    > possibilities when it
    > comes to doing them.
    >
    > If anyone has an algorithm that he would like to test on all those
    > Ruby implementations
    > I would be glad to test it. With the exception of testing Rails, which
    > I have never used
    > and don't know enough about it to willingly make myself available when
    > testing it. :)
    >
    > Cheers,
    > Joao
    >
    Ezra Zygmuntowicz, Dec 1, 2008
    #4
  5. Joao Pedrosa

    Shane Emmons Guest

    [Note: parts of this message were removed to make it a legal post.]

    On Mon, Dec 1, 2008 at 2:17 PM, Ezra Zygmuntowicz <>wrote:

    >
    > Benchmarking ruby on windows is fairly useless. MRI 1.8.* is known to be
    > very slow on windows and harldy anyone deploys production apps to windows
    > because of this. If you want your benches to be taken seriously you need to
    > run them on linux, bsd or osx where ruby performs much better. Also make
    > sure to compile ruby with --disable-pthreads for best perf.
    >
    >
    > Cheers-
    > Ezra Zygmuntowicz
    >



    Though it might be slow, I really don't think it's useless. Where I work, we
    are stuck in windows no matter the cost or benefit of another platform.
    Because of this these windows numbers actually matter to me.

    --
    Shane Emmons
    Shane Emmons, Dec 1, 2008
    #5
  6. Joao Pedrosa

    Post Tudbc Guest

    Post Tudbc wrote:
    > According to my own tests, "Thin" should be better than your current
    > best, which is Mongrel. Furthermore, TudbcJRubyServlet should help JRuby
    > win over everyone.
    >
    > http://www.tudbc.org/tudbcjrubyservlet
    >
    > You should also give more details about how you test.


    I notice that you use the best of 5 runs to report numbers, which I
    think it is going to vary a lot. You can use my free tool to do web
    benchmarking, which I think is much more accurate.
    http://www.tudbc.org/tudbcwebbenchmark

    I think your Machine is about same as my old test machine. Here are some
    numbers:

    Using "Echo" test: (copied from http://www.tudbc.org/jrubywebapp)

    0.98ms TudbcJRubyServlet_RB
    1.73ms Thin
    4.04ms Mongrel (estimated, because Mongrel crashed in the middle)
    8.51ms WEBrick


    Using your "time now+version" test:

    1.11ms TudbcJRubyServlet_RB
    1.56ms Thin
    4.00ms Mongrel (estimated, because Mongrel crashed in the middle)
    8.18ms WEBrick

    As you can see in your "time now" test on your machine, your Mongrel
    results were probably inaccurate, because you used the "best of 5" to
    measure.

    Both my "Echo" and your "Time now+version" tests seem to produce similar
    benchmarks. As you can see, TudbcJRubyServlet_RB still win easily, and
    Thin is faster than Mongrel by a wide margin.

    Note: The default measurement of TudbcWebBenchmark is average of last 10
    of 20 runs (with each run for 10 seconds). Because Mongrel crashed, so I
    only estimate it by 3rd run of 5 seconds each.

    Note: Ruby version used: JRuby 1.1.5 and Ruby 1.8.7-p72 on Windows
    --
    Posted via http://www.ruby-forum.com/.
    Post Tudbc, Dec 1, 2008
    #6
  7. Joao Pedrosa

    Post Tudbc Guest

    Oops! Forgot to paste the codes I used for TudbcJRubyServlet

    --------- timeversion.rb --------
    a = []
    a << 'Hello Rack!'
    a << Time.new
    a << RUBY_PLATFORM
    a << RUBY_VERSION
    ctx.print(a.join('<br />'))
    ---------------------------------

    Your "time now+version" uses 5 array ops. My guess is that maybe JRuby
    array processing is relatively not as efficient, so that ratio between
    TudbcJRubyServlet and Thin for "time now+version" is not as great as
    "Echo" test.

    Note: My 4ms means your 1000/4 or 250 reqs/sec.

    --
    Posted via http://www.ruby-forum.com/.
    Post Tudbc, Dec 1, 2008
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alf P. Steinbach
    Replies:
    21
    Views:
    9,570
    =?iso-8859-1?q?Stephan_Br=F6nnimann?=
    Dec 15, 2004
  2. Alf P. Steinbach
    Replies:
    28
    Views:
    810
    Alf P. Steinbach
    Dec 18, 2004
  3. Alf P. Steinbach
    Replies:
    13
    Views:
    672
    Alf P. Steinbach
    Feb 1, 2005
  4. Alf P. Steinbach
    Replies:
    7
    Views:
    3,075
    chris
    Jan 24, 2005
  5. Alf P. Steinbach
    Replies:
    10
    Views:
    776
    Alf P. Steinbach
    Feb 23, 2005
Loading...

Share This Page