Help interpreting benchmark results

Discussion in 'Ruby' started by Juan Alvarez, Feb 24, 2009.

  1. Juan Alvarez

    Juan Alvarez Guest

    I ran this simple benchmark:

    Benchmark.bm do |x|
    x.report('Curl::Easy.http_get') do
    Curl::Easy.http_get(text)
    end

    x.report('RestClient.get') do
    RestClient.get(text)
    end
    end

    RestClient uses the standard Net:HTTP library so the benchmark is really
    trying to compare Net::HTTP vs curb (libcurl bindings).

    These are the results I get:

    user system total real
    Curl::Easy.http_get 0.070000 0.130000 0.200000 ( 26.319104)
    RestClient.get 0.970000 0.580000 1.550000 ( 23.557923)

    I get these results consistently where curb's user, system and total
    times are dramatically lower than Net::HTTP's. However, Net:HTTP is
    reported to run faster by the real column. Shouldn't the reported real
    time be consistent with the other columns?

    I'm running this benchmark on a 2.4Ghz Core 2 Duo with 2GB RAM MacBook
    Pro, if that helps.

    Thanks,
    Juan
    --
    Posted via http://www.ruby-forum.com/.
    Juan Alvarez, Feb 24, 2009
    #1
    1. Advertising

  2. On 24.02.2009, at 23:35, Juan Alvarez wrote:

    > These are the results I get:
    >
    > user system total =20
    > real
    > Curl::Easy.http_get 0.070000 0.130000 0.200000 ( 26.319104)
    > RestClient.get 0.970000 0.580000 1.550000 ( 23.557923)
    >
    > I get these results consistently where curb's user, system and total
    > times are dramatically lower than Net::HTTP's. However, Net:HTTP is
    > reported to run faster by the real column. Shouldn't the reported real
    > time be consistent with the other columns?


    No. "real" measures the time that is passed from start to finish.

    Pseudocode
    real_start =3D Time.now
    yield # run test
    real_end =3D Time.now
    real_start.to_f - real_end.to_f


    If the Kernel of your OS schedules another processes for, say 20 =20
    seconds,
    then the difference between total and real is >=3D 20 seconds.
    I think the time called "real" should not be trusted in an evaluation.

    regards, Sandor Sz=FCcs
    --
    Sandor Szücs, Feb 25, 2009
    #2
    1. Advertising

  3. 2009/2/25 Sandor Sz=FCcs <>:
    >
    > On 24.02.2009, at 23:35, Juan Alvarez wrote:
    >
    >> These are the results I get:
    >>
    >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 user =A0=

    =A0 system =A0 =A0 =A0total =A0 =A0 =A0 =A0real
    >> Curl::Easy.http_get =A0 0.070000 =A0 0.130000 =A0 0.200000 ( 26.319104)
    >> RestClient.get =A0 =A0 =A0 =A0 =A00.970000 =A0 0.580000 =A0 1.550000 ( 2=

    3.557923)
    >>
    >> I get these results consistently where curb's user, system and total
    >> times are dramatically lower than Net::HTTP's. However, Net:HTTP is
    >> reported to run faster by the real column. Shouldn't the reported real
    >> time be consistent with the other columns?

    >
    > No. "real" measures the time that is passed from start to finish.
    >
    > Pseudocode
    > =A0real_start =3D Time.now
    > =A0yield # run test
    > =A0real_end =3D Time.now
    > =A0real_start.to_f - real_end.to_f
    >
    >
    > If the Kernel of your OS schedules another processes for, say 20 seconds,
    > then the difference between total and real is >=3D 20 seconds.
    > I think the time called "real" should not be trusted in an evaluation.


    Not so fast. If another process has been running that may well be a
    process spawned by our process.

    13:19:09 Temp$ ruby19 bb.rb
    user system total real
    other process 0.000000 0.000000 0.139000 ( 10.375000)
    in process 4.594000 0.000000 4.594000 ( 4.765000)
    13:19:30 Temp$ cat bb.rb
    require 'benchmark'

    Benchmark.bm 20 do |b|
    b.report 'other process' do
    system *%w{sleep 10}
    end

    b.report 'in process' do
    10_000_000.times {|i| i + 1}
    end
    end
    13:19:35 Temp$

    I'd check the curl lib - I would assume that it uses the command line
    utility of the same name - which then is another process.

    Kind regards

    robert

    --=20
    remember.guy do |as, often| as.you_can - without end
    Robert Klemme, Feb 25, 2009
    #3
  4. On 25.02.2009, at 13:20, Robert Klemme wrote:

    > 2009/2/25 Sandor Sz=FCcs <>:
    >>
    >> No. "real" measures the time that is passed from start to finish.
    >>
    >> Pseudocode
    >> real_start =3D Time.now
    >> yield # run test
    >> real_end =3D Time.now
    >> real_start.to_f - real_end.to_f
    >>
    >>
    >> If the Kernel of your OS schedules another processes for, say 20 =20
    >> seconds,
    >> then the difference between total and real is >=3D 20 seconds.
    >> I think the time called "real" should not be trusted in an =20
    >> evaluation.

    >
    > Not so fast. If another process has been running that may well be a
    > process spawned by our process.


    Full ack that's right, if a process forks, like system() do you can =20
    measure
    that one too.

    Benchmark.bm(20) do |b|
    b.report 'fork' do
    pid =3D Process.fork { sleep 3 }
    Process.waitpid pid
    end
    end
    user system total real
    fork 0.000000 0.010000 0.020000 ( 3.008273)


    regards, Sandor Sz=FCcs
    --
    Sandor Szücs, Feb 25, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Dang Griffith

    Re: any benchmark results for python?

    Dang Griffith, Jun 25, 2003, in forum: Python
    Replies:
    0
    Views:
    1,271
    Dang Griffith
    Jun 25, 2003
  2. Rodrigo Daunaravicius

    interpreting hotshot results

    Rodrigo Daunaravicius, Jun 16, 2004, in forum: Python
    Replies:
    0
    Views:
    308
    Rodrigo Daunaravicius
    Jun 16, 2004
  3. Will Ware

    Interpreting python profiler results

    Will Ware, Mar 8, 2006, in forum: Python
    Replies:
    0
    Views:
    289
    Will Ware
    Mar 8, 2006
  4. Hans Mull

    Benchmark results unrealistic?

    Hans Mull, Feb 11, 2008, in forum: C++
    Replies:
    3
    Views:
    409
    James Kanze
    Feb 12, 2008
  5. Daniel Berger

    StringIO affecting Benchmark results

    Daniel Berger, Aug 25, 2004, in forum: Ruby
    Replies:
    0
    Views:
    145
    Daniel Berger
    Aug 25, 2004
Loading...

Share This Page