Questions about benchmark and measuring performance

[adamomitcheney]

Hi All,

I have a few questions about the benchmark module and performance
testing in general - my apologies in advance because the questions may
be outside the scope of this group. What I'm actually trying to do is
to come up with an objective measure of the performance of an
application (ClearCase) so that we can play with a variety of
parameters and get an idea of the relative value of different changes.

My script is very simple, really - it sets up the test environment
(creates a couple of views) then runs a subroutine that executes a few
ClearCase commands (mostly getting information) using backticks. I'm
simply using timethis() to measure a hundred million runs of this
subroutine. Finally it cleans up after itself.

I've just run it a few times. First time I got (what looked like) a
meaningful time, next two I got negative times and a warning that I
wasn't using enough iterations - these were identical runs, performed
within moments of each other.

My questions:
Is this giving me what I hope - that is, an indication of the time that
it takes to execute the *ClearCase* commands?
If not, is there another method I could use?

Thanks - Adam...

 

[Brian Wakem]

My questions:
Is this giving me what I hope - that is, an indication of the time that
it takes to execute the *ClearCase* commands?
If not, is there another method I could use?

Thanks - Adam...

A simple gettimeofday (Time::HiRes) before and after will give you the time
taken.

 

[xhoster]

My script is very simple, really - it sets up the test environment
(creates a couple of views) then runs a subroutine that executes a few
ClearCase commands (mostly getting information) using backticks. I'm
simply using timethis() to measure a hundred million runs of this
subroutine. Finally it cleans up after itself.

It would help if you show us the relevant part of the script. For example,
what is your first argument to "timethis"?

I've just run it a few times. First time I got (what looked like) a
meaningful time, next two I got negative times and a warning that I
wasn't using enough iterations - these were identical runs, performed
within moments of each other.

My questions:
Is this giving me what I hope - that is, an indication of the time that
it takes to execute the *ClearCase* commands?

No.

from "perldoc Benchmark":

timethis ( COUNT, CODE, [ TITLE, [ STYLE ]] )
***
CPU seconds is, in UNIX terms, the user time plus the
system time of the process itself, as opposed to the real
(wall- clock) time and the time spent by the child
processes. ***

So, you are using child processes, and their CPU times are not being
properly counted.

If not, is there another method I could use?

There are any number of other methods you could use. For example, you
could specify the number of iterations to "timethis", and then look at the
walltime it returns (as opposed to the CPU time).

[~/perl_misc]$ perl -le 'use Benchmark; timethis(10, sub {system "sleep
1"})'

timethis 10: 10 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.01 csys
= 0.01 CPU)

(warning: too few iterations for a reliable count)

Xho