a timer for benchmarking

[Dan Christensen]

I've been doing some benchmarking of time critical code using Python.
I'd like to use a timer with reasonable accuracy and high precision,
and would love something that combines the best aspects of time.time
and time.clock.

(I'm running Debian Linux on various Intel P4's, but am also interested
in solutions which work on other OS's and other hardware (Opterons
and Alpha's in particular).)

Here's my understanding of time.time and time.clock:

time.time returns the current "human" time represented in seconds
since a certain date and time, usually Jan 1, 1970. On my system it
has high resolution, but the documentation says that on others it has
low resolution. The problem I have with using this for benchmarking
is that if the human form of the date/time is changed, the measurement
of elapsed time using this function is incorrect. For example, on my
laptop, the ntp daemon changes the time in small steps every once in a
while, and this has resulting in confusion and inaccurate benchmarks
until I tracked this down.

Under Linux/Unix, time.clock returns the cpu time used by the current
process, but has low resolution (1/100 of a second on my system).
If the human form of the date/time is changed, this function still
gives correct results.

Under Windows, the documentation states that time.clock returns
elapsed time with high resolution. I don't know whether this
elapsed time is affected by changes to the human date/time.

I guess I'm wondering whether the Python developers have considered
providing a more systematic and platform independent set of functions
for accessing time. For example, it would be nice to have access to
the following choices:

1) cpu time used by the current process (e.g. time.clock under Unix)
"time.cpu"?
2) time as humans care about it (e.g. time.time under Unix/Windows)
"time.time"
3) time with respect to some arbitrary earlier point, with as
high resolution as possible (maybe time.clock under Windows?)
"time.elapsed" or "time.benchmark"?

[time.clock would be deprecated because of its inconsistent semantics]

For benchmarking, one could use time.cpu for longer jobs where you
want to ignore the time taken by other jobs competing for the cpu.
And one could use time.elapsed for short tests where the resolution is
important (a common case for me).

Of course, if the OS doesn't provide the necessary information,
then python will have to fall back to another method or raise
an exception. But I think Linux and many OS's can provide all
three on most hardware.

For example, a Pentium-class cpu can provide very high resolution time
measurements through the time-stamp counter which counts the number of
instructions since boot. For example, this code excerpt

#include <stdio.h>
#include <asm/msr.h>

unsigned long long t, prev;
rdtscll(prev);
rdtscll(t);
printf("%lld %lld\n", t, t-prev);

produces the output "84" on a 2.0GHz machine, which basically says
that the rdtscll code itself takes about 42 billionths of a second.
Pretty good resolution! And unaffected by changes to the human form
of the system clock. If Python can find out the clock speed, then
this seems like a possible way to implement case 3) above. And
experts (which I am not) can probably come up with other ways.

I've read some earlier discussions about timing in this group, and
while speed comparisons are a popular topic (!), I didn't see these
points addressed, so I'm wondering what people think. This kind of
thing seems like a nice feature to have...

Dan

 

[Tristan Seligmann]

Pretty good resolution! And unaffected by changes to the human form
of the system clock. If Python can find out the clock speed, then

It IS, however, affected by changes to the CPU clock speed. Such changes
are quite frequent due to various power saving strategies on laptops,
etc.
--
mithrandi, i Ainil en-Balandor, a faer Ambar

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA4FqMpNuXDQIV94oRAnnmAJ9ekwB3hC1VYc+ZauM/l9mmYUe5vACdHpQo
i4NnWrGtagZFbPT73HwZSPg=
=Tm3q
-----END PGP SIGNATURE-----

 

[Dan Christensen]

I guess I'm wondering whether the Python developers have considered
providing a more systematic and platform independent set of functions
for accessing time. For example, it would be nice to have access to
the following choices:

1) cpu time used by the current process (e.g. time.clock under Unix)
"time.cpu"?
2) time as humans care about it (e.g. time.time under Unix/Windows)
"time.time"
3) time with respect to some arbitrary earlier point, with as
high resolution as possible (maybe time.clock under Windows?)
"time.elapsed" or "time.benchmark"?
....

For example, a Pentium-class cpu can provide very high resolution time
measurements through the time-stamp counter which counts the number of
instructions since boot.
....

Pretty good resolution! And unaffected by changes to the human form
of the system clock. If Python can find out the clock speed, then

It IS, however, affected by changes to the CPU clock speed. Such changes
are quite frequent due to various power saving strategies on laptops,
etc.

That's a good point. So the time-stamp counter doesn't work for 3),
but works for 4):

4) cpu cycles since some arbitrary earlier point.

And, now that I think about it, this might be better than 3) for
benchmarking precisely because of possible cpu clock speed changes.
The number of cycles shouldn't be affected by cpu clock speed changes
as much as the elapsed time would.

Of course, for any serious benchmarking, one should disable variable
cpu clock speed in the BIOS, and then a high resolution measurement of
elapsed time would be nice to have. The time-stamp counter was just
one possible method; are there alternatives available (in Linux on a
Pentium, say) that aren't affected by ntp and other changes to the
system clock?

Or maybe I should just do several runs and take the *second* best
time.time interval (instead of the best, as is often recommended)?

What about the issue of portability across OS's, and the fact that
time.clock isn't consistent. Is this regarded as an issue?

Dan