Portable general timestamp format, not 2038-limited

[John W. Kennedy]

GPS time is UTC time

No it isn't. GPS has never introduced a leap second, and is still on
uncorrected UTC-as-of-1980. However, the GPS signal also includes an
occasional UTC correction figure, so it can be used to obtain UTC.

 

[Scott David Daniels]

Stick to unix timestamps but store them as a double precision floating
point number. The 53 bit mantissa gives you currently a resolution of
about 200 ns, slowly deteriorating (you will hit ms resolution in about
280,000 years, if I haven't miscalculated). Any language and database
should be able to handle double-precision FP numbers, so that's as
portable as it gets and conversion from/to system time should be
trivial.

If you need to represent milliseconds exactly, you can just multiply the
timestamp with 1000 (and get java timestamps).

hp

TOPS-20 did an interesting format which suggest an interesting variant:
Tops-20: 36-bit (the machine word size) fixed-bit representation
of days since a given moment (the first Photographic
plates of the sky). The "binary point" was at the middle
of the word; the low order 18 bits were the time of day
(GMT), the high-order 18 bits were the days-since date.

Inspired format:
Days since a some standard date (the TAI date may be a good such
date) expressed as fixed point 64-bit (32-bit day part, 32-bit
day-fraction part) or floating point (using Intel's double-precision,
for example, gives you 26 bits each for day and day-fraction, though
the binary point moves for particular stamps).

Advantages of such a format:
Using simple arithmetic for the difference between two such stamps
is reasonably accurate even without knowing about when the leap seconds
occur. Better resolution is available with leap-second aware software.
A leap second affects the resolution only in intervals where there
_are_ leap seconds, and ignoring them leaves you almost 5 digits of
accuracy even when you naively ignore them.

Disadvantages:
time-of-day is not simple (but I maintain it shouldn't be).
No external way to know if a stamp is leap-second aware or not;
you'll just have to know for a whole group.
Once you have done a naive difference, there is no way to correct it.


--Scott David Daniels
(e-mail address removed)

 

[Peter J. Holzer]

I have a requirement to store timestamps in a database. Simple enough
you might think but finding a suitably general format is not easy. The
specifics are

[...]
Stick to unix timestamps but store them as a double precision floating
point number. The 53 bit mantissa gives you currently a resolution of
about 200 ns, slowly deteriorating (you will hit ms resolution in about
280,000 years, if I haven't miscalculated). Any language and database
should be able to handle double-precision FP numbers, so that's as
portable as it gets and conversion from/to system time should be
trivial.

TOPS-20 did an interesting format which suggest an interesting variant:
Tops-20: 36-bit (the machine word size) fixed-bit representation
of days since a given moment (the first Photographic
plates of the sky). The "binary point" was at the middle
of the word; the low order 18 bits were the time of day
(GMT), the high-order 18 bits were the days-since date.

Inspired format:
Days since a some standard date (the TAI date may be a good such
date) expressed as fixed point 64-bit (32-bit day part, 32-bit
day-fraction part) or floating point (using Intel's double-precision,
for example, gives you 26 bits each for day and day-fraction, though
the binary point moves for particular stamps).

Doesn't MS-Excel store timestamps in such a format?

This requires you to define what a "day" is:

a) 86400 seconds

b) the time between two consecutive readings of 00:00:00 on a UTC clock

c) something else.

Definition b) is probably the most useful.

Advantages of such a format:
Using simple arithmetic for the difference between two such stamps
is reasonably accurate even without knowing about when the leap seconds
occur. Better resolution is available with leap-second aware software.
A leap second affects the resolution only in intervals where there
_are_ leap seconds, and ignoring them leaves you almost 5 digits of
accuracy even when you naively ignore them.

Since a day with a leap second has 86401 seconds (or 86399, but that
hasn't happened yet), a leap second aware counter could record the time
HH:MM:SS on such a day as (HH*3600+MM*60+SS)/86401. If you know that
there was a leap second on that day you can still recover the exact time
wall clock time, otherwise you will be off by up to one second, but the
time is still monotonic and you don't have a sudden jump at the end of
the day.

hp

 

[Paul Rubin]

As for the primacy of UTC vs. TAI, this is the classical chicken and
egg problem. The bureaucratic reality is opposed to the physical
reality.

Well, if you're trying to pick just one timestamp standard, I'd say
you're better off using a worldwide one rather than a national one, no
matter how the bureaucracies work. TAI is derived from atomic clocks
all over the world, while the national metrology labs are more subject
to error and desynchronization, and whatever legal primacy they have
is good in only one country.

 

[Richard Heathfield]

Paul Rubin said:

Well, if you're trying to pick just one timestamp standard, I'd say
you're better off using a worldwide one rather than a national one, no
matter how the bureaucracies work.

In that case, the obvious choice is Greenwich Mean Time.

Seriously, GMT is recognised all over the world (far more so, in fact,
than UTC, which tends to be recognised only by some well-educated
people, and there are precious few of those), so why not use it?

I always leave my PC's clock set to GMT, partly out of this desire to
support a single timestamp standard, and (it must be said) partly out
of general cussedness.

 

[Martin Gregorie]

I think there's a definite practical advantage in storing dates as a day
count from a base date and providing a standard set of
procedures/methods to convert it to and from a human-readable format. It
makes all sorts of date calculations much easier. For instance, if
there's a requirement to produce statements dated for the last day of
the current month the pseudo code is simply:
- convert the date to ccyymmdd format
- add 1 to the month, adjusting the year and century to fix
year roll-over and set the day to 1
- convert back to day count and subtract 1
- the result, output in readable form is the last day of the month
irrespective of month length, leap years, etc.

I don';t think it matters what the base date is, though the Astronomical
base date [12 noon on 1 JAN -4712 (4713 BC)] may be as good as any.
Other bases I've seen (apart from UNIX date) are ICL 1900 mainframes,
which set day zero as 31 Dec 1899 and held the time separately. ICL 2900
systems held the date and time as microseconds since 00:00:00 1900-01-01
in a 64 bit word, which is also easy to deal with and allows the same
set of date arithmetic operations as a straight day number.

BTW, be sure to distinguish Julian Day and Modified Julian Day (used by
astronomers from the "Julian Date" that [used to] be used by IBM
mainframes. The former is a day count but the latter is day within year
(yyddd). JD and MJD are described in:

http://tycho.usno.navy.mil/mjd.html

 

[Peter J. Holzer]

Paul Rubin said:

In that case, the obvious choice is Greenwich Mean Time.

Hardly. That hasn't been in use for over 35 years (according to
Wikipedia).

Seriously, GMT is recognised all over the world (far more so, in fact,
than UTC, which tends to be recognised only by some well-educated
people, and there are precious few of those), so why not use it?

While the layman may recognize the term "GMT", he almost certainly means
"UTC" when he's talking about GMT. GMT was based on astronomical
observations and the be best approximation available today is probably
UT1, which may differ from UTC by up to 0.5 seconds.

I always leave my PC's clock set to GMT,

Your PC is directly linked to an observatory? Impressive . If you
synchronize your PC to any external time source, it's almost certainly
UTC, not GMT or UT1. If you don't synchronize it it's so far off that it
doesn't matter.

hp

 

[Richard Heathfield]

Peter J. Holzer said:

Hardly. That hasn't been in use for over 35 years (according to
Wikipedia).

Nonsense. I use it every day, and have been doing so for - well, rather
more than 35 years.

While the layman may recognize the term "GMT", he almost certainly
means "UTC" when he's talking about GMT.

Most people of my acquaintance who use the term "GMT" mean precisely
that - Greenwich Mean Time.

Your PC is directly linked to an observatory?

Nope. My PC *defines* GMT. If the observatory wants to know what the
exact time is, they only have to ask.

 

[sla29970]

Well, if you're trying to pick just one timestamp standard, I'd say
you're better off using a worldwide one rather than a national one, no
matter how the bureaucracies work. TAI is derived from atomic clocks
all over the world, while the national metrology labs are more subject
to error and desynchronization, and whatever legal primacy they have
is good in only one country.

For the purposes of an operational system there is an important
difference between a time scale which is practically available in real
time and a time scale which is not available until next month. There
is no available source for TAI, and in the current scheme of things
there cannot be one for there is no mechanism for distributing it.

There are two reasonably reliable worldwide time sources right now:
Russia's GLONASS and US GPS. GPS time is based on UTC(USNO).
UTC(USNO) is TA(USNO) minus leap seconds. Note that is TA(USNO), not
TAI(USNO), for USNO cannot define anything named TAI.

 

[CBFalconer]

.... snip ...


Hardly. That hasn't been in use for over 35 years (according to
Wikipedia).

I am glad to see you depend on absolutely reliable sources.

 

[Dr.Ruud]

Peter J. Holzer schreef:

Since a day with a leap second has 86401 seconds (or 86399, but that
hasn't happened yet)

Many systems allow a seconds value of 0..61, so minutes (actually
months) with two leap seconds are foreseen.

A leap second may be introduced at the end of any month, the preferred
dates are at the end of June and the end of December.

At the estimated rate of decrease, the earth would lose about 1/2 day
after 4,000 years, and about two leap seconds a
month would be needed to keep UTC in step with Earth time, UT1.

(source:
<URL:http://www.allanstime.com/Publications/DWA/Science_Timekeeping/TheS
cienceOfTimekeeping.pdf>)

 

[Peter J. Holzer]

I am glad to see you depend on absolutely reliable sources.

Mostly I relied on my own memory (which is of course even less reliable
than Wikipedia). I checked Wikipedia for the date (Jan 1st 1972) when
GMT was replaced by UTC as the basis for civil time. Since that
coincided with my own recollection (sometime in the 1970's), I see no
reason to doubt it.

It is possible that the observatory at Greenwich still keeps and
announces GMT, but it has no practical importance anymore. Certainly
what everybody (except specialists in the field) means when they talk
about "GMT" is UTC.

hp

 

[Peter J. Holzer]

Peter J. Holzer schreef:

Many systems allow a seconds value of 0..61, so minutes (actually
months) with two leap seconds are foreseen.

That comes from the ANSI C standard. It is unclear why the standard
specifies 0..61 instead of 0..60. The most plausible explanation I've
read is that it's the result of a misunderstanding: Up to two leap
seconds in a year are expected, and somebody thought they would be
applied both at the end of the year (instead of one at the end of each
semester).

A leap second may be introduced at the end of any month, the preferred
dates are at the end of June and the end of December.

At the estimated rate of decrease, the earth would lose about 1/2 day
after 4,000 years, and about two leap seconds a
month would be needed to keep UTC in step with Earth time, UT1.

C is already ready for this, although I doubt that it's authors planned
that far ahead.

hp

 

[Richard Heathfield]

Peter J. Holzer said:

It is possible that the observatory at Greenwich still keeps and
announces GMT, but it has no practical importance anymore. Certainly
what everybody (except specialists in the field) means when they talk
about "GMT" is UTC.

I am not a specialist in the field. When I talk about GMT, I mean GMT,
not UTC. Therefore, I am a counter-example to your claim.

 

[Ben Finney]

I am glad to see you depend on absolutely reliable sources.

Wikipedia is not an absolutely reliable source. I know of no
"absolutely resliable source". We work with imperfect human-provided
data all the time.

 

[James Harris]

Stick to unix timestamps but store them as a double precision floating
point number. The 53 bit mantissa gives you currently a resolution of
about 200 ns, slowly deteriorating (you will hit ms resolution in about
280,000 years, if I haven't miscalculated). Any language and database
should be able to handle double-precision FP numbers, so that's as
portable as it gets and conversion from/to system time should be
trivial.

If you need to represent milliseconds exactly, you can just multiply the
timestamp with 1000 (and get java timestamps).

Interesting option. I think my choice is between separate day and sub-
day 32-bit unsigned integers, text, and this 64-bit float option.

I'm not clear, though. Did you mean to store double precision numbers
where the seconds are the units (I assume this) or where the days are
the units? And what do you think of the other option?

 

[James Harris]

Inspired format:
Days since a some standard date (the TAI date may be a good such
date) expressed as fixed point 64-bit (32-bit day part, 32-bit
day-fraction part) or floating point (using Intel's double-precision,
for example, gives you 26 bits each for day and day-fraction, though
the binary point moves for particular stamps).

This is close to or the same as my original suggestion. The break
between days and sub-days seems to make more sense than breaking the
fractional part elsewhere. It also gives a convenient point to hang
datestamps on rather than just timestamps.

FWIW I wonder if a 64-bit version of the above would cope with all
practical time needs. With that the time would range to +/- 9000
quintillion years (18 digits) and there would be over 200 trillion
ticks per second or 200 in a picosecond making, I think, each tick 5
femtoseconds.

 

[Peter J. Holzer]

Interesting option. I think my choice is between separate day and sub-
day 32-bit unsigned integers, text, and this 64-bit float option.

I'm not clear, though. Did you mean to store double precision numbers
where the seconds are the units (I assume this) or where the days are
the units? And what do you think of the other option?

I was thinking about using the seconds as units (so
2007-07-04T23:02:04.123 CET is represented as 1183582924.123).
It's a natural extension of the unix time stamp, so you can often just
pass the values to the normal date routines (especially in languages
like perl which don't really distinguish between integers and floating
point numbers).

But it really doesn't matter much. If you ignore leap seconds, using
days instead of seconds is just a constant factor (in fact, the unix
timestamp ignores leap seconds, too, so it's always a constant factor).
You can't represent a second exactly if the unit is one day (1/86400 is
not a multiple of a power of two), but that probably doesn't matter.


hp

 

[James Harris]

But it really doesn't matter much. If you ignore leap seconds, using
days instead of seconds is just a constant factor (in fact, the unix
timestamp ignores leap seconds, too, so it's always a constant factor).
You can't represent a second exactly if the unit is one day (1/86400 is
not a multiple of a power of two), but that probably doesn't matter.

Sure. However, the proposal was to define ticks as 25 microseconds.

 

[greg]

With that the time would range to +/- 9000
quintillion years (18 digits)

Use the Big Bang as the epoch, and you won't have
to worry about negative timestamps.