Floating point bug?

[Zentrader]

I disagree with this statement
<quote>But that doesn't change the fact that it will expose the same
rounding-errors as floats do - just for different numbers. </quote>
The example used has no rounding errors. For anything less that 28
significant digits it rounds to 1.0. With floats 1.0/3 yields
0.33333333333333331<-- on my machine. Also you can compare two
decimal.Decimal() objects for equality. With floats you have to test
for a difference less than some small value. BTW, a college professor
who also wrote code for a living made this offhand remark "In general
it is best to multiply first and then divide." Good general advice.

 

[Jeff Schwab]

I disagree with this statement
<quote>But that doesn't change the fact that it will expose the same
rounding-errors as floats do - just for different numbers. </quote>
The example used has no rounding errors.

I think we're using the term "rounding error" to mean different things.
If the value 1/3 is represented as a finite sequence of threes divided
by a power of ten (e.g. 0.3333333333333333333333333333), then it is
actually representing a different value than the original 1/3
(3333333333333333333333333333/10000000000000000000000000000). This is
what most of us mean by "rounding error." There are data types than can
represent rational numbers exactly, such that 1 / 3 * 3 = 1.

For anything less that 28
significant digits it rounds to 1.0.

I'm not sure what you mean here. Do you mean that Decimal can represent
any sequence of up to 28 decimal digits without any loss in precision?
No one disputes that.

With floats 1.0/3 yields
0.33333333333333331<-- on my machine.

Right, that is != 1/3. But if the last digit were a 3, the number still
would not be 1/3, only a different approximation of it.

Also you can compare two
decimal.Decimal() objects for equality. With floats you have to test
for a difference less than some small value.

A small margin of error (epsilon) is required for both Decimal and
floats. Both of these data types have "floating points;" they just use
different radixes. Particular sorts of rounding-errors have become
acceptable in base-10 arithmetic within certain (mainly financial)
contexts, so Decimal is preferable for calculations in those contexts.
For the same reason, a major U.S. stock exchange switched from base-2 to
base-10 arithmetic a few years ago. There's nothing inherently more
accurate about base-10.

BTW, a college professor
who also wrote code for a living made this offhand remark "In general
it is best to multiply first and then divide." Good general advice.

It is best to keep track of how many significant digits remain in
whatever base is being used. "M before D" may help preserve sig.
digits, but it is not a substitute for knowing how much error each
calculation introduces.

 

[Diez B. Roggisch]

Surely you jest. Your example is exact to 28 digits. Your attempted
trick is to use a number that never ends (1/3=0.3333...). It would
only convert back to one if you have and infinite number of
significant digits. That has nothing to do with the Python decimal
module (which does what it claims). It is one of the idiosyncrasies
of the base 10 number system. Remember we are working with base 10
decimals and not fractions.

The point is that all numbering systems with a base + precision will
have (rational) values they can't exactly represent. Q\R is of course
out of the question by definition....

And the poster I replied to said

"""
This is true. Fortunately Python does provide a module which allows
you to work with exact floating point quantities.
"""

Which is not more true for decimal as it is for IEEE754 floating points.
Just for other values.

Diez

 

[Mark Dickinson]

The point is that all numbering systems with a base + precision will
have (rational) values they can't exactly represent. Q\R is of course
out of the question by definition....

This 'Decimal is exact' myth has been appearing often enough that I
wonder whether it's worth devoting a prominent paragraph to in the
docs.

Mark

 

[Steven D'Aprano]

I disagree with this statement
<quote>But that doesn't change the fact that it will expose the same
rounding-errors as floats do - just for different numbers. </quote> The
example used has no rounding errors.

Of course it does. Three thirds equals one, not 0.9999, or
0.9999999999999999999999999999, or any other finite decimal
representation. That's a rounding error, regardless of whether you are
rounding in base 2 (like floats) or in base 10 (like Decimal does).

Also you can compare two
decimal.Decimal() objects for equality. With floats you have to test
for a difference less than some small value.

That's a superstition.

As a heuristic, it is often wise to lower your expectations when testing
for equality. Since you often don't know how much rounding error will be
introduced into a calculation, demanding that a calculation has zero
rounding error is often foolish.

But good advice becomes a superstition when it becomes treated as a law:
"never test floats for equality". That's simply not true. For example,
floats are exact for whole numbers, up to the limits of overflow. If you
know your floats are whole numbers, and still writing something like this:

x = 1234.0
y = 1000.0 + 200.0 + 30.0 + 4.0
if abs(x-y) < 1e-12:
print "x and y are equal"

then you are wasting your time and guilty of superstitious behaviour.

Similarly for fractional powers of two.

x = 1/2 + 1/4 + 1/8 + 1/16
y = 1.0 - 1/32 - 1/32
assert x == y

 

[Jeff Schwab]

Of course it does. Three thirds equals one, not 0.9999, or
0.9999999999999999999999999999, or any other finite decimal
representation. That's a rounding error, regardless of whether you are
rounding in base 2 (like floats) or in base 10 (like Decimal does).



That's a superstition.

As a heuristic, it is often wise to lower your expectations when testing
for equality. Since you often don't know how much rounding error will be
introduced into a calculation, demanding that a calculation has zero
rounding error is often foolish.

But good advice becomes a superstition when it becomes treated as a law:
"never test floats for equality". That's simply not true. For example,
floats are exact for whole numbers, up to the limits of overflow.

That's not true. Epsilon becomes large (in absolute terms) for large
numbers. For 64-bit IEEE floats, eps > 1 at about 10**16.

If you
know your floats are whole numbers, and still writing something like this:

x = 1234.0
y = 1000.0 + 200.0 + 30.0 + 4.0
if abs(x-y) < 1e-12:
print "x and y are equal"

then you are wasting your time and guilty of superstitious behaviour.

In what way? It's true that you hard-coded integers for which the
margin of error happens to be < 1, but there's nothing magical about
whole numbers. If you need to test for exact equality, you still can
only check within the limits of the float's precision.
False

 

[Steven D'Aprano]

That's not true. Epsilon becomes large (in absolute terms) for large
numbers. For 64-bit IEEE floats, eps > 1 at about 10**16.

Ah yes, good point. Sorry for my brain-fart, I was conflating integer
floats with the comp data type, as used in the Standard Apple Numerics
Environment (SANE). It used a floating point data type to store what was
effectively a 64-bit integer, which was quite significant in the days
when PCs used 16-bit integers!

Nevertheless, I stand by my claim, and support it by quoting from
Professor W Kahan of Berkley, who wrote in the forward to the Apple
Numerics Manual (2nd Edition):

.... because so many computers in the 1960's and 1970's possessed so many
different arithmetic anomalies, computational lore has become encumbered
with a vast body of superstition purporting to cope with them. One such
superstitious rule is "*Never* ask whether floating-point numbers are
exactly equal."
[end quote]

In practice, you often do want to test floating point values within some
tolerance. But it isn't a universal law: see also "What Every Computer
Scientist Should Know About Floating Point Arithmetic".

Incidentally, some people think that the solution to such anomalies is
never to compare floating-point numbers for equality, but instead to
consider them equal if they are within some error bound E. This is hardly
a cure-all because it raises as many questions as it answers.
[end quote]

http://docs.sun.com/source/806-3568/ncg_goldberg.html


Not directly related to this issue, but to get a great overview of some
of the problems with floating point, you could do a lot worse than to
read the following interview with Kahan:

http://www.ddj.com/184410314

and this wonderful lecture:

http://www.cs.berkeley.edu/~wkahan/ieee754status/ieee754.ps

where he details how optimizing compilers cause arithmetic errors, how
and why the facilities provided by IEEE arithmetic are underused, and
finally gives his idea of some things that could be done to turn the
situation around.

Sadly this lecture was given almost twelve years ago, and things have
barely changed. Compilers still do the wrong thing, especially optimizing
ones; computations that would be easy with NANs and infinities are
needlessly difficult; and benchmarks still over-value speed and under-
value getting the right answer, let alone simplicity of programming.

In what way? It's true that you hard-coded integers for which the
margin of error happens to be < 1,

No, in this case the error is *precisely* zero. There simply are no
rounding errors in this calculation, and the error is zero. If you wrote
the test as "if abs(x-y) < 1.0" you'd still be wasting your time.

It's true that I gave a hard-coded example, but it is hardly a special
case. There are many problem domains that don't require the full range of
floats and the type of rounding error you give can't occur.

(If your application calculates the amount of concrete needed to build a
house, say, then you pretty much know the amounts aren't going to be
measured in the trillions of megatonnes. If a user insists on specifying
that the house has 1.73e820 stories built on a base measuring 1.82e-87
metres squared, then roundoff is the least of your problems: the
algorithms you are applying will no longer be valid.)

Or possible you have already rounded the numbers yourself, earlier:
True

Why would you do this instead?
True


The important thing is that your numbers have appropriately similar
scales. If you know that it going to be the case, then you know that
addition won't cause the sort of round-off error. I'm talking about.
Naturally there may be other forms of round-off, but rounding error
doesn't just appear from nowhere, it is predictable and understandable.

 

[Lie]

Would all these problems with floating points be a rational reason to
add rational numbers support in Python or Py3k? (pun not intended)

I agree, there are some numbers that is rationals can't represent
(like pi, phi, e) but these rounding problems also exist in floating
points, and rational numbers wouldn't be so easily fooled by something
like 1 / 3 * 3, and 1/10 (computer) is exactly 0.1 (human). The first
problem with rational is that to get an infinite precision rational,
the language would have to have an infinite length integers, which
Python have given us. The second problem with rationals is to keep
rationals in its most simple, trivial form. This can be solved with a
good GCD algorithm, which can also be a nice addition to Python's math
library.

 

[Jeff Schwab]

Would all these problems with floating points be a rational reason to
add rational numbers support in Python or Py3k? (pun not intended)

I agree, there are some numbers that is rationals can't represent
(like pi, phi, e) but these rounding problems also exist in floating
points, and rational numbers wouldn't be so easily fooled by something
like 1 / 3 * 3, and 1/10 (computer) is exactly 0.1 (human). The first
problem with rational is that to get an infinite precision rational,
the language would have to have an infinite length integers, which
Python have given us. The second problem with rationals is to keep
rationals in its most simple, trivial form. This can be solved with a
good GCD algorithm, which can also be a nice addition to Python's math
library.

http://www.python.org/dev/peps/pep-0239/

 

[Lie]

http://www.python.org/dev/peps/pep-0239/

Yes, I'm aware of the PEP and actually have been trying for some time
to reopen the PEP.

The reason that PEP is rejected is because Decimal is accepted, which
I think is a completely absurd reason as Decimal doesn't actually
solve the rounding problems and equality comparison problem. Some
people have also pointed out that Decimal IS Inexact, while a rational
number is always exact except if you have an operation with a (binary
or decimal) floating point involved (this can be easilty resolved by
making fraction recessive, i.e. an operation that receive a fraction
and a float should return a float).

 

[Mark Dickinson]

Would all these problems with floating points be a rational reason to
add rational numbers support in Python or Py3k? (pun not intended)

It's already in the trunk! Python will have a rational type (called
Fraction) in Python 2.6 and Python 3.0, thanks largely to the work of
Jeffrey Yaskin.

Mark

 

[Mark Dickinson]

Would all these problems with floating points be a rational reason to
add rational numbers support in Python or Py3k? (pun not intended)

Forgot to give the link:

http://docs.python.org/dev/library/fractions.html

Mark

 

[Lie]

It's already in the trunk!  Python will have a rational type (called
Fraction) in Python 2.6 and Python 3.0, thanks largely to the work of
Jeffrey Yaskin.

Mark

Really? I didn't know that (last time I checked they frowned at me for
asking that). It's going to be be great if Python supports fractional
number. Although rationals have its limitations too, it is a much
better choice compared to floats/Decimals for most cases. One thing
though, the time I've spent on creating a rational class myself would
be 'somewhat' wasted, although I don't really mind that since it was
fun in any case.

 

[mensanator]

Would all these problems with floating points be a rational reason to
add rational numbers support in Python or Py3k? (pun not intended)

I agree, there are some numbers that is rationals can't represent
(like pi, phi, e) but these rounding problems also exist in floating
points, and rational numbers wouldn't be so easily fooled by something
like 1 / 3 * 3, and 1/10 (computer) is exactly 0.1 (human). The first
problem with rational is that to get an infinite precision rational,
the language would have to have an infinite length integers, which
Python have given us. The second problem with rationals is to keep
rationals in its most simple, trivial form. This can be solved with a
good GCD algorithm, which can also be a nice addition to Python's math
library.

Have you looked at the gmpy madule? That's what I
use whenever this comes up. Works very nice to
eliminate the issues that prevent a float olution
for the problems I'm working on.

And some irrationals can be represented by infite
sequences of rationals that, coupled with gmpy's
unlimited precision floats, allows any number of
accurate decimal places to be calculated.

If you would like to see an example, check out

http://members.aol.com/mensanator/polynomial.py

 

[Dennis Lee Bieber]

Would all these problems with floating points be a rational reason to
add rational numbers support in Python or Py3k? (pun not intended)

Not to me, at least...

...at least, not without adding a full symbolic algebra package to
use them (a la the HP48, 49, 50 series calculators) <G>

Epsilon comparison of floats has never been a concern for me,
compounded with my high school chemistry/physics classes which enforced
the rule that one never computes results with more significance than
went into the equation (which, presumably, were supplied by reading
various physical measuring devices, in which the least significant digit
is already a +/- range: 0.5 on an instrument graduated in 10ths meaning
a value between 0.45 and 0.55). So 1/3 => 0.3...

Granted, a math theoretician wouldn't be happy...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Carl Banks]

Although rationals have its limitations too, it is a much
better choice compared to floats/Decimals for most cases.

Maybe that's true for your use cases, but it's not true for most cases
in general.

Rationals are pretty useless for almost any extended calculations,
since the denominator tends to grow in size till it's practically
unusbale, which means you have to periodically do non-exact reductions
to keep things running, and if you do that you might as well be using
floating point.

Rationals have their occasional special purpose uses, but for most
cases they're at best marginally better then floats and more often
incomparably worse.


Carl Banks

 

[Paul Rubin]

Rationals are pretty useless for almost any extended calculations,
since the denominator tends to grow in size till it's practically
unusbale, which means you have to periodically do non-exact reductions
to keep things running, and if you do that you might as well be using
floating point.

This only happens if you're chaining many divisions by non-constants,
which in the typical business calculation does not happen. Those
calculations divide by 10 a lot, and sometimes by 12 or 365, but they
rarely divide a calculated quantity by another calculated quantity,
especially to more than one level. I.e. they might compute the ratio
of widgets sold to widgets repaired, but they wouldn't then divide
that ratio by some other weird number.

I did in fact implement rational arithmetic in a business app a long
time ago, because management wanted it. I had the same concerns that
you express, but they just weren't a problem in practice.

 

[Lie]

Maybe that's true for your use cases, but it's not true for most cases
in general.

OK, that might have been an overstatement, but as I see it, it is
safer to handle something in a Fraction compared to floats (which is
why I uses fractions whenever possible in non-computer maths).

Rationals are pretty useless for almost any extended calculations,
since the denominator tends to grow in size till it's practically
unusbale, which means you have to periodically do non-exact reductions
to keep things running, and if you do that you might as well be using
floating point.

Rationals aren't that good if the same piece of variable is to be
calculated again and again because of its growth, but there are a lot
of cases where the data would only require five or six or so
operations done on it (and there are thousands or millions of such
datas), rationals is the perfect choice for those situations because
it is easier to use thanks to the comparison safety. Or in the
situations where speed isn't as important and accuracy is required,
Fraction may be faster than decimal and more accurate at the same time
(someone need to test that though).

 

[Jeff Schwab]

Maybe that's true for your use cases, but it's not true for most cases
in general.

Rationals are pretty useless for almost any extended calculations,
since the denominator tends to grow in size till it's practically
unusbale,

What do you mean by "practically unusable?" I heard similar arguments
made against big integers at one point ("Primitive types are usually big
enough, why risk performance?") but I fell in love with them when I
first saw them in Smalltalk, and I'm glad Python supports them natively.

 

[Carl Banks]

What do you mean by "practically unusable?" I heard similar arguments
made against big integers at one point ("Primitive types are usually big
enough, why risk performance?") but I fell in love with them when I
first saw them in Smalltalk, and I'm glad Python supports them natively.