Newb math question

[brian]

Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996

I'm expecting 0.01 as the result. Do I have to specify the precision?
Even with the full precision available on the platform (Windows XP), the
correct answer is still 0.01, not what's printed. What's the deal? I
like this language so far, but this is confusing...

Brian

 

[ES]

Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996

I'm expecting 0.01 as the result. Do I have to specify the precision?
Even with the full precision available on the platform (Windows XP), the
correct answer is still 0.01, not what's printed. What's the deal? I
like this language so far, but this is confusing...

Floating point math :/ You can use BigDecimal to avoid most problems.

Brian

E

 

[Bill Kelly]

Hi Brian,

Floating point math :/ You can use BigDecimal to avoid most problems.

If you'd like more info about the problem, the first
paragraphs of the section entitled Rounding Error in
this document give the gist of it:
http://docs.sun.com/source/806-3568/ncg_goldberg.html



Regards,

Bill

 

[brian]

Thanks, I'll give it a read. Thanks to Bill too. I have to say this is
one of the friendlier communities I've come across. The BigDecimal is
working out. Newb's not getting burned with RTFM's, tends to keep them
coming back for more. Thanks again!

 

[Jacob Fugal]

Can someone tell me why this code:
puts (9.0-8.9).to_s
results in:
0.0999999999999996
=20
I'm expecting 0.01 as the result. Do I have to specify the precision?
Even with the full precision available on the platform (Windows XP), the
correct answer is still 0.01, not what's printed. What's the deal? I
like this language so far, but this is confusing...
=20
Brian

Along with all the other good responses, I thought I'd just take the
moment to (once again) state that this behaviour isn't unique to ruby:

ruby: "%.16f" % (9.0-8.9) =3D=3D> 0.0999999999999996
perl: sprintf("%.16f", 9.0-8.9) =3D=3D> 0.0999999999999996
python: "%.16f" % (9.0-8.9) =3D=3D> 0.0999999999999996
php: sprintf("%.16f", 9.0-8.9) =3D=3D> 0.0999999999999996
C: sprintf(dest, "%.16f", 9.0-8.9) =3D=3D> 0.0999999999999996

In all of the cases, specifying a lower precision allows the
float->string conversion to round to the "expected" result:

ruby: "%.10f" % (9.0-8.9) =3D=3D> 0.1000000000
perl: sprintf("%.10f", 9.0-8.9) =3D=3D> 0.1000000000
python: "%.10f" % (9.0-8.9) =3D=3D> 0.1000000000
php: sprintf("%.10f", 9.0-8.9) =3D=3D> 0.1000000000
C: sprintf(dest, "%.10f", 9.0-8.9) =3D=3D> 0.1000000000

(The source for round_error_c_2 is identical to round_error_c but with
only 10 instead of 16 digits of precision in the printf.)

The major difference is the default precision used by languages in the
float->string conversion. From experimentation (not actually looking
at sources), I came to the following conclusions on the default
precision:

* ruby and perl both use up to 15 significant digits
* python uses up to 12 significant digits
* php uses up to 14 significant digits
* C uses 6 places after the decimal, always
(with %f instead of %.10f in the printf)

For all of ruby, perl, python and php I say "up to X significant
digits" because significant 0's at the end are truncated (e.g. (1.0 -
0.5).to_s returns '0.5', not '0.500000000000000') and significant
digit rules are obeyed (significant digits before the decimal point
reduce the number of digits after the decimal point, leading zeros
after the decimal point are not counted).

Jacob Fugal