floating point problem... floating indeed :(

Discussion in 'Ruby' started by teeshift, Dec 1, 2006.

  1. teeshift

    teeshift Guest

    I recently wrote a code and haven't been found the bug and start
    suspecting that it was
    the floating computation that causing the problem. I did a little
    experiement and found the
    following phenomena:

    I am using

    Linux source 2.6.15-27-386 #1 PREEMPT Sat Sep 16 01:51:59 UTC 2006 i686
    GNU/Linux
    Ubuntu is the OS.

    #####################
    #!/usr/bin/ruby1.9

    a = 1.0

    a.to_f

    while(a < 15.0)
    a = a + 0.1
    print a,"\t",a * a,"\n"
    end
    #####################

    and the output is...(as following) quite odd. Anyone can explain it?
    or a way to avoid it (while
    keeping the precision)


    1.1 1.21
    1.2 1.44
    1.3 1.69
    1.4 1.96
    1.5 2.25
    1.6 2.56
    1.7 2.89
    1.8 3.24
    1.9 3.61
    2.0 4.0
    2.1 4.41
    2.2 4.84
    2.3 5.29000000000001
    2.4 5.76000000000001
    2.5 6.25000000000001
    2.6 6.76000000000001
    2.7 7.29000000000001
    2.8 7.84000000000001
    2.9 8.41000000000001
    3.0 9.00000000000001
    3.1 9.61000000000001
    3.2 10.24
    3.3 10.89
    3.4 11.56
    3.5 12.25
    3.6 12.96
    3.7 13.69
    3.8 14.44
    3.9 15.21
    4.0 16.0
    4.1 16.81
    4.2 17.64
    4.3 18.49
    4.4 19.36
    4.5 20.25
    4.6 21.16
    4.7 22.09
    4.8 23.04
    4.9 24.01
    5.0 25.0
    5.1 26.01
    5.2 27.04
    5.3 28.09
    5.4 29.16
    5.5 30.25
    5.6 31.36
    5.7 32.49
    5.8 33.64
    5.9 34.81
    6.0 35.9999999999999
    6.1 37.2099999999999
    6.19999999999999 38.4399999999999
    6.29999999999999 39.6899999999999
    6.39999999999999 40.9599999999999
    6.49999999999999 42.2499999999999
    6.59999999999999 43.5599999999999
     
    teeshift, Dec 1, 2006
    #1
    1. Advertisements

  2. See this writeup for some details on why this happens.
    http://en.wikipedia.org/wiki/IEEE_754
     
    Wilson Bilkovich, Dec 1, 2006
    #2
    1. Advertisements

  3. teeshift

    Chris Pearl Guest

    Someone else might wish to give a more concrete explanation, but as a
    rule, floating-point calculations - using the usual Float datatype -
    are not accurate, and prone to rounding errors such as the above.

    If you're dealing with limited precision (i.e. X digits after the
    decimal point) you should consider using the Decimal datatype instead.
     
    Chris Pearl, Dec 1, 2006
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.