what's wrong with these calculations?

[Ivo Dancet]

Hi

I thought, why search for two hours being totally frustrated if I can
just ask...

why does this happen:
=> 9.06999999999999 # why oh why?

What should I do about it?

Thanks
-c

 

[Ivo Dancet]

Ok

I found this thread which explains this problem: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/9654

So, now I understand why this happens, but I still would like to get
the proper result. How can I achieve that?

Thanks
-c

Op 5-dec-07, om 12:30 heeft Ivo Dancet het volgende geschreven:

 

[hemant]

Hi

I thought, why search for two hours being totally frustrated if I can
just ask...

why does this happen:

=> 9.06999999999999 # why oh why?

Thats folly of floats, you can't do much apart from :

sprintf("%.2f",(73.07-64.00)).to_f



--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

 

[hemant]

Thats folly of floats, you can't do much apart from :

sprintf("%.2f",(73.07-64.00)).to_f

generally, I add a method to Float class like this:

class Float
def r2p places
sprintf("%.#{places}f",self).to_f
end
end

 

[Jason Roelofs]

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

generally, I add a method to Float class like this:

class Float
def r2p places
sprintf("%.#{places}f",self).to_f
end
end

Also look into using the BigDecimal class. It does special processing to
keep floating point precision.

Jason

 

[Phrogz]

generally, I add a method to Float class like this:

class Float
def r2p places
sprintf("%.#{places}f",self).to_f
end
end

Note that, due to the nature of floats, calling to_f on that string
brings you back into the problem again:

x = ( 73.07 - 64.00 )
y = x.r2p( 2 )
p x, y, "%.20f" % y

#=> 9.06999999999999
#=> 9.07
#=> "9.07000000000000028422"

 

[Jason Roelofs]

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

Note that, due to the nature of floats, calling to_f on that string
brings you back into the problem again:

x = ( 73.07 - 64.00 )
y = x.r2p( 2 )
p x, y, "%.20f" % y

#=> 9.06999999999999
#=> 9.07
#=> "9.07000000000000028422"

require 'bigdecimal'
require 'bigdecimal/math'
include BigMath

(BigDecimal("73.07") - BigDecimal("64.00")).to_f #=> 9.07

http://www.ruby-doc.org/stdlib/libdoc/bigdecimal/rdoc/index.html

Jason

 

[Phrogz]

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

require 'bigdecimal'
require 'bigdecimal/math'
include BigMath

(BigDecimal("73.07") - BigDecimal("64.00")).to_f #=> 9.07

http://www.ruby-doc.org/stdlib/libdoc/bigdecimal/rdoc/index.html

irb(main):002:0> require 'bigdecimal'
irb(main):003:0> require 'bigdecimal/math'
irb(main):004:0> include BigMath
irb(main):005:0> y = (BigDecimal("73.07") - BigDecimal("64.00")).to_f
=> 9.07
irb(main):006:0> "%.20f" % y
=> "9.07000000000000028422"

You might not see the problem on that particular output, but the
potential inaccuracy still exists as long as you're in the Float
domain.

 

[Ivo Dancet]

Thanks to everyone helping to find me/us a nice solution.

I'm using a method to 'round' the float now.

btw, I noticed the same 'problem' in mysql, so just because someone
may be searching here, the page handling that info is here: http://dev.mysql.com/doc/refman/5.0/en/problems-with-float.html

-c