Forums
New posts
Search forums
Members
Current visitors
Log in
Register
What's new
Search
Search
Search titles only
By:
New posts
Search forums
Menu
Log in
Register
Install the app
Install
Forums
Archive
Archive
Ruby
Rounding to X digits
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Message
[QUOTE="trans. (T. Onoma), post: 4446143"] | | irb(main):033:0> printf "%20.15f",(134.45 * (1.0/0.1)) | 1344.500000000000000=> nil | irb(main):034:0> printf "%20.15f",(134.45/0.1) | 1344.499999999999773=> nil That isn't too... er... reassuring. | I suppose this is not unexpected (my mama warned me 'bout floats) but it | is a little unexpected--no, I'm wrong, 1/5 is a repeating decimal base | 2, so it's perfectly expected. Expected, really? I'm not all that familiar with floating point standards, but I would have though this kind of thing would have been worked out long ago. It's a pretty substantiate miscalculation. --And the fact that simply using '*(1.0/0.1)' fixes it, makes me think so even more. T. [/QUOTE]
Verification
Post reply
Forums
Archive
Archive
Ruby
Rounding to X digits
Top