Floating point calculation problem

Discussion in 'Ruby' started by Hyun chul Park, Jul 8, 2008.

  1. hi.
    i'm confused by float point calculation.
    like this!

    ------------------------------------------
    irb(main):001:0> 123.6 - 123
    => 0.599999999999994

    ------------------------------------------

    how to i get result 0.6 ?
    --
    Posted via http://www.ruby-forum.com/.
     
    Hyun chul Park, Jul 8, 2008
    #1
    1. Advertising

  2. Hi,

    On Tue, Jul 8, 2008 at 10:39 AM, Hyun chul Park <> wrote:
    > hi.
    > i'm confused by float point calculation.
    > like this!
    >
    > ------------------------------------------
    > irb(main):001:0> 123.6 - 123
    > => 0.599999999999994
    >
    > ------------------------------------------
    >
    > how to i get result 0.6 ?


    try using BigDecimal, like:

    irb(main):001:0> require 'bigdecimal'
    => true
    irb(main):002:0> x = BigDecimal("123.6") - BigDecimal("123")
    => #<BigDecimal:8b13c,'0.6E0',4(16)>
    irb(main):003:0> x.to_f
    => 0.6

    zoran
    --
    Human by day user by night
     
    Zoran Regvart, Jul 8, 2008
    #2
    1. Advertising

  3. Zoran Regvart wrote:
    > Hi,
    >
    > On Tue, Jul 8, 2008 at 10:39 AM, Hyun chul Park <>
    > wrote:
    >> how to i get result 0.6 ?

    > try using BigDecimal, like:
    >
    > irb(main):001:0> require 'bigdecimal'
    > => true
    > irb(main):002:0> x = BigDecimal("123.6") - BigDecimal("123")
    > => #<BigDecimal:8b13c,'0.6E0',4(16)>
    > irb(main):003:0> x.to_f
    > => 0.6
    >
    > zoran


    thank you, you're reply.
    --
    Posted via http://www.ruby-forum.com/.
     
    Hyun chul Park, Jul 8, 2008
    #3
  4. Hyun chul Park

    Axel Etzold Guest

    -------- Original-Nachricht --------
    > Datum: Tue, 8 Jul 2008 17:39:50 +0900
    > Von: Hyun chul Park <>
    > An:
    > Betreff: Floating point calculation problem


    Hi --

    > hi.
    > i'm confused by float point calculation.
    > like this!
    >
    > ------------------------------------------
    > irb(main):001:0> 123.6 - 123
    > => 0.599999999999994
    >
    > ------------------------------------------
    >
    > how to i get result 0.6 ?


    you may want to take a look at this:

    http://docs.sun.com/source/806-3568/ncg_goldberg.html

    If you want to do exact calculations, you can use fractions

    http://www.ruby-doc.org/stdlib/libdoc/rational/rdoc/classes/Rational.html

    To convert between the two, for each real number (Float), you can use its continued fraction
    approximation.

    http://www.math.mtu.edu/mathlab/COURSES/holt/dnt/pell1.html

    Here's a link to a Ruby program that calculates digits of Pi from its continued fraction representation
    (and also a Java one, which is much longer):

    http://www.cs.utsa.edu/~wagner/pi/pi_cont.html

    Best regards,

    Axel

    --
    Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten
    Browser-Versionen downloaden: http://www.gmx.net/de/go/browser
     
    Axel Etzold, Jul 8, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. bei
    Replies:
    0
    Views:
    246
  2. bei
    Replies:
    4
    Views:
    409
  3. Saraswati lakki
    Replies:
    0
    Views:
    1,428
    Saraswati lakki
    Jan 6, 2012
  4. Chris Angelico

    Re: Floating point calculation problem

    Chris Angelico, Feb 2, 2013, in forum: Python
    Replies:
    16
    Views:
    255
    Michael Torrie
    Feb 3, 2013
  5. Chris Rebert

    Re: Floating point calculation problem

    Chris Rebert, Feb 2, 2013, in forum: Python
    Replies:
    0
    Views:
    112
    Chris Rebert
    Feb 2, 2013
Loading...

Share This Page