Floating point conversion error?

Discussion in 'Ruby' started by Brandon Hoult, Jan 23, 2008.

  1. Brandon Hoult, Jan 23, 2008
    #1
    1. Advertising

  2. Brandon Hoult wrote:
    > >> (2241.18 * 100).to_i

    >
    > => 224117
    >
    > Anyone know what is up with that?


    Floating point arithmetic isn't entirely accurate.
    >> "%.30f" % (2241.18 * 100)

    => "224117.999999999970896169543266296387"

    And since 224117.999999999970896169543266296387.to_i is 224117, that's what
    you get.

    HTH,
    Sebastian
    --
    NP: Immortal - Withstand The Fall Of Time
    Jabber:
    ICQ: 205544826
    Sebastian Hungerecker, Jan 23, 2008
    #2
    1. Advertising

  3. Hello!

    On Jan 23, 2008 4:25 PM, Brandon Hoult <> wrote:
    > >> (2241.18 * 100).to_i

    > => 224117
    >
    > Anyone know what is up with that?
    >


    Floating point numbers are never 100% accurate. See
    http://en.wikipedia.org/wiki/Floating_point for a detailed
    explanation.
    Thomas Wieczorek, Jan 23, 2008
    #3
  4. Sebastian Hungerecker wrote:
    > Brandon Hoult wrote:
    >> >> (2241.18 * 100).to_i

    >>
    >> => 224117
    >>
    >> Anyone know what is up with that?

    >
    > Floating point arithmetic isn't entirely accurate.
    >>> "%.30f" % (2241.18 * 100)

    > => "224117.999999999970896169543266296387"
    >
    > And since 224117.999999999970896169543266296387.to_i is 224117, that's
    > what
    > you get.
    >
    > HTH,
    > Sebastian



    Thanks... found this after poking around... guess I should have spent
    more time looking.

    http://www.ruby-forum.com/topic/132986#new
    --
    Posted via http://www.ruby-forum.com/.
    Brandon Hoult, Jan 23, 2008
    #4
  5. Brandon Hoult

    Gary Wright Guest

    On Jan 23, 2008, at 10:25 AM, Brandon Hoult wrote:

    >>> (2241.18 * 100).to_i

    > => 224117
    >
    > Anyone know what is up with that?


    What Every Computer Scientist Should Know About Floating-Point
    Arithmetic

    <http://docs.sun.com/source/806-3568/ncg_goldberg.html>
    Gary Wright, Jan 23, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. H aka N
    Replies:
    15
    Views:
    15,600
    Ben Jones
    Mar 2, 2006
  2. Motaz Saad
    Replies:
    7
    Views:
    6,447
  3. Replies:
    4
    Views:
    1,268
    Default User
    Feb 22, 2006
  4. Saraswati lakki
    Replies:
    0
    Views:
    1,277
    Saraswati lakki
    Jan 6, 2012
  5. teeshift
    Replies:
    2
    Views:
    237
    Chris Pearl
    Dec 1, 2006
Loading...

Share This Page