weird math problem!!

Discussion in 'ASP .Net' started by Todd, Oct 1, 2007.

  1. Todd

    Todd Guest

    OK...I have a form that has two text boxes. These two fields are being
    filled from a file import, which works fine. I am then passing these
    values into a custom class with both values typed as DOUBLE (IE:
    double.parse(mytextbox1.text) ).

    When I subtract these two values, I'm get an erroneous .00000000000002
    being added to the result of the subtraction!!

    The form fills the instance of the class...

    myclass.value1 = double.parse(mytextbox1.text)
    myclass.value2 = double.parse(mytextbox2.text)

    ....then in another procedure I'm trying to do the math...


    Dim MyValue as double = myclass.value1 - myclass.value2

    No matter what the source numbers, I always get a .00000000000002
    added into the figure!!!! The class values are always correct, it's
    just the subtraction that generates the incorect figure!!

    This is driving me crazy!!!!!!

    Anyone have any idea!?!?!?!?!?!?!?!
    Todd, Oct 1, 2007
    #1
    1. Advertising

  2. Todd

    bruce barker Guest

    thats expected behavior. the datatype double which is implemented as
    twos complement, does not hold 1/100's accurately, similar to decimal
    not holding thirds.

    use the decimal datatype.

    -- bruce (sqlwork.com)


    Todd wrote:
    > OK...I have a form that has two text boxes. These two fields are being
    > filled from a file import, which works fine. I am then passing these
    > values into a custom class with both values typed as DOUBLE (IE:
    > double.parse(mytextbox1.text) ).
    >
    > When I subtract these two values, I'm get an erroneous .00000000000002
    > being added to the result of the subtraction!!
    >
    > The form fills the instance of the class...
    >
    > myclass.value1 = double.parse(mytextbox1.text)
    > myclass.value2 = double.parse(mytextbox2.text)
    >
    > ...then in another procedure I'm trying to do the math...
    >
    >
    > Dim MyValue as double = myclass.value1 - myclass.value2
    >
    > No matter what the source numbers, I always get a .00000000000002
    > added into the figure!!!! The class values are always correct, it's
    > just the subtraction that generates the incorect figure!!
    >
    > This is driving me crazy!!!!!!
    >
    > Anyone have any idea!?!?!?!?!?!?!?!
    >
    bruce barker, Oct 1, 2007
    #2
    1. Advertising

  3. Todd

    mike Guest

    Hi Todd

    I found this URL which may help to explain things a bit. In summary -
    the double, like float is a floating point representation of a number.
    As such there is some error built into it. The article advocates using
    Decimal data types when you want the precision. Your alternative is to
    round the value. Not having done any work with arithmetic in .NET, I
    don't know how.

    This is well known with the float data type. If you need an example,
    try to determine how many $0.1 items you can purchase with $1.00 (the
    decimal places are important). The answer (using float data types) is
    9.

    http://techembassy.blogspot.com/2006/07/data-types-single-double-and-decimal.html

    Hope this helps
    Mike
    mike, Oct 1, 2007
    #3
  4. Todd

    Juan Romero Guest

    Use decimal data type.

    "Todd" <> wrote in message
    news:...
    > OK...I have a form that has two text boxes. These two fields are being
    > filled from a file import, which works fine. I am then passing these
    > values into a custom class with both values typed as DOUBLE (IE:
    > double.parse(mytextbox1.text) ).
    >
    > When I subtract these two values, I'm get an erroneous .00000000000002
    > being added to the result of the subtraction!!
    >
    > The form fills the instance of the class...
    >
    > myclass.value1 = double.parse(mytextbox1.text)
    > myclass.value2 = double.parse(mytextbox2.text)
    >
    > ...then in another procedure I'm trying to do the math...
    >
    >
    > Dim MyValue as double = myclass.value1 - myclass.value2
    >
    > No matter what the source numbers, I always get a .00000000000002
    > added into the figure!!!! The class values are always correct, it's
    > just the subtraction that generates the incorect figure!!
    >
    > This is driving me crazy!!!!!!
    >
    > Anyone have any idea!?!?!?!?!?!?!?!
    >
    Juan Romero, Oct 2, 2007
    #4
  5. Todd

    Todd Guest

    On 1 Oct, 23:58, mike <> wrote:
    > ....I found this URL which may help to explain things a bit. .....http://techembassy.blogspot.com/2006/07/data-types-single-double-and-...
    >
    > Hope this helps
    > Mike



    Thanks Mike, and to you all. This is my first app heavy on math. I can
    not believe this is "expected behaviour". Then again, I just read the
    article about Excel 2007 addition problem! Can this be called
    "Microsoft Math"?
    Todd, Oct 3, 2007
    #5
  6. Todd

    mike Guest

    > I can
    > not believe this is "expected behaviour". Then again, I just read the
    > article about Excel 2007 addition problem! Can this be called
    > "Microsoft Math"?


    Hi Todd

    This isn't a Microsoft thing. It's the IEEE or ISO (can't remember
    which) specification. It's built into the standard for floating point
    data types like float and double. Microsoft are obviously storing
    their numeric data as floating point numbers as it takes less storage
    than something more accurate.

    Regards
    Mike
    mike, Oct 16, 2007
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. chirs
    Replies:
    18
    Views:
    750
    Chris Uppal
    Mar 2, 2004
  2. AciD_X
    Replies:
    4
    Views:
    8,075
    Jonathan Turkanis
    Apr 1, 2004
  3. Mark Healey
    Replies:
    7
    Views:
    1,462
    Tim Prince
    May 22, 2006
  4. Philipp
    Replies:
    9
    Views:
    1,097
    Mark Space
    Jul 23, 2008
  5. VK
    Replies:
    15
    Views:
    1,126
    Dr J R Stockton
    May 2, 2010
Loading...

Share This Page