Basic calculation returns incorrect values

D

Dale Reed

We're having a lot of problems getting visual studio to calculate even the
most basic arithmetic correctly.

For example:

Dim MyValue AS Double = 2.155 * 100

This should give me 215.5 right? Wrong. It gives me 215.499999999999999997

Change it to 1.155, it works. 3.155 it works. 2.155 it doesn't work.
We've found several values that don't seem to work correctly. I'm staggered
at this, we're trying to develop accounts based systems and yet visual
studio can't even multiply an item by 100 correctly?! It's not like I'm
doing anything complicated!

Has anyone else come across this and is there some kind of hot fix or patch
that can be applied to sort this out?

We've already been having problems with the round function in .net 1.1 and
had to write our own. The above problem occurs in 1.1 and 2.

Thanks in advance...
 
F

Flinky Wisty Pomm

I doubt there's a hot fix for it; the problem, in essence, is that .Net
doesn't know that (215.5 * 100) is an "easier" sum than (214.9871 *
99.31048) - because for a computer, it isn't.

If the latter was wrong by 0.0000000000000003 you'd be hard pressed to
notice. You only notice the specific case because humans think in base
10.

That's an aside though, and not very useful. If you need to know that
your numbers are accurate, avoid floating points. Stick to longs and
divide 'em at the end. That's generally how you are recommended to
store money values in databases, too, for exactly the same reason -
floating point arithmetic is not 100% accurate, and accounting
departments are.
 
L

Lukas Kurka

why trivial calc.exe in windows calculate 2.155*100 right but .NET doesn't?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,777
Messages
2,569,604
Members
45,227
Latest member
Daniella65

Latest Threads

Top