Bug in double precision computing?

D

Daniel Geisenhoff

Hi all,

can anyone explain to me why cdbl(2.8+2.16+2.85) does result in
7.81000000005 instead of 7.81 ?

Our configuration:

Win2k Server SP4
PIII-550
Dot Net Framework 1.1
VS 2003

Many thanks in advance.
Daniel.
 
S

Shan Plourde

Use the decimal data type. Otherwise you'll get binary rounding, typical
with any real data types. See the Decimal MSDN reference.
Shan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,763
Messages
2,569,562
Members
45,039
Latest member
CasimiraVa

Latest Threads

Top