floating-point precision

[Enrique Cruiz]

Hi guys,

I use LUTs to accelerate certain parts of my app. The LUTs contain
float numbers, and I generated the numbers very precisely, up to 10
decimals. All seems fine, however, when I printf the numbers in the
LUT, I do not get the same exact numbers. The first 6 decimals are ok,
but it got beserk after that. To illustrate further:

float myLUT = [0.123456789, 0.123456789, 0.123456789, ................] ;

printf("myLUT: .10f", myLUT[1]);

##### output

myLUT: 0.123456111


I am not quite sure why that is the case, although I believe float do
not have enough precision, so the compiler (VC 8) gives the closest
approximation that can be had with float. Is that so? Can somebody
enlighten me on that issue?

Thanks in advance,


Enrique

 

[Martin Ambuhl]

Hi guys,

I use LUTs to accelerate certain parts of my app. The LUTs contain float
numbers, and I generated the numbers very precisely, up to 10 decimals.
All seems fine, however, when I printf the numbers in the LUT, I do not
get the same exact numbers. The first 6 decimals are ok, but it got
beserk after that. To illustrate further:

float myLUT = [0.123456789, 0.123456789, 0.123456789, ................] ;

floats are guaranteed only a precision of 6 decimal places; doubles (and
long doubles) are guaranteed 10. Many implementations provide more
(sometimes much more) than what is guaranteed, but that cannot be relied on.

printf("myLUT: .10f", myLUT[1]);

##### output

myLUT: 0.123456111


I am not quite sure why that is the case, although I believe float do
not have enough precision, so the compiler (VC 8) gives the closest
approximation that can be had with float. Is that so? Can somebody
enlighten me on that issue?

If you have a C text, read it. If you do not have one, buy one and read it.