Python -- floating point arithmetic

[david mainzer]

Dear Python-User,


today i create some slides about floating point arithmetic. I used an
example from

http://docs.python.org/tutorial/floatingpoint.html

so i start the python shell on my linux machine:

dm@maxwell $ python
Python 2.6.5 (release26-maint, May 25 2010, 12:37:06)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information..... sum += 0.1
....But thats looks a little bit wrong for me ... i must be a number greater
then 1.0 because 0.1 = 0.100000000000000005551115123125782702118158340454101562500000000000
in python ... if i print it.

So i create an example program:

sum = 0.0
n = 10
d = 1.0 / n
print "%.60f" % ( d )
for i in range(n):
print "%.60f" % ( sum )
sum += d

print sum
print "%.60f" % ( sum )


-------- RESULTs ------
0.100000000000000005551115123125782702118158340454101562500000
0.000000000000000000000000000000000000000000000000000000000000
0.100000000000000005551115123125782702118158340454101562500000
0.200000000000000011102230246251565404236316680908203125000000
0.300000000000000044408920985006261616945266723632812500000000
0.400000000000000022204460492503130808472633361816406250000000
0.500000000000000000000000000000000000000000000000000000000000
0.599999999999999977795539507496869191527366638183593750000000
0.699999999999999955591079014993738383054733276367187500000000
0.799999999999999933386618522490607574582099914550781250000000
0.899999999999999911182158029987476766109466552734375000000000
1.0
0.999999999999999888977697537484345957636833190917968750000000

and the jump from 0.50000000000000*** to 0.59999999* looks wrong
for me ... do i a mistake or is there something wrong in the
representation of the floating points in python?

my next question, why could i run

print "%.66f" % ( sum )

but not

print "%.67f" % ( sum )

can anybody tell me how python internal represent a float number??


Best and many thanks in advanced,
David

 

[Mark Dickinson]

Dear Python-User,

today i create some slides about floating point arithmetic. I used an
example from

http://docs.python.org/tutorial/floatingpoint.html

so i start the python shell on my linux machine:

dm@maxwell $ python
Python 2.6.5 (release26-maint, May 25 2010, 12:37:06)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.>>> >>> sum = 0.0
...     sum += 0.1
...>>> >>> sum
0.99999999999999989

But thats looks a little bit wrong for me ... i must be a number greater
then 1.0 because 0.1 = 0.100000000000000005551115123125782702118158340454101562500000000000
in python ... if i print it.

So you've identified one source of error here, namely that 0.1 isn't
exactly representable (and you're correct that the value stored
internally is actually a little greater than 0.1). But you're
forgetting about the other source of error in your example: when you
do 'sum += 0.1', the result typically isn't exactly representable, so
there's another rounding step going on. That rounding step might
produce a number that's smaller than the actual exact sum, and if
enough of your 'sum += 0.1' results are rounded down instead of up,
that would easily explain why the total is still less than 1.0.

So i create an example program:

sum = 0.0
n = 10
d = 1.0 / n
print "%.60f" % ( d )
for i in range(n):
    print "%.60f" % ( sum )
    sum += d

print sum
print "%.60f" % ( sum )

-------- RESULTs ------
0.100000000000000005551115123125782702118158340454101562500000
0.000000000000000000000000000000000000000000000000000000000000
0.100000000000000005551115123125782702118158340454101562500000
0.200000000000000011102230246251565404236316680908203125000000
0.300000000000000044408920985006261616945266723632812500000000
0.400000000000000022204460492503130808472633361816406250000000
0.500000000000000000000000000000000000000000000000000000000000
0.599999999999999977795539507496869191527366638183593750000000
0.699999999999999955591079014993738383054733276367187500000000
0.799999999999999933386618522490607574582099914550781250000000
0.899999999999999911182158029987476766109466552734375000000000
1.0
0.999999999999999888977697537484345957636833190917968750000000

and the jump from 0.50000000000000*** to 0.59999999* looks wrong
for me ... do i a mistake or is there something wrong in the
representation of the floating points in python?

Look at this more closely: you're adding

0.500000000000000000000000....

to

0.1000000000000000055511151231257827021181583404541015625

The *exact* result is, of course

0.6000000000000000055511151231257827021181583404541015625

However, that's not a number that can be exactly represented as a C
double (which is how Python stores floats internally). This value
falls between the two (consecutive) representable values:

0.59999999999999997779553950749686919152736663818359375

and

0.600000000000000088817841970012523233890533447265625

But of these two, the first is closer to the exact value than the
second, so that's the result that you get.

You can convince yourself of these results by using the fractions
module to do exact arithmetic:

True


my next question, why could i run

print "%.66f" % ( sum )

but not

print "%.67f" % ( sum )

That's a historical artefact resulting from use of a fixed-length
buffer somewhere deep in Python's internals. This restriction is
removed in Python 2.7 and Python 3.x.

can anybody tell me how python internal represent a float number??

In CPython, it's stored as a C double, which typically means in IEEE
754 binary64 format. (Of course, since it's a Python object, it's not
just storing the C double itself; it also has fields for the object
type and the reference count, so a Python float typically takes 16
bytes of memory on a 32-bit machine, and 24 bytes on a 64-bit
machine.)

 

[Raymond Hettinger]

Dear Python-User,

today i create some slides about floating point arithmetic. I used an
example from

so i start the python shell on my linux machine:

dm@maxwell $ python
Python 2.6.5 (release26-maint, May 25 2010, 12:37:06)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.>>> >>> sum = 0.0

for i in range(10):

...     sum += 0.1
...>>> >>> sum
0.99999999999999989

But thats looks a little bit wrong for me ... i must be a number greater
then 1.0 because 0.1 = 0.100000000000000005551115123125782702118158340454101562500000000000
in python ... if i print it.

[Mark Dickinson]

So you've identified one source of error here, namely that 0.1 isn't
exactly representable (and you're correct that the value stored
internally is actually a little greater than 0.1).  But you're
forgetting about the other source of error in your example: when you
do 'sum += 0.1', the result typically isn't exactly representable, so
there's another rounding step going on.  That rounding step might
produce a number that's smaller than the actual exact sum, and if
enough of your 'sum += 0.1' results are rounded down instead of up,
that would easily explain why the total is still less than 1.0.

One key for understanding floating point mysteries is to look at the
actual binary sums rather that their approximate representation as a
decimal string. The hex() method can make it easier to visualize
Mark's explanation:
.... s += 0.1
.... print s.hex(), repr(s)


0x1.999999999999ap-4 0.10000000000000001
0x1.999999999999ap-3 0.20000000000000001
0x1.3333333333334p-2 0.30000000000000004
0x1.999999999999ap-2 0.40000000000000002
0x1.0000000000000p-1 0.5
0x1.3333333333333p-1 0.59999999999999998
0x1.6666666666666p-1 0.69999999999999996
0x1.9999999999999p-1 0.79999999999999993
0x1.cccccccccccccp-1 0.89999999999999991
0x1.fffffffffffffp-1 0.99999999999999989

Having used hex() to understand representation error (how the binary
partial sums are displayed), you can use the Fractions module to gain
a better understanding of rounding error introduced by each addition:
exact = Fraction.from_float(s) + Fraction.from_float(0.1)
s += 0.1
actual = Fraction.from_float(s)
error = actual - exact
print '%-35s%-35s\t%s' % (actual, exact, error)


3602879701896397/36028797018963968 3602879701896397/36028797018963968
0
3602879701896397/18014398509481984 3602879701896397/18014398509481984
0
1351079888211149/4503599627370496 10808639105689191/36028797018963968
1/36028797018963968
3602879701896397/9007199254740992
14411518807585589/36028797018963968 -1/36028797018963968
1/2
18014398509481985/36028797018963968 -1/36028797018963968
5404319552844595/9007199254740992
21617278211378381/36028797018963968 -1/36028797018963968
3152519739159347/4503599627370496
25220157913274777/36028797018963968 -1/36028797018963968
7205759403792793/9007199254740992
28823037615171173/36028797018963968 -1/36028797018963968
2026619832316723/2251799813685248
32425917317067569/36028797018963968 -1/36028797018963968
9007199254740991/9007199254740992
36028797018963965/36028797018963968 -1/36028797018963968

Hope this helps your slides,


Raymond

 

[David Mainzer]

Dear Python-User,

today i create some slides about floating point arithmetic. I used an
example from

so i start the python shell on my linux machine:

dm@maxwell $ python
Python 2.6.5 (release26-maint, May 25 2010, 12:37:06)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.>>> >>> sum = 0.0
for i in range(10):

... sum += 0.1
...>>> >>> sum
0.99999999999999989

But thats looks a little bit wrong for me ... i must be a number greater
then 1.0 because 0.1 = 0.100000000000000005551115123125782702118158340454101562500000000000
in python ... if i print it.

[Mark Dickinson]

So you've identified one source of error here, namely that 0.1 isn't
exactly representable (and you're correct that the value stored
internally is actually a little greater than 0.1). But you're
forgetting about the other source of error in your example: when you
do 'sum += 0.1', the result typically isn't exactly representable, so
there's another rounding step going on. That rounding step might
produce a number that's smaller than the actual exact sum, and if
enough of your 'sum += 0.1' results are rounded down instead of up,
that would easily explain why the total is still less than 1.0.

One key for understanding floating point mysteries is to look at the
actual binary sums rather that their approximate representation as a
decimal string. The hex() method can make it easier to visualize
Mark's explanation:
... s += 0.1
... print s.hex(), repr(s)


0x1.999999999999ap-4 0.10000000000000001
0x1.999999999999ap-3 0.20000000000000001
0x1.3333333333334p-2 0.30000000000000004
0x1.999999999999ap-2 0.40000000000000002
0x1.0000000000000p-1 0.5
0x1.3333333333333p-1 0.59999999999999998
0x1.6666666666666p-1 0.69999999999999996
0x1.9999999999999p-1 0.79999999999999993
0x1.cccccccccccccp-1 0.89999999999999991
0x1.fffffffffffffp-1 0.99999999999999989

Having used hex() to understand representation error (how the binary
partial sums are displayed), you can use the Fractions module to gain
a better understanding of rounding error introduced by each addition:
exact = Fraction.from_float(s) + Fraction.from_float(0.1)
s += 0.1
actual = Fraction.from_float(s)
error = actual - exact
print '%-35s%-35s\t%s' % (actual, exact, error)


3602879701896397/36028797018963968 3602879701896397/36028797018963968
0
3602879701896397/18014398509481984 3602879701896397/18014398509481984
0
1351079888211149/4503599627370496 10808639105689191/36028797018963968
1/36028797018963968
3602879701896397/9007199254740992
14411518807585589/36028797018963968 -1/36028797018963968
1/2
18014398509481985/36028797018963968 -1/36028797018963968
5404319552844595/9007199254740992
21617278211378381/36028797018963968 -1/36028797018963968
3152519739159347/4503599627370496
25220157913274777/36028797018963968 -1/36028797018963968
7205759403792793/9007199254740992
28823037615171173/36028797018963968 -1/36028797018963968
2026619832316723/2251799813685248
32425917317067569/36028797018963968 -1/36028797018963968
9007199254740991/9007199254740992
36028797018963965/36028797018963968 -1/36028797018963968

Hope this helps your slides,


Raymond

Thanks to all for your help

All your comments helped me and now i know how it works in python !

Best
David