NaN handling

[Andy McDonagh]

Dear python experts,

I am new to python and this site, so I apologize if this is off topic (i.e. is it a SciPy question?). I will try to demonstrate my problem below:
--------------------------------------------------------
#!/usr/local/bin/python

from scipy import *
from scipy.stats import *
a=norm(loc=0,scale=1)
a_data = a.rvs(10)

problem = zeros(10)
print problem

h_x1_x2 = -sum(problem * log2(a_data))

print h_x1_x2
#NaN
----------------------------------------------------------

I need a way of handling NaNs for example R has the 'na.omit' option. Does anybody know if this exists?

Many thanks

Andy

 

[Grant Edwards]

Dear python experts,

I am new to python and this site, so I apologize if this is off topic (i.e. is it a SciPy question?). I will try to demonstrate my problem below:
--------------------------------------------------------
#!/usr/local/bin/python

from scipy import *
from scipy.stats import *
a=norm(loc=0,scale=1)
a_data = a.rvs(10)

problem = zeros(10)
print problem

h_x1_x2 = -sum(problem * log2(a_data))

print h_x1_x2
#NaN

NaNs are handled.

Apparently they aren't handled the way you want them to be?

for example R has the 'na.omit' option. Does anybody know if
this exists?

It would help if you explain how you want NaNs handled, but I
don't recall that tehre are any "options" for handling NaNs
other than the default way in scipy.

 

[Ivan Vinogradov]

<snip>
NaNs are handled.

Throwing an exception would be nice in regular Python (non-scipy).

This works to catch NaN on OSX and Linux:

# assuming x is a number
if x+1==x or x!=x:
#x is NaN

But is expensive as a precautionary measure.
Assert can be used for testing, if production code can be run with -0
or -OO.

 

[Grant Edwards]

Throwing an exception would be nice in regular Python (non-scipy).

That would break most of my Python programs (at least most of
the ones in which I do floating point). My main problem with
NaNs (and Infs) is that there isn't a string represention that
is consistent across platforms.

This works to catch NaN on OSX and Linux:

# assuming x is a number
if x+1==x or x!=x:
#x is NaN

But is expensive as a precautionary measure. Assert can be
used for testing, if production code can be run with -0 or
-OO.

There are those of us that need NaNs in production code, so it
would have to be something that could be configured. I find
that in my programs the places where I need to do something
"exceptional" with a NaN are very limited. The vast majority
of the time, I need them to propagate quietly.

 

[Ivan Vinogradov]

There are those of us that need NaNs in production code, so it
would have to be something that could be configured. I find
that in my programs the places where I need to do something
"exceptional" with a NaN are very limited. The vast majority
of the time, I need them to propagate quietly.

Our programming expectations may differ, but an option to catch NaNs as
an exception is a great idea.

Regards, Ivan.

 

[Robert Kern]

Our programming expectations may differ, but an option to catch NaNs as
an exception is a great idea.

numpy lets the programmer control how NaNs are handled in numpy code. Producing
a NaN can be ignored, create a warning, raise an exception or call a function.
It's not well documented at the moment, but the functions are seterr(),
seterrcall(), seterrobj(), geterr(), geterrcall(), and geterrobj().

Pure Python has a similar, but somewhat less flexible method, on UNIX platforms.

http://docs.python.org/dev/lib/module-fpectl.html

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Grant Edwards]

Our programming expectations may differ, but an option to
catch NaNs as an exception is a great idea.

Unless it's done in hardware, it would be hopelessly slow.
There some platforms where it's possible for an application to
enable and handle FP interrupts (all of the exampls for that
seem to be in Fortran). I don't know if the common platforms
(IA32/MS-Windows, IA32/Linux) even have that ability.

 

[Grant Edwards]

Our programming expectations may differ, but an option to catch NaNs as
an exception is a great idea.

[...]

Pure Python has a similar, but somewhat less flexible method, on UNIX platforms.

http://docs.python.org/dev/lib/module-fpectl.html

For which "Unix" platforms? It's not there under Linux:

Python 2.4.2 (#1, Feb 14 2006, 07:55:13)
[GCC 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):

 

[Robert Kern]

Pure Python has a similar, but somewhat less flexible method, on UNIX platforms.

http://docs.python.org/dev/lib/module-fpectl.html

For which "Unix" platforms? It's not there under Linux:

Python 2.4.2 (#1, Feb 14 2006, 07:55:13)
[GCC 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: No module named fpectl

You might have to enable it during the Python build process.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Ivan Vinogradov]

Our programming expectations may differ, but an option to catch
NaNs as
an exception is a great idea.

[...]

Pure Python has a similar, but somewhat less flexible method, on
UNIX platforms.

http://docs.python.org/dev/lib/module-fpectl.html

For which "Unix" platforms? It's not there under Linux:

<snip>

It doesn't seem to be here under OSX either (universal Python install).

Since numpy seems to be working on a variety of platforms/hardware,
how hard would it be to extract this functionality from it to add to
Python proper?

Cheers, Ivan.

 

[Robert Kern]

It doesn't seem to be here under OSX either (universal Python install).

It's not enabled by default. In the source distribution, it is
Modules/fpectlmodule.c .

Since numpy seems to be working on a variety of platforms/hardware,
how hard would it be to extract this functionality from it to add to
Python proper?

Harder than just enabling fpectl.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Dan Bishop]

Throwing an exception would be nice in regular Python (non-scipy).

This works to catch NaN on OSX and Linux:

# assuming x is a number
if x+1==x or x!=x:
#x is NaN

x != x works, but:
True

 

[Alexander Schmolck]

It's not enabled by default. In the source distribution, it is
Modules/fpectlmodule.c .


Harder than just enabling fpectl.

Last thing I heard fpectl was considered to be completely broken -- it's
likely not disabled by default for no reason. A short google turned up this:


Comment By: Michael Hudson (mwh)
Date: 2006-02-20 12:59

Message:
Logged In: YES
user_id=6656

Waaa, the correct thing to is to remove the --with-fpectl from configure!
I've
been meaning to post to python-dev for a while now proposing the ripping out
of this code. It's just not useful any more. At any rate, I am actively
opposed to
advertising its existence more widely.

And in pep 356 (python 2.5 release schedule) we find:

Possible features for 2.5
[...]
- Remove the fpectl module?

So "just enabling fpectl" doesn't look like a viable long term solution.

'as

 

[Felipe Almeida Lessa]

Em Sex, 2006-05-05 às 16:37 -0400, Ivan Vinogradov escreveu:

This works to catch NaN on OSX and Linux:

# assuming x is a number
if x+1==x or x!=x:
#x is NaN

This works everywhere:

nan = float('nan')

..
..
..

if x == nan:
# x is not a number

 

[Alexander Schmolck]

Em Sex, 2006-05-05 às 16:37 -0400, Ivan Vinogradov escreveu:

This works everywhere:

nan = float('nan')

.
.
.

if x == nan:
# x is not a number

No it doesn't.

'as

 

[Terry Reedy]

This works everywhere:

nan = float('nan')
Not.

Traceback (most recent call last):
File "<pyshell#4>", line 1, in -toplevel-
nan = float('nan')
ValueError: invalid literal for float(): nan

Above is Windows, which requires something else.

tjr

 

[Ryan Forsythe]

Traceback (most recent call last):
File "<pyshell#4>", line 1, in -toplevel-
nan = float('nan')
ValueError: invalid literal for float(): nan

Above is Windows, which requires something else.

I think he meant:
nan

That's Python 2.4.1 on Mac OS X.

 

[Terry Reedy]

I think he meant:

nan

That's Python 2.4.1 on Mac OS X.

Traceback (most recent call last):
File "<pyshell#5>", line 1, in -toplevel-
float("NaN")
ValueError: invalid literal for float(): NaN

As Tim Peters has said often enough, this sort of thing is specific to the
underlying C library and will remain so until someone cares enough to write
or fund a cross-platform solution.

Terry Jan Reedy

 

[Robert Kern]

Em Sex, 2006-05-05 às 16:37 -0400, Ivan Vinogradov escreveu:


This works everywhere:

nan = float('nan')

Have you tried it on Windows?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Robert Kern]

Last thing I heard fpectl was considered to be completely broken -- it's
likely not disabled by default for no reason.

Fair enough. If you want to go through numpy's code to rip out its floating
point error handling, knock yourself out. It's not going to be trivial, though.
It's heavily embedded in the ufunc machinery.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco