NaN handling

Discussion in 'Python' started by Andy McDonagh, May 3, 2006.

  1. Dear python experts,

    I am new to python and this site, so I apologize if this is off topic (i.e. is it a SciPy question?). I will try to demonstrate my problem below:
    --------------------------------------------------------
    #!/usr/local/bin/python

    from scipy import *
    from scipy.stats import *
    a=norm(loc=0,scale=1)
    a_data = a.rvs(10)

    problem = zeros(10)
    print problem

    h_x1_x2 = -sum(problem * log2(a_data))

    print h_x1_x2
    #NaN
    ----------------------------------------------------------

    I need a way of handling NaNs for example R has the 'na.omit' option. Does anybody know if this exists?

    Many thanks

    Andy
     
    Andy McDonagh, May 3, 2006
    #1
    1. Advertising

  2. On 2006-05-03, Andy McDonagh <> wrote:
    > Dear python experts,
    >
    > I am new to python and this site, so I apologize if this is off topic (i.e. is it a SciPy question?). I will try to demonstrate my problem below:
    > --------------------------------------------------------
    > #!/usr/local/bin/python
    >
    > from scipy import *
    > from scipy.stats import *
    > a=norm(loc=0,scale=1)
    > a_data = a.rvs(10)
    >
    > problem = zeros(10)
    > print problem
    >
    > h_x1_x2 = -sum(problem * log2(a_data))
    >
    > print h_x1_x2
    > #NaN
    > ----------------------------------------------------------
    >
    > I need a way of handling NaNs


    NaNs are handled.

    Apparently they aren't handled the way you want them to be?

    > for example R has the 'na.omit' option. Does anybody know if
    > this exists?


    It would help if you explain how you want NaNs handled, but I
    don't recall that tehre are any "options" for handling NaNs
    other than the default way in scipy.

    --
    Grant Edwards grante Yow! It's NO USE... I've
    at gone to "CLUB MED"!!
    visi.com
     
    Grant Edwards, May 3, 2006
    #2
    1. Advertising


  3. > <snip>
    > NaNs are handled.


    Throwing an exception would be nice in regular Python (non-scipy).

    This works to catch NaN on OSX and Linux:

    # assuming x is a number
    if x+1==x or x!=x:
    #x is NaN

    But is expensive as a precautionary measure.
    Assert can be used for testing, if production code can be run with -0
    or -OO.
     
    Ivan Vinogradov, May 5, 2006
    #3
  4. On 2006-05-05, Ivan Vinogradov <> wrote:
    >
    >> <snip>
    >> NaNs are handled.

    >
    > Throwing an exception would be nice in regular Python (non-scipy).


    That would break most of my Python programs (at least most of
    the ones in which I do floating point). My main problem with
    NaNs (and Infs) is that there isn't a string represention that
    is consistent across platforms.

    > This works to catch NaN on OSX and Linux:
    >
    > # assuming x is a number
    > if x+1==x or x!=x:
    > #x is NaN
    >
    > But is expensive as a precautionary measure. Assert can be
    > used for testing, if production code can be run with -0 or
    > -OO.


    There are those of us that need NaNs in production code, so it
    would have to be something that could be configured. I find
    that in my programs the places where I need to do something
    "exceptional" with a NaN are very limited. The vast majority
    of the time, I need them to propagate quietly.

    --
    Grant Edwards grante Yow! Thousands of days of
    at civilians... have produced
    visi.com a... feeling for the
    aesthetic modules --
     
    Grant Edwards, May 5, 2006
    #4
  5. > <snip>
    > There are those of us that need NaNs in production code, so it
    > would have to be something that could be configured. I find
    > that in my programs the places where I need to do something
    > "exceptional" with a NaN are very limited. The vast majority
    > of the time, I need them to propagate quietly.


    Our programming expectations may differ, but an option to catch NaNs as
    an exception is a great idea.

    Regards, Ivan.
     
    Ivan Vinogradov, May 5, 2006
    #5
  6. Andy McDonagh

    Robert Kern Guest

    Ivan Vinogradov wrote:
    >><snip>
    >>There are those of us that need NaNs in production code, so it
    >>would have to be something that could be configured. I find
    >>that in my programs the places where I need to do something
    >>"exceptional" with a NaN are very limited. The vast majority
    >>of the time, I need them to propagate quietly.

    >
    > Our programming expectations may differ, but an option to catch NaNs as
    > an exception is a great idea.


    numpy lets the programmer control how NaNs are handled in numpy code. Producing
    a NaN can be ignored, create a warning, raise an exception or call a function.
    It's not well documented at the moment, but the functions are seterr(),
    seterrcall(), seterrobj(), geterr(), geterrcall(), and geterrobj().

    Pure Python has a similar, but somewhat less flexible method, on UNIX platforms.

    http://docs.python.org/dev/lib/module-fpectl.html

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, May 5, 2006
    #6
  7. On 2006-05-05, Ivan Vinogradov <> wrote:
    >> <snip>
    >> There are those of us that need NaNs in production code, so it
    >> would have to be something that could be configured. I find
    >> that in my programs the places where I need to do something
    >> "exceptional" with a NaN are very limited. The vast majority
    >> of the time, I need them to propagate quietly.

    >
    > Our programming expectations may differ, but an option to
    > catch NaNs as an exception is a great idea.


    Unless it's done in hardware, it would be hopelessly slow.
    There some platforms where it's possible for an application to
    enable and handle FP interrupts (all of the exampls for that
    seem to be in Fortran). I don't know if the common platforms
    (IA32/MS-Windows, IA32/Linux) even have that ability.

    --
    Grant Edwards grante Yow! I'm into SOFTWARE!
    at
    visi.com
     
    Grant Edwards, May 5, 2006
    #7
  8. On 2006-05-05, Robert Kern <> wrote:

    >> Our programming expectations may differ, but an option to catch NaNs as
    >> an exception is a great idea.

    >

    [...]

    > Pure Python has a similar, but somewhat less flexible method, on UNIX platforms.
    >
    > http://docs.python.org/dev/lib/module-fpectl.html


    For which "Unix" platforms? It's not there under Linux:

    Python 2.4.2 (#1, Feb 14 2006, 07:55:13)
    [GCC 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import fpectl

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    ImportError: No module named fpectl
    >>>


    --
    Grant Edwards grante Yow! Do I hear th'
    at SPINNING of various
    visi.com WHIRRING, ROUND, and WARM
    WHIRLOMATICS?!
     
    Grant Edwards, May 5, 2006
    #8
  9. Andy McDonagh

    Robert Kern Guest

    Grant Edwards wrote:
    > On 2006-05-05, Robert Kern <> wrote:


    >>Pure Python has a similar, but somewhat less flexible method, on UNIX platforms.
    >>
    >> http://docs.python.org/dev/lib/module-fpectl.html

    >
    > For which "Unix" platforms? It's not there under Linux:
    >
    > Python 2.4.2 (#1, Feb 14 2006, 07:55:13)
    > [GCC 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)] on linux2
    > Type "help", "copyright", "credits" or "license" for more information.
    >
    >>>>import fpectl

    >
    > Traceback (most recent call last):
    > File "<stdin>", line 1, in ?
    > ImportError: No module named fpectl


    You might have to enable it during the Python build process.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, May 5, 2006
    #9
  10. On 5-May-06, at 6:45 PM, Grant Edwards wrote:

    > On 2006-05-05, Robert Kern <> wrote:
    >
    >>> Our programming expectations may differ, but an option to catch
    >>> NaNs as
    >>> an exception is a great idea.

    >>

    > [...]
    >
    >> Pure Python has a similar, but somewhat less flexible method, on
    >> UNIX platforms.
    >>
    >> http://docs.python.org/dev/lib/module-fpectl.html

    >
    > For which "Unix" platforms? It's not there under Linux:

    <snip>

    It doesn't seem to be here under OSX either (universal Python install).

    Since numpy seems to be working on a variety of platforms/hardware,
    how hard would it be to extract this functionality from it to add to
    Python proper?

    Cheers, Ivan.
     
    Ivan Vinogradov, May 6, 2006
    #10
  11. Andy McDonagh

    Robert Kern Guest

    Ivan Vinogradov wrote:

    > It doesn't seem to be here under OSX either (universal Python install).


    It's not enabled by default. In the source distribution, it is
    Modules/fpectlmodule.c .

    > Since numpy seems to be working on a variety of platforms/hardware,
    > how hard would it be to extract this functionality from it to add to
    > Python proper?


    Harder than just enabling fpectl.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, May 6, 2006
    #11
  12. Andy McDonagh

    Dan Bishop Guest

    Ivan Vinogradov wrote:
    > > <snip>
    > > NaNs are handled.

    >
    > Throwing an exception would be nice in regular Python (non-scipy).
    >
    > This works to catch NaN on OSX and Linux:
    >
    > # assuming x is a number
    > if x+1==x or x!=x:
    > #x is NaN


    x != x works, but:

    >>> x = 1e100
    >>> x + 1 == x

    True
     
    Dan Bishop, May 6, 2006
    #12
  13. Robert Kern <> writes:

    > Ivan Vinogradov wrote:
    >
    > > It doesn't seem to be here under OSX either (universal Python install).

    >
    > It's not enabled by default. In the source distribution, it is
    > Modules/fpectlmodule.c .
    >
    > > Since numpy seems to be working on a variety of platforms/hardware,
    > > how hard would it be to extract this functionality from it to add to
    > > Python proper?

    >
    > Harder than just enabling fpectl.


    Last thing I heard fpectl was considered to be completely broken -- it's
    likely not disabled by default for no reason. A short google turned up this:


    Comment By: Michael Hudson (mwh)
    Date: 2006-02-20 12:59

    Message:
    Logged In: YES
    user_id=6656

    Waaa, the correct thing to is to remove the --with-fpectl from configure!
    I've
    been meaning to post to python-dev for a while now proposing the ripping out
    of this code. It's just not useful any more. At any rate, I am actively
    opposed to
    advertising its existence more widely.

    And in pep 356 (python 2.5 release schedule) we find:

    Possible features for 2.5
    [...]
    - Remove the fpectl module?

    So "just enabling fpectl" doesn't look like a viable long term solution.

    'as
     
    Alexander Schmolck, May 6, 2006
    #13
  14. Em Sex, 2006-05-05 às 16:37 -0400, Ivan Vinogradov escreveu:
    > This works to catch NaN on OSX and Linux:
    >
    > # assuming x is a number
    > if x+1==x or x!=x:
    > #x is NaN


    This works everywhere:

    nan = float('nan')

    ..
    ..
    ..

    if x == nan:
    # x is not a number

    --
    Felipe.
     
    Felipe Almeida Lessa, May 6, 2006
    #14
  15. Felipe Almeida Lessa <> writes:

    > Em Sex, 2006-05-05 às 16:37 -0400, Ivan Vinogradov escreveu:
    > > This works to catch NaN on OSX and Linux:
    > >
    > > # assuming x is a number
    > > if x+1==x or x!=x:
    > > #x is NaN

    >
    > This works everywhere:
    >
    > nan = float('nan')
    >
    > .
    > .
    > .
    >
    > if x == nan:
    > # x is not a number


    No it doesn't.

    'as
     
    Alexander Schmolck, May 6, 2006
    #15
  16. Andy McDonagh

    Terry Reedy Guest

    "Felipe Almeida Lessa" <> wrote in message
    news:...
    > This works everywhere:
    >
    > nan = float('nan')


    Not.

    >>> nan = float('nan')


    Traceback (most recent call last):
    File "<pyshell#4>", line 1, in -toplevel-
    nan = float('nan')
    ValueError: invalid literal for float(): nan

    Above is Windows, which requires something else.

    tjr
     
    Terry Reedy, May 6, 2006
    #16
  17. Terry Reedy wrote:
    > "Felipe Almeida Lessa" <> wrote in message
    > news:...
    >> This works everywhere:
    >>
    >> nan = float('nan')

    >
    > Not.
    >
    >>>> nan = float('nan')

    >
    > Traceback (most recent call last):
    > File "<pyshell#4>", line 1, in -toplevel-
    > nan = float('nan')
    > ValueError: invalid literal for float(): nan
    >
    > Above is Windows, which requires something else.


    I think he meant:

    >>> float("NaN")

    nan

    That's Python 2.4.1 on Mac OS X.
     
    Ryan Forsythe, May 6, 2006
    #17
  18. Andy McDonagh

    Terry Reedy Guest

    "Ryan Forsythe" <> wrote in message
    news:...
    > Terry Reedy wrote:
    >> "Felipe Almeida Lessa" <> wrote in message
    >> news:...
    >>> This works everywhere:
    >>>
    >>> nan = float('nan')

    >>
    >> Not.
    >>
    >>>>> nan = float('nan')

    >>
    >> Traceback (most recent call last):
    >> File "<pyshell#4>", line 1, in -toplevel-
    >> nan = float('nan')
    >> ValueError: invalid literal for float(): nan
    >>
    >> Above is Windows, which requires something else.

    >
    > I think he meant:
    >
    > >>> float("NaN")

    > nan
    >
    > That's Python 2.4.1 on Mac OS X.


    >>> float("NaN")


    Traceback (most recent call last):
    File "<pyshell#5>", line 1, in -toplevel-
    float("NaN")
    ValueError: invalid literal for float(): NaN

    As Tim Peters has said often enough, this sort of thing is specific to the
    underlying C library and will remain so until someone cares enough to write
    or fund a cross-platform solution.

    Terry Jan Reedy
     
    Terry Reedy, May 6, 2006
    #18
  19. Andy McDonagh

    Robert Kern Guest

    Felipe Almeida Lessa wrote:
    > Em Sex, 2006-05-05 às 16:37 -0400, Ivan Vinogradov escreveu:
    >
    >>This works to catch NaN on OSX and Linux:
    >>
    >># assuming x is a number
    >>if x+1==x or x!=x:
    >> #x is NaN

    >
    > This works everywhere:
    >
    > nan = float('nan')


    Have you tried it on Windows?

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, May 6, 2006
    #19
  20. Andy McDonagh

    Robert Kern Guest

    Alexander Schmolck wrote:
    > Robert Kern <> writes:
    >>Ivan Vinogradov wrote:


    >>>Since numpy seems to be working on a variety of platforms/hardware,
    >>>how hard would it be to extract this functionality from it to add to
    >>>Python proper?

    >>
    >>Harder than just enabling fpectl.

    >
    > Last thing I heard fpectl was considered to be completely broken -- it's
    > likely not disabled by default for no reason.


    Fair enough. If you want to go through numpy's code to rip out its floating
    point error handling, knock yourself out. It's not going to be trivial, though.
    It's heavily embedded in the ufunc machinery.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, May 6, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. DiscoStu
    Replies:
    9
    Views:
    762
    Harald Hein
    Aug 5, 2003
  2. max(NaN,0) should be NaN

    , Aug 28, 2006, in forum: C Programming
    Replies:
    61
    Views:
    1,260
    Michel Hack
    Sep 8, 2006
  3. John [H2O]
    Replies:
    6
    Views:
    540
    Robert Kern
    Jun 16, 2008
  4. Replies:
    6
    Views:
    1,534
    Richard Tobin
    Mar 19, 2009
  5. John Kopanas
    Replies:
    5
    Views:
    125
    Rob Biedenharn
    Jan 24, 2007
Loading...

Share This Page