struct doesn't handle NaN values?

[Grant Edwards]

Perhaps I'm doing something wrong: the struct module docs say
it's IEE 754, but I can't figure out how to get it to handle
NaN values correctly (either packing or unpacking).
Traceback (most recent call last):
(-6.8056469327705772e+38,)

I don't have my copy of 754 at hand, but I'm pretty sure that
0xffffffff is a NaN (printf on IA32 Linux agrees) and not
-6.8056469327705772e+38 as claimed by struct.unpack().

 

[John Roth]

Perhaps I'm doing something wrong: the struct module docs say
it's IEE 754, but I can't figure out how to get it to handle
NaN values correctly (either packing or unpacking).

Traceback (most recent call last):

(-6.8056469327705772e+38,)

I don't have my copy of 754 at hand, but I'm pretty sure that
0xffffffff is a NaN (printf on IA32 Linux agrees) and not
-6.8056469327705772e+38 as claimed by struct.unpack().

I believe that struct uses the c library as is, like the
float support in general. Unfortunately, that leaves
everyone at the mercy of the rather inconsistent
and ideosyncratic implementations of corner cases
in existing c libraries. See PEP 754 for a discussion
of the issues.

John Roth

 

[David M. Cooke]

Perhaps I'm doing something wrong: the struct module docs say
it's IEE 754, but I can't figure out how to get it to handle
NaN values correctly (either packing or unpacking).

Traceback (most recent call last):
File "<stdin>", line 1, in ?
SystemError: frexp() result out of range

Doing this with the native format is no trouble:'\x00\x00\xc0\x7f'

....which is a NaN since the exponent part is all 1's and the
significand is non-zero.

Again,

(nan,)

Of course, if you want to worry about endian issues here, I think
you're SOL with using struct.

 

[Grant Edwards]

I believe that struct uses the c library as is,

That's not what the docs say. The struct docs says it converts
between native Python values and IEEE 754. If that's not the
case and it converts between Python float format and "native C
library" format, then the docs need to be changed [and I'll
need to write something that converts to/from IEEE 754 format.]

like the float support in general. Unfortunately, that leaves
everyone at the mercy of the rather inconsistent and
ideosyncratic implementations of corner cases in existing c
libraries.

But, my c library seems to handle it correctly:

1) 0xffffffff is treated correctly as a NaN.
2) NaN's are detected and converted to strings as 'nan'.

Which part of the C library is broken?

 

[Grant Edwards]

Doing this with the native format is no trouble:

Of course, if you want to worry about endian issues here, I think
you're SOL with using struct.

Yes, I want to worry about endian issues. I'm communicating
via DeviceNet, and DeviceNet specifies little-endian. I don't
want to assume the host is little-endian, so I guess I'm SOL.

OTOH, I suppose if I assume that 32-bit integers and floats
have the same byte order, I can use 32-bit native integers as
an intermediate format:

nanString = struct.pack("f",float('nan'))
nanInt, = struct.unpack("I",nanString)
extNan = struct.pack("<I",nanInt)

That's ugly, and I'm sure will break someday, but I guess it
works for now.

I suppose I should take a look at the sources for struct and
see if I can fix it...

 

[John Roth]

I believe that struct uses the c library as is,

That's not what the docs say. The struct docs says it converts
between native Python values and IEEE 754. If that's not the
case and it converts between Python float format and "native C
library" format, then the docs need to be changed [and I'll
need to write something that converts to/from IEEE 754 format.]

Actually, it doesn't. It does say IEEE, but it doesn't say
IEEE 754. While that's a nit, it also doesn't say it converts
anything other than byte order. When you look at the source,
I suspect that what you'll find is a straight copy from the buffer
into wherever the float object keeps the actual value, or the
reverse, of course.

But, my c library seems to handle it correctly:

1) 0xffffffff is treated correctly as a NaN.
2) NaN's are detected and converted to strings as 'nan'.

Which part of the C library is broken?

Which C library? Python runs on 20 different systems,
many of which have multiple operating systems, each
of which has its own C library with its own problems.
It's not even a standards issue: the older standards didn't
specify what the C library's conversion routines should do.

I referred you to PEP 754 for a reason. That PEP contains
a thorough discussion of the issues in the treatment of special
values in Python floating point.

http://www.python.org/peps/pep-0754.html

It also contains a reference to a module that handles
the matter.


John Roth

 

[Grant Edwards]

Which C library?

That's what I'm asked? I was told the C library was broken. I
wanted to know what C library.

Python runs on 20 different systems, many of which have
multiple operating systems, each of which has its own C
library with its own problems. It's not even a standards
issue: the older standards didn't specify what the C library's
conversion routines should do.

I presumed that struct was doing the conversion itself. The
doc specified IEEE format, it would have to do it's own
conversion since it couldn't assume that the host used IEEE
format, and as you say, there's no portable library support
that can be relied upon.

I referred you to PEP 754 for a reason. That PEP contains
a thorough discussion of the issues in the treatment of special
values in Python floating point.

http://www.python.org/peps/pep-0754.html

It also contains a reference to a module that handles
the matter.

I read it. It only handles double-precision values, and I'm
working with single-precision values.

 

[John Roth]

That's what I'm asked? I was told the C library was broken. I
wanted to know what C library.

The breakage has to do with the character values they
use to denote infinities and NaN, not (as far as
I know) with the detection of them. Whether they are
"broken" is a matter of interpretation, since the C standard
didn't specify what the library should expect and return
when converting from string to float and vice versa.
What is true is that they aren't consistent.

I presumed that struct was doing the conversion itself. The
doc specified IEEE format, it would have to do it's own
conversion since it couldn't assume that the host used IEEE
format, and as you say, there's no portable library support
that can be relied upon.

I don't think so. I believe that the reference to IEEE means
that it generally expects bit compatible IEEE representations.
There may be differences, but I doubt if struct is aware of
them. As usual, the source would be definitive.

I read it. It only handles double-precision values, and I'm
working with single-precision values.

I expect it would be easy enough to convert. I just read
it, and it doesn't look at all complicated.

John Roth

 

[Grant Edwards]

The breakage has to do with the character values they use to
denote infinities and NaN,

Right. That's got nothing to do with what I'm currently
whining about, which was struct's failure when byte ordering is
specified to convert properly between a native NaN and IEEE 754
NaN (even when native format is IEEE 754) . I'm quite happy
with the way string<->native-float handles NaNs on the host I'm
using. Not that it wouldn't be nice for it to be consistent
across platforms.

not (as far as I know) with the detection of them. Whether
they are "broken" is a matter of interpretation, since the C
standard didn't specify what the library should expect and
return when converting from string to float and vice versa.
What is true is that they aren't consistent.

Though I realize there are consistency issues, they work quite
well enough for my application at the moment. Struct, however,
doesn't.

I don't think so.

Eh? So you think that struct is using a C library to do the
conversion, or that there is portable C library support for
converting between native FP format and IEEE 754?

I believe that the reference to IEEE means that it generally
expects bit compatible IEEE representations.

Hmm, what would be the difference would be between standrd IEEE
representation and a "bit-compatible IEEE representation".

There may be differences, but I doubt if struct is aware of
them. As usual, the source would be definitive.


I expect it would be easy enough to convert. I just read it,
and it doesn't look at all complicated.

Probably not. I've done it before (in C) but having struct
convert the values properly would be right thing to do rather
expect the user to check for certain values that struct doesn't
convert correctly.