Need a library to write/read floats and doubles in IEEE754

P

Pavel

Hello,

Does anyone know a (preferably open-source) multi-platform C or C++
library that would be able to write and read C/C++ doubles and floats
to/from streambuf, char array or similar device in IEEE 754 with
reasonably optimal precision and performance?

The purpose is to exchange serialized doubles and floats between C/C++
and Java programs where Java serialization rules are used.

I tried to search for it on the Internet but, surprisingly, could not
find any mature code for this seemingly common purpose.

Thanks in advance,
-Pavel
 
J

Jack Klein


I'm having trouble understanding what you are asking for.
Does anyone know a (preferably open-source) multi-platform C or C++
library that would be able to write and read C/C++ doubles and floats
to/from streambuf, char array or similar device in IEEE 754 with
reasonably optimal precision and performance?

Assuming that your platform uses an IEEE format for doubles and
floats, about the fastest possible way to write them would be to a
binary file with fwrite().
The purpose is to exchange serialized doubles and floats between C/C++
and Java programs where Java serialization rules are used.

I have no idea what Java serialization of floating point types looks
like. Are you talking about writing to a text file or a binary file?
The IEEE 754 standard does not specify a text representation.
I tried to search for it on the Internet but, surprisingly, could not
find any mature code for this seemingly common purpose.

If you need a text file, fprintf() or stream insertion (<<) to a text
stream with appropriate format and precision specifiers.

If you are looking for some faster method of producing text
representations than the standard C or C++ library formatted output
functions, you might well find some, but they might not meet your
definition of "multi-platform", depending on how broadly you define
that term.

You could write, or find, code that takes advantage of the
implementation-defined details of the internal format that might
convert it to text faster than standard functions, but it would not be
directly portable across platforms with different endian
representations of floating point values.

--
Jack Klein
Home: http://JK-Technology.Com
FAQs for
comp.lang.c http://c-faq.com/
comp.lang.c++ http://www.parashift.com/c++-faq-lite/
alt.comp.lang.learn.c-c++
http://www.club.cc.cmu.edu/~ajo/docs/FAQ-acllc.html
 
J

James Kanze

I'm having trouble understanding what you are asking for.
Assuming that your platform uses an IEEE format for doubles
and floats, about the fastest possible way to write them would
be to a binary file with fwrite().

That would be true if it worked, but it doesn't.

Just saying that doubles and floats are IEEE isn't enough to
specify the format, and while both my Linux PC's and my Sparcs
use IEEE, the format is different on each. What *usually* works
(although formally undefined behavior) is to convert the address
of the double to a uint64_t* (or the address of the float to a
uint32_t*), and output that in the usual way, e.g.:

dest.put( value >> 56 ) ;
dest.put( value >> 48 ) ;
dest.put( value >> 40 ) ;
dest.put( value >> 32 ) ;
dest.put( value >> 24 ) ;
dest.put( value >> 16 ) ;
dest.put( value >> 8 ) ;
dest.put( value ) ;

To be totally generic, of course, you need to extract the
different parts of the double, and reassemble them. Something
like:

bool isNeg = source < 0 ;
if ( isNeg ) {
source = - source ;
}
int exp ;
if ( source == 0.0 ) {
exp = 0 ;
} else {
source = ldexp( frexp( source, &exp ), 53 ) ;
exp += 1022 ;
}
uint64_t mant = static_cast< uint64_t >( value ) ;
dest.put( (isNeg ? 0x80 : 0x00) | exp >> 4 ) ;
dest.put( ((exp << 4) & 0xF0) | ((mant >> 48) & 0x0F) ) ;
dest.put( mant >> 40 ) ;
dest.put( mant >> 32 ) ;
dest.put( mant >> 24 ) ;
dest.put( mant >> 16 ) ;
dest.put( mant >> 8 ) ;
dest.put( mant ) ;

(Interestingly enough, when I measured, I found that it wasn't
as slow as it looks. Note that it should work even if the host
doesn't use IEEE, e.g. an IBM or Unisys mainfram, but I've not
been able to actually test it there.)
I have no idea what Java serialization of floating point types
looks like. Are you talking about writing to a text file or a
binary file? The IEEE 754 standard does not specify a text
representation.

Java uses a binary format: for floating point, it's basically
IEEE, output in big endian, the same as XDR, but (I think)
without the alignment requirements. (Formally, I don't think
that IEEE specifies where in the "word" the bits for the sign,
exponant and mantissa are placed, but practically, I've never
heard of an implementation where the order---from high bit to
low---wasn't sign, exponent, mantissa.)
 
P

Pete Becker

Does anyone know a (preferably open-source) multi-platform C or C++
library that would be able to write and read C/C++ doubles and floats
to/from streambuf, char array or similar device in IEEE 754 with
reasonably optimal precision and performance?

http://netlib2.cs.utk.edu/fp/dtoa.c

The general idea is that you convert floating point values to a base 10
text representation with as many digits after the decimal point as are
needed to distinguish the value being written from the two neighboring
values. Sometimes you need very few ("2.0"), and occasioinally you need
quite a few. Caution: it's hackerhead C code, and requires a fair
amount of study to understand what it's doing. There's also a paper
somewhere (I've lost track of it) that gives the theoretical
underpinnings.
 
P

Pavel

James said:
That would be true if it worked, but it doesn't.

Just saying that doubles and floats are IEEE isn't enough to
specify the format, and while both my Linux PC's and my Sparcs
use IEEE, the format is different on each. What *usually* works
(although formally undefined behavior) is to convert the address
of the double to a uint64_t* (or the address of the float to a
uint32_t*), and output that in the usual way, e.g.:

dest.put( value >> 56 ) ;
dest.put( value >> 48 ) ;
dest.put( value >> 40 ) ;
dest.put( value >> 32 ) ;
dest.put( value >> 24 ) ;
dest.put( value >> 16 ) ;
dest.put( value >> 8 ) ;
dest.put( value ) ;

To be totally generic, of course, you need to extract the
different parts of the double, and reassemble them. Something
like:

bool isNeg = source < 0 ;
if ( isNeg ) {
source = - source ;
}
int exp ;
if ( source == 0.0 ) {
exp = 0 ;
} else {
source = ldexp( frexp( source, &exp ), 53 ) ;
exp += 1022 ;
}
uint64_t mant = static_cast< uint64_t >( value ) ;
dest.put( (isNeg ? 0x80 : 0x00) | exp >> 4 ) ;
dest.put( ((exp << 4) & 0xF0) | ((mant >> 48) & 0x0F) ) ;
dest.put( mant >> 40 ) ;
dest.put( mant >> 32 ) ;
dest.put( mant >> 24 ) ;
dest.put( mant >> 16 ) ;
dest.put( mant >> 8 ) ;
dest.put( mant ) ;

(Interestingly enough, when I measured, I found that it wasn't
as slow as it looks. Note that it should work even if the host
doesn't use IEEE, e.g. an IBM or Unisys mainfram, but I've not
been able to actually test it there.)
Thanks James, this is the closest to my problem I got so far.

I did not want to write myself and was looking for a library because I
am lazy and some corner cases should be worked out (native
representation does not necessarily represent all same set of numbers as
IEEE 754 representations). It's quite a bit of coding and testing so I
wanted to find if someone else has already done that before getting into
it. I guess I will start with this code of yours and then try to work my
way using standard C++. Certainly #ifdef - guarded multi-platform
version would do faster for each supported architecture but hopefully
the performance of the standard C++ code will suffice for my purpose.

Regards
-Pavel
 
P

Pavel

Pete said:
http://netlib2.cs.utk.edu/fp/dtoa.c

The general idea is that you convert floating point values to a base 10
text representation with as many digits after the decimal point as are
needed to distinguish the value being written from the two neighboring
values. Sometimes you need very few ("2.0"), and occasioinally you need
quite a few. Caution: it's hackerhead C code, and requires a fair amount
of study to understand what it's doing. There's also a paper somewhere
(I've lost track of it) that gives the theoretical underpinnings.
Thanks Pete -- that is useful piece of code you suggested, but in this
case I am not after decimal formatting / parsing but about the
conversion of native C++ float/doubles into most suitable single- &
double- precision IEEE 754 and back, basically some API along these
lines (assuming char is an 8-bit byte):

enum RetCode { OK, DOMAIN_ERROR };

RetCode doubleToIeee754Single(double, char *buf); // 4-byte buf
RetCode doubleToIeee754Double(double, char *buf); // 8-byte buf
RetCode floatToIeee754Single(float, char *buf); // 4-byte buf
RetCode floatToIeee754Double(float, char *buf); // 8-byte buf
RetCode singleIeee754ToDouble(const char *buf, double &); // 4-byte buf
RetCode doubleIeee754ToDouble(const char *buf, double &); // 8-byte buf
RetCode singleIeee754ToFloat(const char *buf, float &); // 4-byte buf
RetCode doubleIeee754ToFloat(const char *buf, float &); // 8-byte buf

with all NaNs, infinities etc somehow reasonably worked out, for several
platforms (possibly, on a per-platform basis).

Java does it somehow so now I wonder if I can legally scavenge some
open-source JVM code.. will see what I can find or write.

-Pavel
 
J

James Kanze

James Kanze wrote:
I did not want to write myself and was looking for a library
because I am lazy and some corner cases should be worked out
(native representation does not necessarily represent all same
set of numbers as IEEE 754 representations).

If you have to deal with a native representation that is not
IEEE, then you have to define behavior for corner cases. An IBM
double, for example, has more precision (but less range) than an
IEEE double: I suppose when outputting you round (the code I
posted truncates!), but what happens when you input something
that it too large.

I've heard that there may be similar cases even between IEEE
representations, at least where NaN's are involved---a non
trapping NaN may trap elsewhere, or something like that.
It's quite a bit of coding and testing so I wanted to find if
someone else has already done that before getting into it. I
guess I will start with this code of yours and then try to
work my way using standard C++. Certainly #ifdef - guarded
multi-platform version would do faster for each supported
architecture but hopefully the performance of the standard C++
code will suffice for my purpose.

Well, I'd very definitely not invest the effort in supporting
machines without IEEE until I had to. (The three I know of
today are all mainframes: one uses base 16, one uses base 8, and
the one that uses base 2 has 72 bit doubles. On the latter two,
you'll need special handling for ints as well, because they
don't use 2's complement either: there's one with 48 bit signed
magnitude, with 8 bits required to be 0, and the other uses 36
bit 1's complement. Porting networked software to those should
be quite amusing---except that both have Intel based front-ends
to handle the networking.)
 
S

Shobhit Gupta

Hello,

Does anyone know a (preferably open-source) multi-platform C or C++
library that would be able to write and read C/C++ doubles and floats
to/from streambuf, char array or similar device in IEEE 754 with
reasonably optimal precision and performance?

The purpose is to exchange serialized doubles and floats between C/C++
and Java programs where Java serialization rules are used.

I tried to search for it on the Internet but, surprisingly, could not
find any mature code for this seemingly common purpose.

Would this help ?
http://tpl.sourceforge.net/

Its an open source serialization library written in C.
 
P

Pavel

James said:
If you have to deal with a native representation that is not
IEEE, then you have to define behavior for corner cases.
True, that's why I hoped someone did it before me in a reasonable way :).
An IBM
double, for example, has more precision (but less range) than an
IEEE double: I suppose when outputting you round (the code I
posted truncates!), but what happens when you input something
that it too large.
On input will have to check whether IEEE representation exceeds
implementation-specific limits from <climit> and <cfloat>; if it does,
throw domain_error or do something similar; also, for output need to
check whether the number to be written is within the respective (single-
or double-) IEEE limit. Long story short, lots of work to do..
I've heard that there may be similar cases even between IEEE
representations, at least where NaN's are involved---a non
trapping NaN may trap elsewhere, or something like that. True

Well, I'd very definitely not invest the effort in supporting
machines without IEEE until I had to. (The three I know of
today are all mainframes: one uses base 16, one uses base 8, and
the one that uses base 2 has 72 bit doubles. On the latter two,
you'll need special handling for ints as well, because they
don't use 2's complement either: there's one with 48 bit signed
magnitude, with 8 bits required to be 0, and the other uses 36
bit 1's complement. Porting networked software to those should
be quite amusing---except that both have Intel based front-ends
to handle the networking.)
You are right about those -- I do not need them (not right now, anyway);
the main problem even with mostly IEEE-ish architecture that you still
have to detect it (even if only to make sure the preprocessor balks on
unsupported architectures) and for that you need to detect the compiler
first -- both make and version -- because the macros that define the
architectures are implementation-specific and sometimes change between
versions -- and for that you have to first make a decision what versions
of compilers to support. I will need to support 4 compiler makes at
least; as I said, lots of work, at least 90% of which was already done
by others multiple times, better than I ever will.

Anyway, time to stop whining and start coding. Wish me luck.

-Pavel
 
J

James Kanze

True, that's why I hoped someone did it before me in a
reasonable way :).

The problem isn't just the doing. The problem is specifying
what you mean by "a reasonable way".
On input will have to check whether IEEE representation
exceeds implementation-specific limits from <climit> and
<cfloat>; if it does, throw domain_error or do something
similar; also, for output need to check whether the number to
be written is within the respective (single- or double-) IEEE
limit. Long story short, lots of work to do..

If the absolute value you read is too big for your internal
representation, it's obviously an error (setting failbit would
be the usual behavior). If the absolute value is smaller that
min(), but not zero, however: is it an error, or do you use 0.0?
What if the value you read has more precision than you can
represent? Is it an error, or do you round.
 
P

Pavel

James said:
The problem isn't just the doing. The problem is specifying
what you mean by "a reasonable way".
I meant defining, not implementing. I was answering to your statement above.
If the absolute value you read is too big for your internal
representation, it's obviously an error (setting failbit would
be the usual behavior).
I have already decided on a non-stream-related API, so there will be
nothing to set the failbit on. For signaling errors, I am leaning to
simple return codes but did not completely rejected the idea of throwing
std::domain_error yet.

If the absolute value is smaller that
min(), but not zero, however: is it an error, or do you use 0.0?
What if the value you read has more precision than you can
represent? Is it an error, or do you round.
Will round. It is not my purpose to reject as many inputs as possible;
it is the other way around.

Regards
-Pavel
 
J

Jerry Coffin

[ ... ]
Java uses a binary format: for floating point, it's basically
IEEE, output in big endian, the same as XDR, but (I think)
without the alignment requirements. (Formally, I don't think
that IEEE specifies where in the "word" the bits for the sign,
exponant and mantissa are placed, but practically, I've never
heard of an implementation where the order---from high bit to
low---wasn't sign, exponent, mantissa.)

Yes, IEEE does specify that ordering. The idea was that if you took
floating point numbers and treated them as integers of the same size,
comparisons would produce correct ordering. IIRC, this assumes you don't
have any NaNs or such (which are normally only produced internally and
never stored).
 
J

James Kanze

[ ... ]
Java uses a binary format: for floating point, it's basically
IEEE, output in big endian, the same as XDR, but (I think)
without the alignment requirements. (Formally, I don't think
that IEEE specifies where in the "word" the bits for the sign,
exponant and mantissa are placed, but practically, I've never
heard of an implementation where the order---from high bit to
low---wasn't sign, exponent, mantissa.)
Yes, IEEE does specify that ordering. The idea was that if you
took floating point numbers and treated them as integers of
the same size, comparisons would produce correct ordering.

Presumably, supposing signed integers, and 2's complement used
for negative representation. It obviously doesn't work if you
type pun them to an unsigned integral type, and if you aren't
using 2's complement, and trap negative zeros, then you can't
even attempt it with signed integers. (And that's without
considering what's supposed to happen on a system where ints are
36 bits, or more. But then, I doubt such systems would support
IEEE anyway.)

In practice, of course, any new architecture will be byte
addressed, with access sizes of 8, 16, 32, and 64 bits (and
maybe 128 or 256 in the future---but always 8*2^n bits), will
use 2's complement integers and IEEE arithmetic. This isn't the
case for existing architectures, of course, but those which
deviate from this model all predate IEEE, and have their own
floating point format. (I know of at least three, in addition
to IEEE, which are currently being used, on machines still being
built and sold.) And on such machines, I can't imagine anyone
ordering the bits other than sign, exponent, mantissa (from high
byte to low), although some early Intel implementations (e.g.
the Microsoft C compiler) didn't do this when the 8087 floating
point unit was used---if you type punned a float to a long, the
sign bit would be right in the middle. (Microsoft later
recognized their mistake, and changed the byte order of a long
to conform with Intel's view of things. Which doubtlessly
caused much fun for those who'd been writing memory images of
data to disk.)
 
R

Richard Herring

In message
[ ... ]
Java uses a binary format: for floating point, it's basically
IEEE, output in big endian, the same as XDR, but (I think)
without the alignment requirements. (Formally, I don't think
that IEEE specifies where in the "word" the bits for the sign,
exponant and mantissa are placed, but practically, I've never
heard of an implementation where the order---from high bit to
low---wasn't sign, exponent, mantissa.)
Yes, IEEE does specify that ordering. The idea was that if you
took floating point numbers and treated them as integers of
the same size, comparisons would produce correct ordering.

Presumably, supposing signed integers, and 2's complement used
for negative representation. It obviously doesn't work if you
type pun them to an unsigned integral type, and if you aren't
using 2's complement, and trap negative zeros, then you can't
even attempt it with signed integers. (And that's without
considering what's supposed to happen on a system where ints are
36 bits, or more. But then, I doubt such systems would support
IEEE anyway.)

In practice, of course, any new architecture will be byte
addressed, with access sizes of 8, 16, 32, and 64 bits (and
maybe 128 or 256 in the future---but always 8*2^n bits), will
use 2's complement integers and IEEE arithmetic. This isn't the
case for existing architectures, of course, but those which
deviate from this model all predate IEEE, and have their own
floating point format. (I know of at least three, in addition
to IEEE, which are currently being used, on machines still being
built and sold.) And on such machines, I can't imagine anyone
ordering the bits other than sign, exponent, mantissa (from high
byte to low), although some early Intel implementations (e.g.
the Microsoft C compiler) didn't do this when the 8087 floating
point unit was used---if you type punned a float to a long, the
sign bit would be right in the middle.

VAX-11 floating formats do something similar: the sign and exponent
start at the MSB end of byte 1, not byte 0, and are followed by the
fraction in the rest of byte 0 followed by bytes 3, 2, 5, 4, .... It's
not entirely obvious, because DEC's diagrams of data layout are ordered
by 2-byte word and have to be read right to left.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top