C++ streams and serialization

[Adam H. Peterson]

I have an application that uses C++ file streams for storing the state
of complex objects to be later read in, ideally reconstructing the state
of the object as when it was serialized. One issue I'm encountering,
though, is that when I serialize a double and then read it back in, I've
lost precision on the magnitude of 10^-6. Is there a way to tell a
stream to write a double (or any number format) to whatever precision is
required to recover the same number later?

Or should I be doing serialization another way? (I'm hoping I can do it
with streams. I want my data files to be portable, and it's valuable to
be able to look at them in an editor and see what's going on.)

Thanks in advance for any help.

Adam H. Peterson

 

[John Carson]

I have an application that uses C++ file streams for storing the state
of complex objects to be later read in, ideally reconstructing the
state of the object as when it was serialized. One issue I'm
encountering, though, is that when I serialize a double and then read
it back in, I've lost precision on the magnitude of 10^-6. Is there
a way to tell a stream to write a double (or any number format) to
whatever precision is required to recover the same number later?

Or should I be doing serialization another way? (I'm hoping I can do
it with streams. I want my data files to be portable, and it's
valuable to be able to look at them in an editor and see what's going
on.)

Thanks in advance for any help.

Adam H. Peterson

I am not an expert in this, but a double on Windows is accurate to 15-16
decimal places. If you set the precision of the output stream to some number
greater than this, then the number written out appears to be successfully
recovered, e.g.,

int main()
{
ofstream out("file.txt");
// set precision of output stream to 20
out.precision(20);
// more precision than a double can handle
double x = 9.0839879879798689898;
out << x << endl;
out.close();

ifstream in("file.txt");
double y;
in >> y;
if (x==y)
cout << "success\n";
else
cout << "failure\n";
in.close();
return 0;
}

 

[Thomas Matthews]

I have an application that uses C++ file streams for storing the state
of complex objects to be later read in, ideally reconstructing the state
of the object as when it was serialized. One issue I'm encountering,
though, is that when I serialize a double and then read it back in, I've
lost precision on the magnitude of 10^-6. Is there a way to tell a
stream to write a double (or any number format) to whatever precision is
required to recover the same number later?

Or should I be doing serialization another way? (I'm hoping I can do it
with streams. I want my data files to be portable, and it's valuable to
be able to look at them in an editor and see what's going on.)

Thanks in advance for any help.

Adam H. Peterson

For best portability, convert your numbers to text format. Text format
usually is independent of machine characteristics; including when the
compiler libraries or OS libraries change.

Try reading these sections of the FAQ:
http://www.parashift.com/c++-faq-lite/serialization.html
http://www.parashift.com/c++-faq-lite/newbie.html#faq-29.16
http://www.eskimo.com/~scs/c-faq/s14.html
http://groups.google.com/groups?hl=...=Google+Search&meta=group=comp.lang.c%2B%2B.*

If you are storing the double variables and retrieving them without
any conversion, there _should_ be no loss of precision. But then
there are many numbers that cannot be precisely represented by
a fixed real number format.

Also search the web for "IEEE floating point format".

--
Thomas Matthews

C++ newsgroup welcome message:
http://www.slack.net/~shiva/welcome.txt
C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.learn.c-c++ faq:
http://www.raos.demon.uk/acllc-c++/faq.html
Other sites:
http://www.josuttis.com -- C++ STL Library book

 

[Adam H. Peterson]

I am not an expert in this, but a double on Windows is accurate to 15-16

decimal places. If you set the precision of the output stream to some number
greater than this, then the number written out appears to be successfully
recovered, e.g.,

Thank you for the response. I am actually working under Linux on x86,
but I assume the precision is the same.

I was hoping for a more general (and possibly portable) solution, so
that I could store and retrieve such numbers without knowing beforehand
their intrinsic precision. Is there a way to do this? Perhaps using
std::numerical_limits<>?

Ideally, I would like to be able to do this with a template type T so
that without knowing more about it, I could store and retrieve it
without accuracy loss. But this at least looks like a solution to my
current dilemma. Thank you.

Adam H. Peterson

 

[Adam H. Peterson]

For best portability, convert your numbers to text format. Text format

usually is independent of machine characteristics; including when the
compiler libraries or OS libraries change.

Try reading these sections of the FAQ:
http://www.parashift.com/c++-faq-lite/serialization.html
http://www.parashift.com/c++-faq-lite/newbie.html#faq-29.16
http://www.eskimo.com/~scs/c-faq/s14.html
http://groups.google.com/groups?hl=...=Google+Search&meta=group=comp.lang.c%2B%2B.*


If you are storing the double variables and retrieving them without
any conversion, there _should_ be no loss of precision. But then
there are many numbers that cannot be precisely represented by
a fixed real number format.

Also search the web for "IEEE floating point format".

Portability is one of the reason I'm using text streams. (Other reasons
are transparency and clarity.) So I thought that that was what I was doing.

It appears to me that I am in fact getting an accuracy loss in this
process. This program appears to demonstrate that this occurs:

#include <iostream>
#include <sstream>
#include <cmath>

int signed main() {
double sqrt2=std::sqrt(2.0);
std:stringstream ot;
ot << sqrt2;
std::istringstream it(ot.str());
double sqrt2io;
it >> sqrt2io;
std::cout << (sqrt2-sqrt2io) << std::endl;
}

// Output on my machine (GCC 3.2.2, RH Linux 9) is:
// 3.56237e-06
// I would expect 0 if the same number was being read.

But this is the naive approach to it and perhaps I'm missing some
subtleties. (If so, I would love to be enlightened.)

On a side note, I _know_ I read the faq cover to cover, but I'm positive
this serialization stuff wasn't in there. I assume it was added since
(about six years ago), which I suppose is good, meaning it's adapting to
new needs. I guess I better rebrowse it.

But it doesn't look like the faq (or at least this page of it) addresses
my precision issues.

Anyway, thank you for your time to respond.

Adam H. Peterson

 

[John Carson]

Thank you for the response. I am actually working under Linux on x86,
but I assume the precision is the same.

I was hoping for a more general (and possibly portable) solution, so
that I could store and retrieve such numbers without knowing
beforehand their intrinsic precision. Is there a way to do this?
Perhaps using std::numerical_limits<>?

Ideally, I would like to be able to do this with a template type T so
that without knowing more about it, I could store and retrieve it
without accuracy loss. But this at least looks like a solution to my
current dilemma. Thank you.

Adam H. Peterson

std::numeric_limits<T>::digits10

seems to be what you want. Note that setting a precision above that
applicable to a particular type doesn't seem to cause any problems, so you
can always allow a "margin for error" in this respect.

 

[Adam H. Peterson]

If anyone is interested, I think I've found a general solution. Instead
of using:

double x=whatever;
file << x;

.... I use:

double x=whatever;
file << boost::lexical_cast<std::string>(x);

.... it seems to store enough digits to reconstruct the number. (I get a
difference of zero in my test program I posted earlier.

Of course, you have to have boost installed for this solution, but
that's not a problem for me.

Thanks for the help, those who posted or read my musings.

Adam H. Peterson

 

[djfitz]

Using precision on the stream is really what you want. Setting precision to a really high number will give you enough digits so you won't get round-off for your floating point type. You can also use a stream manipulator to set the precision:

ofstream out("file.txt");
double x = 9.0839879879798689898;
// set precision of output stream to 20
out << setprecision(20) << x << endl;
out.close();

Using gcc 4.0, I get 0 for the result in the sample code in this post when I use the setprecision manipulator.

This simplifies the code a lot over explicitly converting to a string and back.