converting char to float (reading binary data from file)

I

itdevries

Hi,
I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?
regards,
Igor

float floatRead = 0;

UINT32* ptr = (UINT32 *) (&floatRead);

int offset = 0;

for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;
};
 
M

Michael DOUBEZ

itdevries a écrit :
I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?

I suppose charBuf is of the type char[] ? In this case 'charBuf[ii] <<
offset' is 0 as soon as offset>=8 so you get only 0s.
float floatRead = 0;

UINT32* ptr = (UINT32 *) (&floatRead);

int offset = 0;

for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;
};

Your use of bitwise operator looks clumsy unless you have some logic to
handle different byte ordering.

What's wrong with
memcpy(&floatRead,charBuf+startInd,sizeof(floatRead));
?
 
I

itdevries

itdevries a écrit :
I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?

I suppose charBuf is of the type char[] ? In this case 'charBuf[ii] <<
offset' is 0 as soon as offset>=8 so you get only 0s.
float floatRead = 0;
UINT32*    ptr  = (UINT32 *) (&floatRead);
int offset   = 0;
for (int ii=startInd; ii<=endInd; ii++){
  *ptr |= (charBuf[ii] << offset);
  offset  += 8;
};

Your use of bitwise operator looks clumsy unless you have some logic to
handle different byte ordering.

What's wrong with
memcpy(&floatRead,charBuf+startInd,sizeof(floatRead));
?

thanks, that works... however, I don't understand what's wrong with my
original code. any ideas?
Igor
 
M

Michael DOUBEZ

itdevries a écrit :
itdevries a écrit :
float floatRead = 0;
UINT32* ptr = (UINT32 *) (&floatRead);
int offset = 0;
for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;
};

I don't understand what's wrong with my
original code. any ideas?

Unrolling your loop:
*ptr |= (charBuf[startInd+0] << 0);
*ptr |= (charBuf[startInd+1] << 8);
*ptr |= (charBuf[startInd+2] << 16);
*ptr |= (charBuf[startInd+3] << 24);

Becomes (as I have mentionned):
*ptr |= charBuf[startInd];
*ptr |= 0;
*ptr |= 0;
*ptr |= 0;

Because you shift-left a char more times than its size in bits (usually
8) so it becomes 0.

Example:
char x=0xBF;
assert( (x<< 8) == 0);
assert( (x<<12) == 0);
assert( (x<<42) == 0);
 
I

itdevries

itdevries a écrit :
itdevries a écrit :
float floatRead = 0;
UINT32*    ptr  = (UINT32 *) (&floatRead);
int offset   = 0;
for (int ii=startInd; ii<=endInd; ii++){
  *ptr |= (charBuf[ii] << offset);
  offset  += 8;
};
I don't understand what's wrong with my
original code. any ideas?

Unrolling your loop:
*ptr |= (charBuf[startInd+0] <<  0);
*ptr |= (charBuf[startInd+1] <<  8);
*ptr |= (charBuf[startInd+2] << 16);
*ptr |= (charBuf[startInd+3] << 24);

Becomes (as I have mentionned):
*ptr |= charBuf[startInd];
*ptr |= 0;
*ptr |= 0;
*ptr |= 0;

Because you shift-left a char more times than its size in bits (usually
8) so it becomes 0.

Example:
char x=0xBF;
assert( (x<< 8) == 0);
assert( (x<<12) == 0);
assert( (x<<42) == 0);

Thanks so much for taking the time to respond. I understand the logic
you're using and it was one of the initial concerns I had with the
code, however it seemed to work fine for int types (maybe by
coincidence) so I thought it would work for float types as well. One
thing I don't understand however is that when I step through the loop
with the debugger I see the value of the float change even though from
step 2 I thought I'd be doing "*ptr |= 0" which I thought shouldn't
alter the value of the float. that lead me to the conclusion that the
bitwise shift worked differently from what I expected.
Igor
 
J

James Kanze

itdevries a écrit :
I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?

I suppose charBuf is of the type char[] ? In this case 'charBuf[ii] <<
offset' is 0 as soon as offset>=8 so you get only 0s.
float floatRead = 0;
UINT32* ptr = (UINT32 *) (&floatRead);
int offset = 0;
for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;
};

Your use of bitwise operator looks clumsy unless you have some logic to
handle different byte ordering.

What's wrong with
memcpy(&floatRead,charBuf+startInd,sizeof(floatRead));
?
 
J

James Kanze

Hi,
I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?
regards,
Igor

float floatRead = 0;

UINT32* ptr = (UINT32 *) (&floatRead);

int offset = 0;

for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;

};
 
J

James Kanze

itdevries a écrit :
I suppose charBuf is of the type char[] ? In this case
'charBuf[ii] << offset' is 0 as soon as offset>=8 so you get
only 0s.

Since when? That's not what my compiler does, and it's not what
the standard requires. In expressions, char's are promoted to
int's.
float floatRead = 0;
UINT32* ptr = (UINT32 *) (&floatRead);
int offset = 0;
for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;
};

Without seeing the other declarations, it's hard to say. What
is charBuf? Normally, I'd expect it to be an unsigned char*,
otherwise, the integral promotion of charBuf[ii] will extend the
sign, which will definitely not give you the results you want.
(If it's char*, you can always mask out the unwanted bits from
the extension, i.e.:
((charBuf[ ii ] & 0xFF) << offset)
You don't show startInd and endInd either. Obviously, they make
a difference.

When all is said and done, I'd unwind the loop:

*ptr = ((charBuf[ 0 ] & 0xFF) ) ;
*ptr |= ((charBuf[ 0 ] & 0xFF) << 8) ;
*ptr |= ((charBuf[ 0 ] & 0xFF) << 16) ;
*ptr |= ((charBuf[ 0 ] & 0xFF) << 24) ;

Note that you are also assuming that the representation of the
that you are reading more or less corresponds to the internal
representation on your machine (modulo byte order). This isn't
really something you can portably count on. (On the other hand,
doing it really right, in a 100% portable fashion, is relatively
difficult. I'd probably put in some sort of #if and and #error
to catch the case, and only add the complexity if the need
arose.)
Your use of bitwise operator looks clumsy unless you have some
logic to handle different byte ordering.
What's wrong with
memcpy(&floatRead,charBuf+startInd,sizeof(floatRead));
?

Maybe the fact that it doesn't have the correct semantics? (It
might work, sometimes, but it's certainly a risky procedure.)
 
M

Michael DOUBEZ

James Kanze a écrit :
itdevries a écrit :
I suppose charBuf is of the type char[] ? In this case
'charBuf[ii] << offset' is 0 as soon as offset>=8 so you get
only 0s.

Since when? That's not what my compiler does, and it's not what
the standard requires. In expressions, char's are promoted to
int's.

Then, it is my mistake and I am the worse for not checking.
I am always careful to avoid writing code like that.
Maybe the fact that it doesn't have the correct semantics? (It
might work, sometimes, but it's certainly a risky procedure.

More risky than using bitwise operators to build a float ?

Concerning the semantic, I don't see what could be a correct one in this
case since reading a binary flow to build a float is however heretical;
unless you have some kind of standard transform and assuming the
capacities of representation are equivalent across addressed architectures.

What is uncorrect with dumping the memory area representing a POD and
reading it back in the same program ?
 
J

James Kanze

James Kanze a écrit :
itdevries a écrit :
I'm trying to convert some char data I read from a binary
file (using ifstream) to a float type. I've managed to
convert the int types but now I need to do the float types
as well but it doesn't seem to work. The code below is what
I'm trying to use. Anyone see any obvious errors? or have
any hints/pointers?
I suppose charBuf is of the type char[] ? In this case
'charBuf[ii] << offset' is 0 as soon as offset>=8 so you get
only 0s.
Since when? That's not what my compiler does, and it's not
what the standard requires. In expressions, char's are
promoted to int's.
Then, it is my mistake and I am the worse for not checking. I
am always careful to avoid writing code like that.
More risky than using bitwise operators to build a float ?

Yes.

Basically, using the bitwise operators depends on the external
representation being the same as the internal, i.e. they are
both IEEE (or both something else, but I've never seen that
case). In practice, a lot of modern protocols do use IEEE, and
most modern small and medium sized computers also use it
internally, so if you are only targetting Windows and mainstream
Unix, you're safe (and a few judicious #if on the values defined
in <climits>, controlling an #error, can assure that it probably
won't compile on systems which don't use IEEE).

It's ugly, but the alternative is a lot more complex.
(Interestingly, it's not that much slower on my machine. Which
rather surprised me.)
Concerning the semantic, I don't see what could be a correct
one in this case since reading a binary flow to build a float
is however heretical; unless you have some kind of standard
transform and assuming the capacities of representation are
equivalent across addressed architectures.

As mentionned above, a lot of protocols (BER being the only
exception which comes to mind) use IEEE format for floating
point, as do a lot of machines. In this case, reading the value
as if it were the same sized unsigned integer type (uint32_t or
uint64_t, depending on whether you're reading float or double),
followed by some sort of nasty cast, will work.
What is uncorrect with dumping the memory area representing a
POD and reading it back in the same program ?

The fact that there are still significant differences in the
binary representation, particularly with regards to byte order.
If you want to be really correctly, you need something like:

uint32_t tmp ;
operator>>( tmp ) ;
if ( *this ) {
float f = 0.0 ;
if ( (tmp & 0x7FFFFFFF) != 0 ) {
f = ldexp( ((tmp & 0x007FFFFF) | 0x00800000),
(int)((tmp & 0x7F800000) >> 23) - 126 - 24 ) ;
}
if ( (tmp & 0x80000000) != 0 ) {
f = -f ;
}
dest = f ;
}
return *this ;

You still read the type into the unsigned int (using shifts),
but then you extract the fields as they are specified in the
format, and use them to create a float, using standard functions
to exploit their values in a portable fashion. As I've said
previously, this is probably more work than necessary, and until
needed, I'd just go with the ugly cast (and the #if/#error, to
ensure that it doesn't get used when not appropriate).
 
C

coal

James Kanze a écrit :
itdevries a écrit :
I'm trying to convert some char data I read from a binary
file (using ifstream) to a float type. I've managed to
convert the int types but now I need to do the float types
as well but it doesn't seem to work.  The code below is what
I'm trying to use. Anyone see any obvious errors? or have
any hints/pointers?
I suppose charBuf is of the type char[] ? In this case
'charBuf[ii] << offset' is 0 as soon as offset>=8 so you get
only 0s.
Since when?  That's not what my compiler does, and it's not
what the standard requires.  In expressions, char's are
promoted to int's.
Then, it is my mistake and I am the worse for not checking.  I
am always careful to avoid writing code like that.
What's wrong with
memcpy(&floatRead,charBuf+startInd,sizeof(floatRead));
?
Maybe the fact that it doesn't have the correct semantics?  (It
might work, sometimes, but it's certainly a risky procedure.
More risky than using bitwise operators to build a float ?

Yes.

Basically, using the bitwise operators depends on the external
representation being the same as the internal, i.e. they are
both IEEE (or both something else, but I've never seen that
case).  In practice, a lot of modern protocols do use IEEE, and
most modern small and medium sized computers also use it
internally, so if you are only targetting Windows and mainstream
Unix, you're safe (and a few judicious #if on the values defined
in <climits>, controlling an #error, can assure that it probably
won't compile on systems which don't use IEEE).

It's ugly, but the alternative is a lot more complex.
(Interestingly, it's not that much slower on my machine.  Which
rather surprised me.)
Concerning the semantic, I don't see what could be a correct
one in this case since reading a binary flow to build a float
is however heretical; unless you have some kind of standard
transform and assuming the capacities of representation are
equivalent across addressed architectures.

As mentionned above, a lot of protocols (BER being the only
exception which comes to mind) use IEEE format for floating
point, as do a lot of machines.  In this case, reading the value
as if it were the same sized unsigned integer type (uint32_t or
uint64_t, depending on whether you're reading float or double),
followed by some sort of nasty cast, will work.
What is uncorrect with dumping the memory area representing a
POD and reading it back in the same program ?

The fact that there are still significant differences in the
binary representation, particularly with regards to byte order.
If you want to be really correctly, you need something like:

    uint32_t            tmp ;
    operator>>( tmp ) ;
    if ( *this ) {
        float               f = 0.0 ;
        if ( (tmp & 0x7FFFFFFF) != 0 ) {
            f = ldexp( ((tmp & 0x007FFFFF) | 0x00800000),
                       (int)((tmp & 0x7F800000) >> 23) - 126 - 24 ) ;
        }
        if ( (tmp & 0x80000000) != 0 ) {
            f = -f ;
        }
        dest = f ;
    }
    return *this ;

You still read the type into the unsigned int (using shifts),
but then you extract the fields as they are specified in the
format, and use them to create a float, using standard functions
to exploit their values in a portable fashion.  As I've said
previously, this is probably more work than necessary, and until
needed, I'd just go with the ugly cast (and the #if/#error, to
ensure that it doesn't get used when not appropriate).



In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:

#ifndef BOOST_ARCHIVE_BASIC_BINARY_OPRIMITIVE_HPP
#define BOOST_ARCHIVE_BASIC_BINARY_OPRIMITIVE_HPP

// MS compatible compilers support #pragma once
#if defined(_MSC_VER) && (_MSC_VER >= 1020)
# pragma once
#endif

/////////
1/////////2/////////3/////////4/////////5/////////6/////////7/////////8
// basic_binary_oprimitive.hpp

// (C) Copyright 2002 Robert Ramey - http://www.rrsd.com .
// Use, modification and distribution is subject to the Boost Software
// License, Version 1.0. (See accompanying file LICENSE_1_0.txt or
copy at
// http://www.boost.org/LICENSE_1_0.txt)

// See http://www.boost.org for updates, documentation, and revision
history.

// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attempt to
// convert to any canonical form.

// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

#include <iosfwd>
#include <cassert>
#include <locale>
#include <cstddef> // size_t
#include <streambuf> // basic_streambuf
#include <string>

#include <boost/config.hpp>
#if defined(BOOST_NO_STDC_NAMESPACE)
namespace std{
using ::size_t;
} // namespace std
#endif

#include <boost/cstdint.hpp>
//#include <boost/limits.hpp>
//#include <boost/io/ios_state.hpp>
#include <boost/scoped_ptr.hpp>
#include <boost/throw_exception.hpp>

#include <boost/archive/basic_streambuf_locale_saver.hpp>
#include <boost/archive/archive_exception.hpp>
#include <boost/archive/detail/auto_link_archive.hpp>
#include <boost/serialization/is_bitwise_serializable.hpp>
#include <boost/mpl/placeholders.hpp>
#include <boost/serialization/array.hpp>
#include <boost/archive/detail/abi_prefix.hpp> // must be the last
header

namespace boost {
namespace archive {

/////////////////////////////////////////////////////////////////////////
// class basic_binary_oprimitive - binary output of prmitives

template<class Archive, class Elem, class Tr>
class basic_binary_oprimitive
{
#ifndef BOOST_NO_MEMBER_TEMPLATE_FRIENDS
friend class save_access;
protected:
#else
public:
#endif
std::basic_streambuf<Elem, Tr> & m_sb;
// return a pointer to the most derived class
Archive * This(){
return static_cast<Archive *>(this);
}
boost::scoped_ptr<std::locale> archive_locale;
basic_streambuf_locale_saver<Elem, Tr> locale_saver;

// default saving of primitives.
template<class T>
void save(const T & t)
{
save_binary(& t, sizeof(T));
}

/////////////////////////////////////////////////////////
// fundamental types that need special treatment

// trap usage of invalid uninitialized boolean which would
// otherwise crash on load.
void save(const bool t){
int i = t;
assert(0 == i || 1 == i);
save_binary(& t, sizeof(t));
}
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const std::string &s);
#ifndef BOOST_NO_STD_WSTRING
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const std::wstring &ws);
#endif
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const char * t);
BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
save(const wchar_t * t);

BOOST_ARCHIVE_OR_WARCHIVE_DECL(void)
init();

BOOST_ARCHIVE_OR_WARCHIVE_DECL(BOOST_PP_EMPTY())
basic_binary_oprimitive(
std::basic_streambuf<Elem, Tr> & sb,
bool no_codecvt
);
BOOST_ARCHIVE_OR_WARCHIVE_DECL(BOOST_PP_EMPTY())
~basic_binary_oprimitive();
public:

// we provide an optimized save for all fundamental types
// typedef serialization::is_bitwise_serializable<mpl::_1>
// use_array_optimization;
// workaround without using mpl lambdas
struct use_array_optimization {
template <class T>
struct apply : public serialization::is_bitwise_serializable<T>
{};
};


// the optimized save_array dispatches to save_binary
template <class ValueType>
void save_array(serialization::array<ValueType> const& a, unsigned
int)
{
save_binary(a.address(),a.count()*sizeof(ValueType));
}

void save_binary(const void *address, std::size_t count);
};

template<class Archive, class Elem, class Tr>
inline void
basic_binary_oprimitive<Archive, Elem, Tr>::save_binary(
const void *address,
std::size_t count
){
//assert(
//
static_cast<std::size_t>((std::numeric_limits<std::streamsize>::max)
()) >= count
//);
// note: if the following assertions fail
// a likely cause is that the output stream is set to "text"
// mode where by cr characters recieve special treatment.
// be sure that the output stream is opened with ios::binary
//if(os.fail())
//
boost::throw_exception(archive_exception(archive_exception::stream_error));
// figure number of elements to output - round up
count = ( count + sizeof(Elem) - 1)
/ sizeof(Elem);
std::streamsize scount = m_sb.sputn(
static_cast<const Elem *>(address),
count
);
if(count != static_cast<std::size_t>(scount))

boost::throw_exception(archive_exception(archive_exception::stream_error));
//os.write(
// static_cast<const BOOST_DEDUCED_TYPENAME OStream::char_type
*>(address),
// count
//);
//assert(os.good());
}

} //namespace boost
} //namespace archive

#include <boost/archive/detail/abi_suffix.hpp> // pop pragams

#endif // BOOST_ARCHIVE_BASIC_BINARY_OPRIMITIVE_HPP




I guess it boils down to a call of basic_streambuf<>::sputn(). Is
that any
better IYO than memcpy?

Brian Wood
Ebenezer Enterprises
www.webEbenezer.net
 
J

James Kanze

On May 22, 1:58 am, James Kanze <[email protected]> wrote:

[...]
In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:

Note however:

[...]
// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attempt to
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.

In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.
 
G

gpderetta

On May 22, 1:58 am, James Kanze <[email protected]> wrote:

    [...]
In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types.  Here is a copy
of a file that is involved:

Note however:

    [...]
// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects.  It makes no attempt to
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options.  If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.

In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.

Modulo what is guaranteed by the compiler/platform ABI, I guess.

In particular, the Boost.Serialization binary format is primarily used
by Boost.MPI (which obviously is a wrapper around MPI) for inter
process communication. I think that the idea is that the MPI layer
will take care of marshaling between peers and thus resolve any
representation difference. I think that in practice most (but not all)
MPI implementations just assume that peers use the same layout format
(i.e. same CPU/compiler/OS) and just network copy bytes back and
forward.

In a sense the distributed program is a logical single run of the same
program even if in practice are different processes running on
different machines, so your observation is still valid
 
J

James Kanze

On May 26, 8:15 pm, (e-mail address removed) wrote:
On May 22, 1:58 am, James Kanze <[email protected]> wrote:
[...]
In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:
Note however:
[...]
// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attempt to
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON
Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.
In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.
Modulo what is guaranteed by the compiler/platform ABI, I guess.

Supposing you can trust them to be stable:). In actual
practice, I've seen plenty of size changes, and I've seen long
and the floating point types change their representation, just
between different versions of the compiler. Not to mention
changes in padding which, at least in some cases depend on
compiler options. (For that matter, on most of the machines I
use, the size of a long depends on compiler options. And is the
sort of option that someone is likely to change in the makefile,
because e.g. they suddenly have to deal with big files.)
In particular, the Boost.Serialization binary format is
primarily used by Boost.MPI (which obviously is a wrapper
around MPI) for inter process communication. I think that the
idea is that the MPI layer will take care of marshaling
between peers and thus resolve any representation difference.
I think that in practice most (but not all) MPI
implementations just assume that peers use the same layout
format (i.e. same CPU/compiler/OS) and just network copy bytes
back and forward.
In a sense the distributed program is a logical single run of
the same program even if in practice are different processes
running on different machines, so your observation is still
valid

If the programs are not running on different machines, what's
the point of marshalling. Just put the objects in shared
memory. Marshalling is only necessary if the data is to be used
in a different place or time (networking or persistency). And a
different place or time means a different machine (sooner or
later, in the case of time).
 
G

gpderetta

On May 26, 8:15 pm, (e-mail address removed) wrote:
    [...]
In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types.  Here is a copy
of a file that is involved:
Note however:
    [...]
// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects.  It makes no attempt to
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON
Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options.  If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.
In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.
Modulo what is guaranteed by the compiler/platform ABI, I guess.

Supposing you can trust them to be stable:).  In actual
practice, I've seen plenty of size changes, and I've seen long
and the floating point types change their representation, just
between different versions of the compiler.  Not to mention
changes in padding which, at least in some cases depend on
compiler options.  (For that matter, on most of the machines I
use, the size of a long depends on compiler options.  And is the
sort of option that someone is likely to change in the makefile,
because e.g. they suddenly have to deal with big files.)

The size of long or that of off_t?
If the programs are not running on different machines, what's
the point of marshalling.  Just put the objects in shared
memory.  Marshalling is only necessary if the data is to be used
in a different place or time (networking or persistency).  And a
different place or time means a different machine (sooner or
later, in the case of time).

Well, MPI programs runs on large clusters of, usually, homogeneous
machines, connected via LAN. The same program will spawn
multiple copies of itself on every machine in the cluster, and every
copy communicates via message passing.
So you have one logical program which is partitioned on multiple
machines. I guess that most MPI implementations do not bother (in fact
I do not even know if it is required by the standard) to convert
messages to a machine agnostic format before sending it to another
peer.
 
J

James Kanze

On May 26, 8:15 pm, (e-mail address removed) wrote:
[...]
In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:
Note however:
[...]
// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attempt to
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON
Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.
In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.
Modulo what is guaranteed by the compiler/platform ABI, I guess.
Supposing you can trust them to be stable:). In actual
practice, I've seen plenty of size changes, and I've seen long
and the floating point types change their representation, just
between different versions of the compiler. Not to mention
changes in padding which, at least in some cases depend on
compiler options. (For that matter, on most of the machines I
use, the size of a long depends on compiler options. And is the
sort of option that someone is likely to change in the makefile,
because e.g. they suddenly have to deal with big files.)
The size of long or that of off_t?

No matter. The point is that they have to compile with
different options, and suddenly long has changed its size.
Well, MPI programs runs on large clusters of, usually,
homogeneous machines, connected via LAN.

That's original. I don't think I've ever seen a cluster of
machines where every system in the cluster was identical. At
the very least, you'll have different versions of Sparc, or PC.
Some of which are 32 bit, and others 64. The cluster may start
out homogeneous, but one of the machines breaks down, and is
replaced with a newer model...

The real question, however, doesn't concern just the machines.
If all of the machines are running a single executable, loaded
from the same shared disk, it will probably work. If not, then
sooner or later, some of the machines will have different
compiles of the program, which may or may not be binary
compatible. In practice, the old rule always holds: identical
copies aren't. (Remember, binary compatibility can be lost just
by changing options, or using a newer version of the compiler.)
The same program will spawn multiple copies of itself on every
machine in the cluster, and every copy communicates via
message passing. So you have one logical program which is
partitioned on multiple machines. I guess that most MPI
implementations do not bother (in fact I do not even know if
it is required by the standard) to convert messages to a
machine agnostic format before sending it to another peer.

Well, I don't know much about that context. In my work, we have
a hetrogeneous network, with PC's under Windows as clients, and
either PC's under Linux or Sparcs under Solaris as servers (and
high level clients). And that more or less corresponds to what
I've seen elswhere as well.
 
G

gpderetta

That's original. I don't think I've ever seen a cluster of
machines where every system in the cluster was identical.

I think that for MPI it is common. Some vendors even sell shrink
wrapped clusters in a box (something like a big closet with thousands
of different computers-on-a-board, each running a different OS image).
Even custom built MPI clusters are fairly homogeneous (i.e. at least
same architecture and OS version).

I think that you work mostly on services applications, while MPI is
more common on high performance computing.
[...]
The real question, however, doesn't concern just the machines.
If all of the machines are running a single executable, loaded
from the same shared disk, it will probably work. If not, then
sooner or later, some of the machines will have different
compiles of the program, which may or may not be binary
compatible. In practice, the old rule always holds: identical
copies aren't. (Remember, binary compatibility can be lost just
by changing options, or using a newer version of the compiler.)

Yep, one need to be careful, but at least with the compiler I use,
options that change the ABI are explicitly documented as such.
Probably a much bigger problem are differences in third party
libraries between machines (i.e. do not expect the layout of objects
you do not control to stay stable).
Well, I don't know much about that context. In my work, we have
a hetrogeneous network, with PC's under Windows as clients, and
either PC's under Linux or Sparcs under Solaris as servers (and
high level clients). And that more or less corresponds to what
I've seen elswhere as well.

Where I work, clusters are composed of hundreds of very different
machines, but all use the same architecture and exact same OS version
(so that we can copy binaries around and not have to worry about
library incompatibilities). We do not use MPI though, but have an in-
house communication framework which does take care of marshaling in a
(mostly) system agnostic format.
 
J

James Kanze

I think that for MPI it is common. Some vendors even sell
shrink wrapped clusters in a box (something like a big closet
with thousands of different computers-on-a-board, each running
a different OS image). Even custom built MPI clusters are
fairly homogeneous (i.e. at least same architecture and OS
version).
I think that you work mostly on services applications, while
MPI is more common on high performance computing.

I realized that much, but I wasn't aware that it was that common
even on high performance computing. The high performance
computing solutions I've seen have mostly involved a lot of
CPU's using the same memory, so marshalling wasn't an issue.
(But I'm not much of an expert in the domain, and I've not seen
that many systems, so what I've seen doesn't mean much.)
[...]
The real question, however, doesn't concern just the machines.
If all of the machines are running a single executable, loaded
from the same shared disk, it will probably work. If not, then
sooner or later, some of the machines will have different
compiles of the program, which may or may not be binary
compatible. In practice, the old rule always holds: identical
copies aren't. (Remember, binary compatibility can be lost just
by changing options, or using a newer version of the compiler.)
Yep, one need to be careful, but at least with the compiler I use,
options that change the ABI are explicitly documented as such.

Lucky guy:). For the most part, what the options actually do
is well documented, and if you understand a bit about what it
means at the hardware level, you can figure out which ones are
safe, and which aren't. But it's far from explicit.

Note that this can be a problem just trying to statically link
libraries; you don't need marshalling at all to get into
trouble. (Or rather: you don't want to have to marshall every
time you pass an std::vector to a function in another module.)
Probably a much bigger problem are differences in third party
libraries between machines (i.e. do not expect the layout of
objects you do not control to stay stable).

That's another problem entirely, and affects linking more than
marshalling. The problem is that compilers may change
representation between versions, etc.
Where I work, clusters are composed of hundreds of very
different machines, but all use the same architecture and
exact same OS version (so that we can copy binaries around and
not have to worry about library incompatibilities). We do not
use MPI though, but have an in- house communication framework
which does take care of marshaling in a (mostly) system
agnostic format.

Yes. We do something more or less like this for the clients:
they're all PC's under Windows, and we use a lowest common
denominator which should work for all Windows systems. Our
machines are geographically distributed, however, so
realistically, ensuring exactly the same version of the OS,
isn't possible.

For the servers, economic considerations result in a decision to
move from Solaris on Sparc to Linux on PC, at least for all but
the most critical systems. Similarly, economic considerations
mean that the entire park won't be upgraded at the same instant.
Are you saying that if a decision comes to upgrade the
architecture, you change all of the machines in a cluster at
once? (But maybe... I can imagine that all of the machines in a
cluster still cost less than one supercomputer. And if you were
using a supercomputer, and wanted to upgrade, you'd change it
all at once. I guess it's just a different mindset.)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,731
Messages
2,569,432
Members
44,832
Latest member
GlennSmall

Latest Threads

Top