implementation issue ..

[ma740988]

I'm faced with endian issues during communication across different
platforms.

So I've recommendations/tips (from one of these newsgroups - cant
recall which) on an approach (or two or three) that seems reasonable
and is based on issues that were encountered and how to handle them in
this situation.

One of which - the one I'm currently investigating - is as follows:

For discussion purposes consider atruct that'll be transmitted between
two plaforms (a and b). Platform a is little endian. b is big endian.
A sample struct is akin to.

//
typedef double tag_word;

struct t2_data
{
tag_word tag_word_;
bool lvt_enable : 1;
bool range : 1;
bool valid_mst : 1;
// more
};

struct t2_status
{
tag_word tag_word_;
bool lvt_status : 1;
bool range_status : 1;
bool msg_is_valid : 1;
unsigned int : 2;
unsigned int : 4;
// more
};

struct msg_header
{
unsigned int msg_size;
unsigned int msg_id;
unsigned int msg_count;
unsigned int msg_chksum;
};


// later
struct platform_a_to_b // little endian (platform a) to big endian
(platform b)
{
msg_header header;
t2_data t2_data_;
};


// later
struct platform_b_to_a // big endian (platform b) to little endian
(platform a)
{
msg_header header;
t2_status t2_stat;
};


The approach under investigation involves allocation of a buffer large
enough to hold the message being transmitted/received. So now: For
the receipt of message (IOW a receives from b or vice versa), I'll:

1. Allocate an unsigned char buffer large enough to hold the message.

2. Read the message into the buffer.
3. Convert the endianism of the data(if necessary).
4. Load each member of the model struct from the buffer.

Now I'm told that transmittal of messages is the reverse order of items
1 to 4, but that puzzles me. Why? With a 'few' tweaks heres the
reverse order as I see it:
1a. Allocate a unsinged char buffer large enough to hold the message.
This assumes a buffer hasn't already been allocated.
2a. Load each member of the model struct INTO (in this case lets take
the information from the struct and put into the buffer) the buffer.
3a. Convert the endianism of the data.

Item 2a seems flawed to me. The idea that I could just dump a struct
ino a byte buffer and send it across a wire seems like a bad idea, for
obvious reasons. Namely, I'm not guaranteed that the two plarforms
pack structs the same way and that would cause more headaches than it
will solve. That said, I'm puzzled about how this 'byte' buffer
approach works.

For those who've used this approach and have code that works, if its
possible to share (conversion code + sample struct) I'd appreaciate it.
Perusing _ACTUAL_ code assists me in 'getting my head around things'.


One other thing:
While it can be said that the requirements per the number of bytes for
a field is 'fixed' in a document, the types, - namely bool - specified
in the structs above need revisiting. I've been advised to follow
protocol below when dealing with structs and bit fields:

1. use only double, unsigned int (32 bits) and unsigned char.
2. if I need bool take and int, if I have to consider platforms using
different bool type sizes.
3. use double int and char (arrays) in that order.
4. make char arr's size a multiple of 4. (better multiple of 8).

I'll be re-visiting/investigating this approach.

 

[Ron Natalie]

For discussion purposes consider atruct that'll be transmitted between
two plaforms (a and b). Platform a is little endian. b is big endian.
A sample struct is akin to.

Bitfield packing is implementation defined. I've seen the order vary
even machines of the same byte ordering.

4. Load each member of the model struct from the buffer.

I guess that depends on what you mean by this.

3a. Convert the endianism of the data.

I would combine this with step with 4 as I'm not sure it's valid to try
to do them separately.

Item 2a seems flawed to me. The idea that I could just dump a struct
ino a byte buffer and send it across a wire seems like a bad idea, for
obvious reasons. Namely, I'm not guaranteed that the two plarforms
pack structs the same way and that would cause more headaches than it
will solve. That said, I'm puzzled about how this 'byte' buffer
approach works.

You're correct, but it's all dealt with by step 4 above. If you always
convert a double to 8 bytes in a known encoding, and the bools into a
certain sequence of bytes, then it should work...you're essentially
not just "copying the struct to the bytestream" but enforcing your own
encoding on it.

 

[Larry I Smith]

I'm faced with endian issues during communication across different
platforms.

So I've recommendations/tips (from one of these newsgroups - cant
recall which) on an approach (or two or three) that seems reasonable
and is based on issues that were encountered and how to handle them in
this situation.

One of which - the one I'm currently investigating - is as follows:

For discussion purposes consider atruct that'll be transmitted between
two plaforms (a and b). Platform a is little endian. b is big endian.
A sample struct is akin to.

//
typedef double tag_word;

struct t2_data
{
tag_word tag_word_;
bool lvt_enable : 1;
bool range : 1;
bool valid_mst : 1;
// more
};

struct t2_status
{
tag_word tag_word_;
bool lvt_status : 1;
bool range_status : 1;
bool msg_is_valid : 1;
unsigned int : 2;
unsigned int : 4;
// more
};

struct msg_header
{
unsigned int msg_size;
unsigned int msg_id;
unsigned int msg_count;
unsigned int msg_chksum;
};


// later
struct platform_a_to_b // little endian (platform a) to big endian
(platform b)
{
msg_header header;
t2_data t2_data_;
};


// later
struct platform_b_to_a // big endian (platform b) to little endian
(platform a)
{
msg_header header;
t2_status t2_stat;
};


The approach under investigation involves allocation of a buffer large
enough to hold the message being transmitted/received. So now: For
the receipt of message (IOW a receives from b or vice versa), I'll:

1. Allocate an unsigned char buffer large enough to hold the message.

2. Read the message into the buffer.
3. Convert the endianism of the data(if necessary).
4. Load each member of the model struct from the buffer.

Now I'm told that transmittal of messages is the reverse order of items
1 to 4, but that puzzles me. Why? With a 'few' tweaks heres the
reverse order as I see it:
1a. Allocate a unsinged char buffer large enough to hold the message.
This assumes a buffer hasn't already been allocated.
2a. Load each member of the model struct INTO (in this case lets take
the information from the struct and put into the buffer) the buffer.
3a. Convert the endianism of the data.

Item 2a seems flawed to me. The idea that I could just dump a struct
ino a byte buffer and send it across a wire seems like a bad idea, for
obvious reasons. Namely, I'm not guaranteed that the two plarforms
pack structs the same way and that would cause more headaches than it
will solve. That said, I'm puzzled about how this 'byte' buffer
approach works.

For those who've used this approach and have code that works, if its
possible to share (conversion code + sample struct) I'd appreaciate it.
Perusing _ACTUAL_ code assists me in 'getting my head around things'.


One other thing:
While it can be said that the requirements per the number of bytes for
a field is 'fixed' in a document, the types, - namely bool - specified
in the structs above need revisiting. I've been advised to follow
protocol below when dealing with structs and bit fields:

1. use only double, unsigned int (32 bits) and unsigned char.
2. if I need bool take and int, if I have to consider platforms using
different bool type sizes.
3. use double int and char (arrays) in that order.
4. make char arr's size a multiple of 4. (better multiple of 8).

I'll be re-visiting/investigating this approach.

Or just send/rcv the data in ascii; or use xml.
If each struct has read/write methods (serialize(), etc)
that write/read the structs contents as text it greatly
eases data xfer across platforms.

Transferring formatted binary data (structs, etc) from
one arch to another is almost never a good idea.

Regards,
Larry

 

[ma740988]

Ron, thanks for the info. Let me make sure I've got my head around
this. Lets assume playform 'a' native tongue is 32 bits. Lets assume
platform 'b's' native tongue is 16 bits.
Same story: a is little endian b is big. Lets further assume we
have a

struct test {
unsinged int some_value;
};

Lets assume // later :
test my_test; my_test.some_value = 655394;

Now a transmits to b, but prior to transmitting. a will need to 'load'
some_value into a byte buffer. ( Now I wish I could draw in usenet
world). We have 4 bytes to represent some_value.
So now:

Byte Buffer
-------------- -------------- --------------
--------------
byte 1 byte 2 byte 3 byte 4
-------------- -------------- --------------
--------------
Endian conversion
Transmit

b receives. b's native tongue is 16 bits. I think I see what needs to
happen here but what would 'b' need to do to map the 4 bytes sent from
a into the test struct? Your answer will perhaps help me 'answer' all
the scenarios I've generated in my head.

 

[Larry I Smith]

Ron, thanks for the info. Let me make sure I've got my head around
this. Lets assume playform 'a' native tongue is 32 bits. Lets assume
platform 'b's' native tongue is 16 bits.
Same story: a is little endian b is big. Lets further assume we
have a

struct test {
unsinged int some_value;
};

Lets assume // later :
test my_test; my_test.some_value = 655394;

Now a transmits to b, but prior to transmitting. a will need to 'load'
some_value into a byte buffer. ( Now I wish I could draw in usenet
world). We have 4 bytes to represent some_value.
So now:

Byte Buffer
-------------- -------------- --------------
--------------
byte 1 byte 2 byte 3 byte 4
-------------- -------------- --------------
--------------
Endian conversion
Transmit

b receives. b's native tongue is 16 bits. I think I see what needs to
happen here but what would 'b' need to do to map the 4 bytes sent from
a into the test struct? Your answer will perhaps help me 'answer' all
the scenarios I've generated in my head.

Read up on the library functions htonl(), htons(), ntohl(),
and ntohs(). They convert 16 and 32 bit ints to/from the
current host's endian and 'network' endian. For example:

on the sending side

uint32_t n = htonl(some_32bit_int); // 'n' now holds the
// 'network' endian version of 'some_32bit_int'
now send 'n' to the remote (put it in the sendbuffer, etc)

on the receiving side

uint32_t n = read 'n' from the network (from the rcvbuffer, etc)
uint32_t some_32bit_int = ntohl(n); // 'some_32bit_int' is now
// the correct endian for THIS host

Regards,
Larry

 

[ma740988]

Read up on the library functions htonl(), htons(), ntohl(),
and ntohs(). They convert 16 and 32 bit ints to/from the
current host's endian and 'network' endian. For example:

on the sending side

uint32_t n = htonl(some_32bit_int); // 'n' now holds the
// 'network' endian version of 'some_32bit_int'
now send 'n' to the remote (put it in the sendbuffer, etc)

on the receiving side

uint32_t n = read 'n' from the network (from the rcvbuffer, etc)
uint32_t some_32bit_int = ntohl(n); // 'some_32bit_int' is now
// the correct endian for THIS host

Regards,
Larry

Interesting that you brought up ntohl and htonl. When I opted to use
them I was told:

These functions do not always swap endian-ness and are _NOT_ always
portable. Interesting. The alternatives:

// 1
inline void endian_swap(unsigned short& x)
{
x = (x>>8) |
(x<<8);
}

inline void endian_swap(unsigned int& x)
{
x = (x>>24) |
((x<<8) & 0x00FF0000) |
((x>>8) & 0x0000FF00) |
(x<<24);
}

// __int64 for MSVC, "long long" for gcc
inline void endian_swap(unsigned __int64& x)
{
x = (x>>56) |
((x<<40) & 0x00FF000000000000) |
((x<<24) & 0x0000FF0000000000) |
((x<<8) & 0x000000FF00000000) |
((x>>8) & 0x00000000FF000000) |
((x>>24) & 0x0000000000FF0000) |
((x>>40) & 0x000000000000FF00) |
(x<<56);
}


/// 2 ( I prefer this route)

#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))

void ByteSwap(unsigned char * b, int n)
{
register int i = 0;
register int j = n-1;
while (i<j)
{
std::swap(b, b[j]);
i++, j--;
}
}

 

[=?ISO-8859-15?Q?Juli=E1n?= Albo]

Ron, thanks for the info. Let me make sure I've got my head around
this. Lets assume playform 'a' native tongue is 32 bits. Lets assume
platform 'b's' native tongue is 16 bits.

If you want binary transfers of data, forget native tongue. You must decide
if you want to use 16, 32 or whatever number of bits data size. Use fixed
size data types if your compiler has it, or use a type that the standard
guarantees have at least that size, and check at runtime that values out of
range are not used.

And you can also forget endianess, just always transfer the numbers to a
unsigned char array byte by byte using >> and & operators. That way you can
use the same code in both platforms without using conditional compilation.

Take care with signed numbers, that may require additional measures if you
want platform independence in that aspect.

 

[ma740988]

Bye the way. My REAL issue is trying to understand how the buffer of
bytes approach solves the current problem. IOW. Notwithstanding the
fact that the endian issue is what drove me to this, I'm more
interested in undertanding how a buffer of bytes - on a much LARGER
scale - assists when dealing with communications across platforms whose
native tongue (16 bit ints on one, 32 bits on the other) might might be
different. Even more interesting is the fact that the native tongue
(32 bit integers) might be the same yet the compiler could wreak
havoc.
To be on the safe side. Again just throw in a buffer of bytes and ..
well!!

 

[ma740988]

Bye the way. My REAL issue is trying to understand how the buffer of
bytes approach solves the current problem. IOW. Notwithstanding the
fact that the endian issue is what drove me to this, I'm more
interested in undertanding how a buffer of bytes - on a much LARGER
scale - assists when dealing with communications across platforms whose
native tongue (16 bit ints on one, 32 bits on the other) might might be
different. Even more interesting is the fact that the native tongue
(32 bit integers) might be the same yet the compiler could wreak
havoc.
To be on the safe side. Again just throw in a buffer of bytes and ..
well!!

 

[Larry I Smith]

Interesting that you brought up ntohl and htonl. When I opted to use
them I was told:

These functions do not always swap endian-ness and are _NOT_ always
portable. Interesting. The alternatives:

I don't believe that's the case. Their whole purpose is to
handle the endian problem.

Anyway, as I said in my other post, XML and the likes are
most often used these days to xfr data host-to-host because
that elminates the numerous problems associated with binary
xfrs.

// 1
inline void endian_swap(unsigned short& x)
{
x = (x>>8) |
(x<<8);
}

inline void endian_swap(unsigned int& x)
{
x = (x>>24) |
((x<<8) & 0x00FF0000) |
((x>>8) & 0x0000FF00) |
(x<<24);
}

// __int64 for MSVC, "long long" for gcc
inline void endian_swap(unsigned __int64& x)
{
x = (x>>56) |
((x<<40) & 0x00FF000000000000) |
((x<<24) & 0x0000FF0000000000) |
((x<<8) & 0x000000FF00000000) |
((x>>8) & 0x00000000FF000000) |
((x>>24) & 0x0000000000FF0000) |
((x>>40) & 0x000000000000FF00) |
(x<<56);
}


/// 2 ( I prefer this route)

#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))

void ByteSwap(unsigned char * b, int n)
{
register int i = 0;
register int j = n-1;
while (i<j)
{
std::swap(b, b[j]);
i++, j--;
}
}

 

[ma740988]

You must decide
if you want to use 16, 32 or whatever number of bits data size. Use
fixed
size data types if your compiler has it, or use a type that the
standard
guarantees have at least that size, and check at runtime that values
out of
range are not used.

Ah!!! This is perhaps where the encoding factors in with Ron's post.
With regard to my earlier post, where I higlighted
test my_test; my_test.some_value = 655394;

I could do.
byte 0 ______ byte 2 ______ byte 3 ______ byte 4 ______
byte 5
start ___________________ 655349 ____________________ stop

Where start and stop a char or ........ The premise: The
sender/receiver upon receipt of the data knows when to start and stop
(loosely speaking).
measures if you
want platform independence in that aspect.

Got it!!

Thanks

 

[ma740988]

Thanks. I'll look into the XML and ascii.. Add another approach to my
arsenal if/most likely when I get confused, I'll be back

 

[Andrew McDonagh]

(e-mail address removed) wrote:

snipped


Anyway, as I said in my other post, XML and the likes are
most often used these days to xfr data host-to-host because
that elminates the numerous problems associated with binary
xfrs.

snipped

Whilst its true that some middleware technologies use XML as the data
transfer type (SOAP), I would not say its 'most'.

XML as a means of marshaling data for transmission greatly increases the
total amount of data transfered - as everything is a String.

CORBA or ICE would be better alternative for the OP to consider seeing
as there is communicates between two (or more) different types of
machines. These middleware technologies are designed for this very
problem. They also have the benefit that either side can be written in
any number programming languages.

 

[Larry I Smith]

snipped

Whilst its true that some middleware technologies use XML as the data
transfer type (SOAP), I would not say its 'most'.

XML as a means of marshaling data for transmission greatly increases the
total amount of data transfered - as everything is a String.

CORBA or ICE would be better alternative for the OP to consider seeing
as there is communicates between two (or more) different types of
machines. These middleware technologies are designed for this very
problem. They also have the benefit that either side can be written in
any number programming languages.

Well, there are many xfr tech's. CORBA is a PITA and often
requires proprietary solutions (we used to use ORBIX from IONA).
The nice thing about XML is that it's platfrom portable, is easy
to use, can easily be done with C/C++/Java, has free implemenations
on most OS'es, etc, etc.

There is no 'silver bullet', but these days many companies are
choosing XML for multi-platform data exchange - mine did.

Regards,
Larry