Reading binary packet

[rahul]

I am reading a binary packet : 32, 8, 8, 2, 1, 1, 4, 128
I am using the following structure to parse the data:
struct header {
unsigned int a:32;
unsigned int b:8;
unsigned int c:8;
unsigned int d:2;
unsigned int e:1;
unsigned int f:1;
unsigned int g:4;
unsigned int h[4];
};
struct header head;
fread (&head, sizeof head, fp);

My data is getting garbled due to byte ordering differences. The bit
fieds are giving me incorrect values.
I suppose that I will have to use/create some byte ordering routines
( htonl / ntohl ).
I tried reading the data in a char array and then coercing it into
structure pointers so as to avoid multi-byte values but obviously the
data is dependent on how it is written and not how it is read.
Is my approach right or is their a better way to do it?

 

[vippstar]

I am reading a binary packet : 32, 8, 8, 2, 1, 1, 4, 128
I am using the following structure to parse the data:
struct header {
unsigned int a:32;
unsigned int b:8;
unsigned int c:8;
unsigned int d:2;
unsigned int e:1;
unsigned int f:1;
unsigned int g:4;
unsigned int h[4];};

struct header head;
fread (&head, sizeof head, fp);

My data is getting garbled due to byte ordering differences. The bit
fieds are giving me incorrect values.

If that struct was written to the file previously with a call to
fwrite(), it should give you the correct result.

 

[Antoninus Twink]

I am reading a binary packet : 32, 8, 8, 2, 1, 1, 4, 128
I am using the following structure to parse the data:
struct header {
unsigned int a:32;
unsigned int b:8;
unsigned int c:8;
unsigned int d:2;
unsigned int e:1;
unsigned int f:1;
unsigned int g:4;
unsigned int h[4];
};
struct header head;
fread (&head, sizeof head, fp);

fread takes 4 arguments: you need fread (&head, sizeof head, 1, fp);

You asked in another thread about packing. If you don't tell your
compiler to pack this struct then it's very likely to include one byte
of padding between g and h. Use __attribute__((packed)) in gcc.

My data is getting garbled due to byte ordering differences. The bit
fieds are giving me incorrect values.

It's likely that your compiler is ordering the byte containing d,e,f,g
like this:

[ . . . . . . . . ]
_______ _ _ ___
g f e d

So you may need to reorder the bit-fields in your struct.

If you also need to swap the endianness of your 32-bit fields, you can
use a macro like this:

#define CHANGE_END(x) (((x >>24) & 0x000000ff) | ((x >> 8) & 0x0000ff00) \
| ((x << 8) & 0x00ff0000) | ((x <<24) & 0xff000000) )

 

[Richard Tobin]

My data is getting garbled due to byte ordering differences. The bit
fieds are giving me incorrect values.

[/QUOTE]

If that struct was written to the file previously with a call to
fwrite(), it should give you the correct result.

Only if it was written by fwrite() on the same (or a compatible)
system.

If you are transferring data between systems you need to either
mark the data with something indicating the byte order, or use a
fixed byte order. In either case you need to be able to convert the
byte order. You can do this either by manipulating the bytes manually
(with shifts and bitwise operators), or by having a set of alternative
struct definitions for the diffferent orders. Any use of structs
for this of course relies on your knowing that the system will
not mess it up by inserting padding.

-- Richard

 

[Ben Bacarisse]

I am reading a binary packet : 32, 8, 8, 2, 1, 1, 4, 128
I am using the following structure to parse the data:
struct header {
unsigned int a:32;
unsigned int b:8;
unsigned int c:8;
unsigned int d:2;
unsigned int e:1;
unsigned int f:1;
unsigned int g:4;
unsigned int h[4];
};

Packet implies data communication and that usually means you want
to pay attention to writing portable code. Not 100% portable maybe
(you might be happy to assume POSIX for example) but certainly I
suggest you do this more portably than using bit fields and an int
array (you don't know the size of an int).

Often, the best way is to treat the data an array of unsigned char and
write a small set of functions that pull out data of various sizes and
representations just from the offset. These functions would use
shifts to build larger numbers from bytes and, in C, this avoids all
the endian issues.

Off-topic digression: there are often functions to transform data from
the accepted "network order" to your machine's order, but using them
means you to have, say, a long already in memory and you'll have tied
your code to you compiler's idea of a long and how it can be packed
into a struct.

 

[Barry Schwarz]

I am reading a binary packet : 32, 8, 8, 2, 1, 1, 4, 128
I am using the following structure to parse the data:
struct header {
unsigned int a:32;
unsigned int b:8;
unsigned int c:8;
unsigned int d:2;
unsigned int e:1;
unsigned int f:1;
unsigned int g:4;
unsigned int h[4];
};
struct header head;
fread (&head, sizeof head, fp);

fread takes 4 arguments: you need fread (&head, sizeof head, 1, fp);

You asked in another thread about packing. If you don't tell your
compiler to pack this struct then it's very likely to include one byte
of padding between g and h. Use __attribute__((packed)) in gcc.

My data is getting garbled due to byte ordering differences. The bit
fieds are giving me incorrect values.

It's likely that your compiler is ordering the byte containing d,e,f,g
like this:

[ . . . . . . . . ]
_______ _ _ ___
g f e d

Doing so would violate the requirement that members of the struct be
in the same order they are declared.

So you may need to reorder the bit-fields in your struct.

If you also need to swap the endianness of your 32-bit fields, you can
use a macro like this:

#define CHANGE_END(x) (((x >>24) & 0x000000ff) | ((x >> 8) & 0x0000ff00) \
| ((x << 8) & 0x00ff0000) | ((x <<24) & 0xff000000) )

Remove del for email

 

[Peter Nilsson]

This violates the constraint of 6.7.2.1p2 if compiled on
an implementation where unsigned int is not at least
32-bits.

   unsigned int b:8;
   unsigned int c:8;
   unsigned int d:2;
   unsigned int e:1;
   unsigned int f:1;
   unsigned int g:4;
   unsigned int h[4];
};

My data is getting garbled due to byte ordering
differences. The bit fieds are giving me incorrect
values.

It's likely that your compiler is ordering the byte
containing d,e,f,g
like this:

[ . . . . . . . . ]
 _______ _ _ ___
   g     f e  d

Doing so would violate the requirement that members
of the struct be in the same order they are declared.

Not necessarily. The ordering of bitfields within an
allocation unit (or even whether a bit-field overlaps
two such units) is implementation-defined [6.7.2.1p10.]

 

[rahul]

This violates the constraint of 6.7.2.1p2 if compiled on
an implementation where unsigned int is not at least
32-bits.

unsigned int b:8;
unsigned int c:8;
unsigned int d:2;
unsigned int e:1;
unsigned int f:1;
unsigned int g:4;
unsigned int h[4];
};

My data is getting garbled due to byte ordering
differences. The bit fieds are giving me incorrect
values.
It's likely that your compiler is ordering the byte
containing d,e,f,g
like this:
[ . . . . . . . . ]
_______ _ _ ___
g f e d

Doing so would violate the requirement that members
of the struct be in the same order they are declared.

Not necessarily. The ordering of bitfields within an
allocation unit (or even whether a bit-field overlaps
two such units) is implementation-defined [6.7.2.1p10.]

I dropped the bit fields in structures and am using a bit library. The
library uses a union to determine the endian-ness
of the machine. It won't help if the data is being written by some
other program but using htonl/ntohl etc serves the
purpose. The library internally uses a char to read one byte and then
returns the desired number of bits using
bitwise shift, and, or.

I had to drop the bit fields as the bits are read into a memory
address and '&' operator is not applicable for bit fields.
I initialized all the members as unsigned char. As the standard goes,
a char is always 1 byte.

 

[Keith Thompson]

I had to drop the bit fields as the bits are read into a memory
address and '&' operator is not applicable for bit fields.
I initialized all the members as unsigned char. As the standard goes,
a char is always 1 byte.

Yes, a char is always 1 byte, but a byte is not always 8 bits.

But you're probably safe in assuming that it is on most platforms.

 

[rahul]

FYI
If anyone is looking for a bit library or is just curious about the
implementation, I got it from here
http://michael.dipperstein.com/bitlibs/
This library has 2 small bugs. You need to 0 the memory area before
reading where the bits are to be read
or else you will be getting incorrect results. Secondly, it does an
unnecessary right shift while reading
multiple bits.

It takes care of the endian-ness. I am posting it as I think some of
you might want to comment on the
implementation.

 

[Nick Keighley]

On 28 May 2008 at  5:40, rahul wrote:

I am reading a binary packet :  32, 8, 8, 2,  1, 1, 4, 128
I am using the following structure to parse the data:
struct header {
   unsigned int a:32;
   unsigned int b:8;
   unsigned int c:8;
   unsigned int d:2;
   unsigned int e:1;
   unsigned int f:1;
   unsigned int g:4;
   unsigned int h[4];
};
struct header head;

You asked in another thread about packing. If you don't tell your
compiler to pack this struct then it's very likely to include one byte
of padding between g and h. Use __attribute__((packed)) in gcc.

it is best to avoid non-portable constructs like this. Other
suggestions
of bit stream libraries etc. are a better way to go

<snip>

 

[Barry Schwarz]

This violates the constraint of 6.7.2.1p2 if compiled on
an implementation where unsigned int is not at least
32-bits.

   unsigned int b:8;
   unsigned int c:8;
   unsigned int d:2;
   unsigned int e:1;
   unsigned int f:1;
   unsigned int g:4;
   unsigned int h[4];
};

My data is getting garbled due to byte ordering
differences. The bit fieds are giving me incorrect
values.

It's likely that your compiler is ordering the byte
containing d,e,f,g
like this:

[ . . . . . . . . ]
 _______ _ _ ___
   g     f e  d

Doing so would violate the requirement that members
of the struct be in the same order they are declared.

Not necessarily. The ordering of bitfields within an
allocation unit (or even whether a bit-field overlaps
two such units) is implementation-defined [6.7.2.1p10.]

What you say is true but it seems to defeat the purpose of footnote
106: "An unnamed bit-field structure member is useful for padding to
conform to externally imposed layouts."

Yes, the footnote is for information only and your quote takes
precedence. But if the order of bit-fields within an allocation unit
is implementation defined it sure reduces the portability of any code
trying to match an externally imposed layout.


Remove del for email

 

[Richard Tobin]

Yes, the footnote is for information only and your quote takes
precedence. But if the order of bit-fields within an allocation unit
is implementation defined it sure reduces the portability of any code
trying to match an externally imposed layout.

"The order" means high-to-low or low-to-high; it doesn't permit
shuffling them around.

-- Richard

 

[Ben Bacarisse]

FYI
If anyone is looking for a bit library or is just curious about the
implementation, I got it from here
http://michael.dipperstein.com/bitlibs/
This library has 2 small bugs. You need to 0 the memory area before
reading where the bits are to be read
or else you will be getting incorrect results. Secondly, it does an
unnecessary right shift while reading
multiple bits.

It takes care of the endian-ness. I am posting it as I think some of
you might want to comment on the
implementation.

It does things in what is, to me, a slightly odd way. The only places
where endianness matters (for this library at least) is when reading
integer values larges than one byte. The author has chosen to do this
by exposing the endianness and using explicit multi-byte arrays. It
is much simpler (usually) to do arithmetic on the desired result type.
That method is fully portable whereas the method used by this library
works only on systems that have been catered for by the programmer (he
considers only two orders). I have written C on a machine where this
would break but of course such machines are rare nowadays, probably
because so much C code breaks on them.

 

[Barry Schwarz]

"The order" means high-to-low or low-to-high; it doesn't permit
shuffling them around.

Yes, I understand that. On my system there is a status word where one
8 bit byte contains four one byte flags (f1, f2, f3, f4) and a four
byte integer (i1), in that order from left to right. If my struct
contains
int f1 : 1;
int f2 : 1;
int f3 : 1;
int f4 : 1;
int i1 : 4;
I have no guarantee that i1 follows f4. The order of the bit-fields
could be i1, f4, f3, f2, f1. And if it compiles the way I want on
version 1 of compiler 1, I have no assurance it will on version 2 or
on compiler 2. If it doesn't I cannot upgrade or switch. In order to
be strictly conforming (and therefore maximally portable) I need to
replace the bit fields with an unsigned char and use the shift and/or
and operators to extract the fields of interest. All of which seems
to defeat the purpose encouraged in footnote 106.


Remove del for email