C
chrisbazley
Greetings,
Over the past few days I have been porting a C program to a platform
that doesn't support unaligned data access, using a compiler that
doesn't support packed structures.
The program currently loads large chunks of data from an uncompressed
binary file format and addresses them using packed structures. To make
it work on big-endian systems, someone has added extra functions to
iterate over arrays of these packed structs and reverse the endianess
of 16 bit and 32 bit members. This takes place after reading data from
file and before writing it back.
I don't particularly like packed structures because they are not
standard C and even when supported they are slower to access and
require more code than properly-aligned data. The solution adopted for
my platform by the last person to port this program (an older version)
was to realign all the data after having loaded it. Yet another ugly
and time-consuming post-fread() step!
I would prefer to abandon the practice of loading data from the file
in large chunks. If the program instead loads one value at a time then
it can simultaneously correctly align it and swap endianness (if
required). I'm assuming that the file is buffered, so the additional
cost of many calls to fread() and/or fgetc() shouldn't be unbearable.
Of course that must be weighed against the benefit of not needing to
post-process the data.
Does that sound like the right strategy?
Having to read data one byte at a time (or even one 16- or 32-bit
value at a time) obviates one of the advantages of using a binary file
format in the first place, although there is still benefit in having a
smaller file size than a text format such as XML.
The second question I have been considering is the best method of
implementing the load-value-and-swap-endianness functions. So far, I
have tried two methods.
The first method is to load a whole value using fread() and then swap
the endianness if necessary:
int read_uint16(uint16_t *s, FILE *f)
{
if (fread(s, sizeof(*s), 1, f) != 1)
return 1; /* non-zero means error */
#ifdef BIG_ENDIAN
*s = ((*s >> 8) & 0xFF) | ((*s << 8) & 0xFF00);
#endif
return 0; /* zero means success */
}
int read_uint32(uint32_t *s, FILE *f)
{
if (fread(s, sizeof(*s), 1, f) != 1)
return 1; /* non-zero means error */
#ifdef BIG_ENDIAN
*s = ((*s >> 24) & 0xFF) | ((*s >> 8) & 0xFF00) | ((*s << 8) &
0xFF0000) | ((*s << 24) & 0xFF000000);
#endif
return 0; /* zero means success */
}
The second method is to load one byte at a time:
static int read_int(unsigned long *s, size_t size, FILE *f)
{
unsigned long value = 0;
size_t byte;
for (byte = 0; byte < size; byte++)
{
int c = fgetc(f); /* assumes char is 8 bit */
if (c == EOF)
break;
value |= ((unsigned long)c & 0xFF) << (8 * byte);
}
*s = value;
return byte < size; /* non-zero means error */
}
int read_uint16(uint16_t *s, FILE *f)
{
unsigned long value;
int error = read_int(&value, sizeof(*s), f);
*s = (uint16_t)value; /* narrowing cast */
return error;
}
int read_uint32(uint32_t *s, FILE *f)
{
unsigned long value;
int error = read_int(&value, sizeof(*s), f);
*s = (uint32_t)value; /* narrowing cast */
return error;
}
At the moment I prefer the second method because it has the advantage
that it doesn't need to be configured for a big-endian or little-
endian system; the code works naturally on either. It seems more of a
'pure' solution for that reason, and I prefer to avoid conditional
compilation where possible.
However, the second method will be slower on both little and big-
endian systems, because of the loop and multiple calls to fgetc().
Also, it relies on an assumption that fgetc will always return an
octet (or EOF). Does anyone know (or can anyone guess) whether that
would be true on a system where the type 'char' is wider than 8 bits?
Unfortunately, I can't think of a portable endian-independent solution
that avoids both conditional compilation and the assumption that
'char' is an 8 bit type (or fgetc() returns an octet). Even methods
for determining endianness at run time would surely rely upon treating
'int' types as arrays of 'char'.
I would be very interested in your opinions.
Cheers,
Over the past few days I have been porting a C program to a platform
that doesn't support unaligned data access, using a compiler that
doesn't support packed structures.
The program currently loads large chunks of data from an uncompressed
binary file format and addresses them using packed structures. To make
it work on big-endian systems, someone has added extra functions to
iterate over arrays of these packed structs and reverse the endianess
of 16 bit and 32 bit members. This takes place after reading data from
file and before writing it back.
I don't particularly like packed structures because they are not
standard C and even when supported they are slower to access and
require more code than properly-aligned data. The solution adopted for
my platform by the last person to port this program (an older version)
was to realign all the data after having loaded it. Yet another ugly
and time-consuming post-fread() step!
I would prefer to abandon the practice of loading data from the file
in large chunks. If the program instead loads one value at a time then
it can simultaneously correctly align it and swap endianness (if
required). I'm assuming that the file is buffered, so the additional
cost of many calls to fread() and/or fgetc() shouldn't be unbearable.
Of course that must be weighed against the benefit of not needing to
post-process the data.
Does that sound like the right strategy?
Having to read data one byte at a time (or even one 16- or 32-bit
value at a time) obviates one of the advantages of using a binary file
format in the first place, although there is still benefit in having a
smaller file size than a text format such as XML.
The second question I have been considering is the best method of
implementing the load-value-and-swap-endianness functions. So far, I
have tried two methods.
The first method is to load a whole value using fread() and then swap
the endianness if necessary:
int read_uint16(uint16_t *s, FILE *f)
{
if (fread(s, sizeof(*s), 1, f) != 1)
return 1; /* non-zero means error */
#ifdef BIG_ENDIAN
*s = ((*s >> 8) & 0xFF) | ((*s << 8) & 0xFF00);
#endif
return 0; /* zero means success */
}
int read_uint32(uint32_t *s, FILE *f)
{
if (fread(s, sizeof(*s), 1, f) != 1)
return 1; /* non-zero means error */
#ifdef BIG_ENDIAN
*s = ((*s >> 24) & 0xFF) | ((*s >> 8) & 0xFF00) | ((*s << 8) &
0xFF0000) | ((*s << 24) & 0xFF000000);
#endif
return 0; /* zero means success */
}
The second method is to load one byte at a time:
static int read_int(unsigned long *s, size_t size, FILE *f)
{
unsigned long value = 0;
size_t byte;
for (byte = 0; byte < size; byte++)
{
int c = fgetc(f); /* assumes char is 8 bit */
if (c == EOF)
break;
value |= ((unsigned long)c & 0xFF) << (8 * byte);
}
*s = value;
return byte < size; /* non-zero means error */
}
int read_uint16(uint16_t *s, FILE *f)
{
unsigned long value;
int error = read_int(&value, sizeof(*s), f);
*s = (uint16_t)value; /* narrowing cast */
return error;
}
int read_uint32(uint32_t *s, FILE *f)
{
unsigned long value;
int error = read_int(&value, sizeof(*s), f);
*s = (uint32_t)value; /* narrowing cast */
return error;
}
At the moment I prefer the second method because it has the advantage
that it doesn't need to be configured for a big-endian or little-
endian system; the code works naturally on either. It seems more of a
'pure' solution for that reason, and I prefer to avoid conditional
compilation where possible.
However, the second method will be slower on both little and big-
endian systems, because of the loop and multiple calls to fgetc().
Also, it relies on an assumption that fgetc will always return an
octet (or EOF). Does anyone know (or can anyone guess) whether that
would be true on a system where the type 'char' is wider than 8 bits?
Unfortunately, I can't think of a portable endian-independent solution
that avoids both conditional compilation and the assumption that
'char' is an 8 bit type (or fgetc() returns an octet). Even methods
for determining endianness at run time would surely rely upon treating
'int' types as arrays of 'char'.
I would be very interested in your opinions.
Cheers,
