Structure size directives

T

Tor Rustad

Richard said:
Obviously on such systems, for packing to be useful, the compiler must
do the work necessary to make it work, probably by using byte acesses
or special instructions.

IMO, "packing" should be avoided anyway. On architectures with alignment
requirements, it's a terrible idea to use e.g.

#pragma pack(1)

"If you use #pragma pack to align struct or union members on boundaries
other than their natural boundaries, accessing these fields usually
leads to a bus error on SPARC."
- http://docs.sun.com/app/docs/doc/819-5265/6n7c29ck7?l=en&a=view&q=pack

Furthermore, it's a rather bad idea on most RISC architectures, it's a
fault. The system may, or may not mask the fault. Typically, unaligned
access will *at least* give a performance penalty, as it do on x86.

Windows, typically mask such alignment faults on the x86 by default,
except for [1]. However, this don't always need to be the case, for e.g.
MIPS, the MSDN doc state an explicit Win32 call was needed:

SetErrorMode(GetErrorMode() | SEM_NOALIGNMENTFAULTEXCEPT);

to make Windows automatically do the fix-up on alignment faults.

MIPS is a rather exotic case, but programmers shouldn't expect that x64
will be so forgiving to alignment faults, as x86 has been. Don't be
surprised if the penalty become significant bigger in this case.


[1] IIRC, unaligned access to 128-bit SSE/SSE2-based "types" trigger GPF.
 
T

Tor Rustad

Keith said:
(e-mail address removed)-cnrc.gc.ca (Walter Roberson) writes:
[...]
[...]
Note --

If you use #pragma pack to align struct or union members on
boundaries other than their natural boundaries, accessing these
fields usually leads to a bus error on SPARC. In order to avoid
such an error, be sure to also specify the -xmemalign option.
See B.2.108 -xmemalign=ab , for the optimal way to compile such
programs.
[...]

I'd consider that to be a bug in the SPARC compiler. If you declare a
structure to be packed, it's the compiler's job to generate the right
code to access the members of that structure, however they happen to
be aligned. The fact that there's a workaround for this bug doesn't
make it any less a bug.

I rather consider it a bug in my code, if it trigger a HW fault via some
non-standard extension I don't understand.

It's beside the point, that some systems happens to mask the fault away,
e.g. via a kernelmode exception handler, or something.
 
C

CBFalconer

CBFalconer wrote:
.... snip ...

You said:


You didn't say it would be a pain or inefficient for an
implementation to support indexing into an array of packed
structures. You said, flat out, that it couldn't be done.

You snipped my statement, as follows:
" No, it may be possible to overcome, but the effort is not
" worthwhile.

which is not "flat out, that it couldn't be done". In other
words, I cavilled in advance.
 
C

CBFalconer

Flash said:
jacob navia wrote:
.... snip ...


If the library headers are written properly then you should not
need to know about the packing because just including win.h (or
whatever) should sort it all out for you.

win.h!! There is no such header mentioned in the C standard. Are
you seriously suggesting letting Microsoft sort things out
anywhere? anytime?
 
K

Keith Thompson

Tor Rustad said:
IMO, "packing" should be avoided anyway. On architectures with
alignment requirements, it's a terrible idea to use e.g.

#pragma pack(1)

"If you use #pragma pack to align struct or union members on
boundaries other than their natural boundaries, accessing these fields
usually leads to a bus error on SPARC."
- http://docs.sun.com/app/docs/doc/819-5265/6n7c29ck7?l=en&a=view&q=pack

I've already expressed my opinion of this behavior (that it's a bug,
even if it's not a violation of the standard).
Furthermore, it's a rather bad idea on most RISC architectures, it's a
fault. The system may, or may not mask the fault. Typically, unaligned
access will *at least* give a performance penalty, as it do on x86.

Windows, typically mask such alignment faults on the x86 by default,
except for [1]. However, this don't always need to be the case, for
e.g. MIPS, the MSDN doc state an explicit Win32 call was needed:

SetErrorMode(GetErrorMode() | SEM_NOALIGNMENTFAULTEXCEPT);

to make Windows automatically do the fix-up on alignment faults.

MIPS is a rather exotic case, but programmers shouldn't expect that
x64 will be so forgiving to alignment faults, as x86 has been. Don't
be surprised if the penalty become significant bigger in this case.

In my opinion, if a compiler is going to support structure members
that don't have the alignment for the member's type, it's the
compiler's job to generate code that will access that member in a way
that (apart from performance and code size) is indistinguishable from
accessing an aligned member.

On an x86, as I understand it, the compiler can just generate code to
access the member normally, and it's handled in hardware.

On a SPARC, the compiler can, for example, do the equivalent of
memcpy() to copy the unaligned member to or from an aligned temporary.
Or maybe it can access it directly and clean up after the fault. The
language, or the extension, doesn't need to specify *how* this is
done, just that it needs to be done properly.

Blindly trying to access the member as if it were aligned, when the
compiler itself chose to misalign it (in response to a pack pragma) is
just dumb.

I've worked in languages that have the equivalent of packed
structures, and this is what they do.
 
J

James Kuyper

CBFalconer said:
The last time I looked there were considerable differences between
the C standard and the Ada standard. I see no reference to packing
in the C standard.

This also illustrates the foolishness of discussing non-standard
compiler features in this newsgroup.

The issue was not whether the C standard currently supports it; the
issue was whether the C standard could be modified to support it. Note
the key phrase "What would be nice is that standard C would allow this
....". That's the subjunctive mood (incorrectly handled, but mishandling
the subjunctive mood is something that even native speakers of English
often do).

CJ responded directly and negatively to that statement with an assertion
that this feature is inherently non-portable. If by "non-portable" he
meant that the C standard didn't currently support this feature, then
his comment was a non-sequitur. I assume that this was not CJ's intent.
Assuming that CJ understood what Jacob was saying, and was actually
attempting to be rationally responsive to Jacob's comment, the only
reasonable interpretation was that CJ was asserting that this feature
would be so inherently non-portable as to prevent such a modification of
the standard.

In that case, pointing out that the Ada standard does support such a
feature is entirely relevant: it proves that it is possible for a
computer language standard to support such a feature; how much
modification of the C standard would be needed to achieve that same goal
is yet to be determined.
 
S

santosh

James said:
The issue was not whether the C standard currently supports it; the
issue was whether the C standard could be modified to support it. Note
the key phrase "What would be nice is that standard C would allow this
...". That's the subjunctive mood (incorrectly handled, but
mishandling the subjunctive mood is something that even native
speakers of English often do).

I suppose that would be: "What would be nice is for standard C to
allow..."
CJ responded directly and negatively to that statement with an
assertion that this feature is inherently non-portable. If by
"non-portable" he meant that the C standard didn't currently support
this feature, then his comment was a non-sequitur. I assume that this
was not CJ's intent. Assuming that CJ understood what Jacob was
saying, and was actually attempting to be rationally responsive to
Jacob's comment, the only reasonable interpretation was that CJ was
asserting that this feature would be so inherently non-portable as to
prevent such a modification of the standard.

In that case, pointing out that the Ada standard does support such a
feature is entirely relevant: it proves that it is possible for a
computer language standard to support such a feature; how much
modification of the C standard would be needed to achieve that same
goal is yet to be determined.

I suspect that it would be harder for the C standard to incorporate this
feature than for Ada, since the C standard attempts to be portable to a
wider range of machines than Ada. Also this might go against C's spirit
of being as efficient as possible and assuming that the programmer
knows what he is doing.
 
K

Kenneth Brody

CJ said:
Yes, and relying on this sort of thing is a bad idea if you're trying to
write portable code.

Sometimes you have to write non-portable code. For example, when
passing structs to system APIs. The system and/or the runtime
library requires the struct to have specific alignments. If you
allow (even via command-line flags as you suggest) a method of
changing the alignment, then you must supply a method of local
overrides.

[...]
Probably gcc/etc. want to allow the programmer to decide *for each
individual struct* whether it should be packed or not. This approach
will create terrible maintenance headaches - the pack/don't pack
directive will need to be present each time the compiler sees a
declaration of the struct.

Unless you forbid any changes in alignment, then you are correct.
However, you then run into problems when calling system API calls
which require a specific alignment on their structs.

--
+-------------------------+--------------------+-----------------------+
| Kenneth J. Brody | www.hvcomputer.com | #include |
| kenbrody/at\spamcop.net | www.fptech.com | <std_disclaimer.h> |
+-------------------------+--------------------+-----------------------+
Don't e-mail me at: <mailto:[email protected]>
 
R

REH

James Kuyper wrote:
I suspect that it would be harder for the C standard to incorporate this
feature than for Ada, since the C standard attempts to be portable to a
wider range of machines than Ada. Also this might go against C's spirit
of being as efficient as possible and assuming that the programmer
knows what he is doing.

Why do you assume that C attempts to be "more portable" than Ada.
What are you basing your assertion on? What you do portably in C than
cannot be done in Ada?

REH
 
F

Flash Gordon

CBFalconer wrote, On 05/12/07 01:33:
win.h!! There is no such header mentioned in the C standard. Are
you seriously suggesting letting Microsoft sort things out
anywhere? anytime?

Yes, I expect them to sort out their headers so that you can include
them and use their API without having to litter your own code with pack
directives even if every structure used in the API requires different
packing. I strongly suspect that MS actually do this (I've seen code use
the Windows API without messing about with packing), and the one time I
found it being done incorrectly it was in a header provided by a third
party for Borland C++ builder, so MS were responsible for neither the
header nor the compiler.
 
R

REH

Why do you assume that C attempts to be "more portable" than Ada.
What are you basing your assertion on? What you do portably in C than
cannot be done in Ada?

REH

Oops, sorry. My poor editing made the above appear as if I am quote
James Kuyper. I apologize James.

REH
 
T

Tor Rustad

Keith said:
I've already expressed my opinion of this behavior (that it's a bug,
even if it's not a violation of the standard).

This isn't a SPARC only thing, in addition to Windows/MIPS, the
Microsoft compiler doc state this:

"Itanium: An application must explicitly call SetErrorMode with
SEM_NOALIGNMENTFAULTEXCEPT to have the system automatically fix
alignment faults. The default setting is for the system to make
alignment faults visible to an application." [1]


people working with HPC, might also be interested in knowing:

"In addition, if the Alignment Mask bit is set in Control Register 0
(CR0), an unaligned memory reference may cause an alignment check
exception. For more information on this topic, see the AMD64
Architecture Programmer’s Manual, Volume 2, order# 24593." [2]

Furthermore, it's a rather bad idea on most RISC architectures, it's a
fault. The system may, or may not mask the fault. Typically, unaligned
access will *at least* give a performance penalty, as it do on x86.

Windows, typically mask such alignment faults on the x86 by default,
except for [1]. However, this don't always need to be the case, for
e.g. MIPS, the MSDN doc state an explicit Win32 call was needed:

SetErrorMode(GetErrorMode() | SEM_NOALIGNMENTFAULTEXCEPT);

to make Windows automatically do the fix-up on alignment faults.

MIPS is a rather exotic case, but programmers shouldn't expect that
x64 will be so forgiving to alignment faults, as x86 has been. Don't
be surprised if the penalty become significant bigger in this case.

In my opinion, if a compiler is going to support structure members
that don't have the alignment for the member's type, it's the
compiler's job to generate code that will access that member in a way
that (apart from performance and code size) is indistinguishable from
accessing an aligned member.

Well, I don't think most system programmers agrees with you, since this
leads to code bloat, hiding of translation details and do hurt performance.

C is close to a WYSIWYG language, which helps in generating compact,
predictable code, the language add little obfuscation for system
developers. The spirit of C, is to trust the programmer and not stop
him/her from doing exactly what asked.

If I instruct the compiler to do explicit alignment, that is what it
should provide. I can understand if application programmers don't care
much if the compiler generate some fix-up code to handle alignment
issues, but others don't want it.

On an x86, as I understand it, the compiler can just generate code to
access the member normally, and it's handled in hardware.

Well, I am not that interested in checking out old x86 manuals on this.
Have my hand full with the x64 manuals. :) However, after a quick peek in

/usr/src/linux-headers-x.x.xx-xx/arch/i386/kernel/traps.c

I do find

void __init trap_init(void)
{
...
set_trap_gate(17,&alignment_check);
...
}

which show that the Linux kernel, at least handle x86 bus issues via
trap gate 17. Perhaps this kernel function address page alignment faults
only, or other alignment faults as well. I don't know.

However, most x86 HW exceptions due to alignment, appears to be masked
away on Windows and Linux kernel, what other kernels do here, is
difficult to tell without checking it. However, I can't remember seeing
such alignment exception code in the FreeDOS kernel...
On a SPARC, the compiler can, for example, do the equivalent of
memcpy() to copy the unaligned member to or from an aligned temporary.
Agreed

Or maybe it can access it directly and clean up after the fault. The
language, or the extension, doesn't need to specify *how* this is
done, just that it needs to be done properly.

No. A C compiler can't generally be granted access to ring 0 of the
kernel, the kernel is the boss of which signals are masked and which
signals are forwarded. The compiler run in user space, and should not
(due to security reasons) have access to higher privilege levels. The
operating system might (like Windows do via the SetErrorMode API)
provide a system call to mask alignment faults, or it might not.

Blindly trying to access the member as if it were aligned, when the
compiler itself chose to misalign it (in response to a pack pragma) is
just dumb.

The cases I have been interested in some particular alignment of struct
members, has either been in network code or in cases where I wanted to
store the struct on disk.

The obvious C solution, is then to do the conversion yourself, e.g.

struct T { ... } object;

unsigned char buffer[sizeof object];

and convert each struct member explicit into 'buffer'. This is a fully
portable method among C implementations and different architectures.
I've worked in languages that have the equivalent of packed
structures, and this is what they do.

And in such languages, packing comes with a price.. e.g. what could have
been done via a single load, now require two loads. In a HPC cluster,
you don't want such "packing" on a memory bound programs.


[1] http://msdn2.microsoft.com/en-us/library/ms680621.aspx
[2] "Software Optimization Guide for AMD Family 10h Processors"
 
M

Mark McIntyre

jacob said:
CJ wrote:
>
Its surely not a "detail".

CJ said it was an IMPLEMENTATION detail.
If I have 100 million of those, the difference is a whooping
25 million!

If you have 100 million of those, you have a badly written algorithm
surely?

And I suspect if you get to the stage where you have enormous arrays of
large structures, you're probably running on a machine with sensible
amounts of memory eg 512GB anyway...
The fact that MANY (if not all) compilers give the programmer
a way to specify this is a testimnoy of the universal need
behind this.

My own experience is that packing is most needed at interfaces eg before
transmitting via a network or storing to disk. I've never to my
knowledge seen it used to save space.
P.S. It would be nice to avoid polemic... "Rubbish" means
here just that you do not agree with me I suppose

You're hardly innocent of this yourself....
 
K

Keith Thompson

Tor Rustad said:
Keith said:
I've already expressed my opinion of this behavior (that it's a bug,
even if it's not a violation of the standard).

This isn't a SPARC only thing, in addition to Windows/MIPS, the
Microsoft compiler doc state this:

"Itanium: An application must explicitly call SetErrorMode with
SEM_NOALIGNMENTFAULTEXCEPT to have the system automatically fix
alignment faults. The default setting is for the system to make
alignment faults visible to an application." [1]

Sure, lots of CPUs don't allow non-aligned accesses, or at least don't
allow them without an extra effort.

[...]
Well, I don't think most system programmers agrees with you, since
this leads to code bloat, hiding of translation details and do hurt
performance.

System programmers who are concerned about code bloat et al can just
not use packed structures.
C is close to a WYSIWYG language, which helps in generating compact,
predictable code, the language add little obfuscation for system
developers. The spirit of C, is to trust the programmer and not stop
him/her from doing exactly what asked.

If I instruct the compiler to do explicit alignment, that is what it
should provide. I can understand if application programmers don't care
much if the compiler generate some fix-up code to handle alignment
issues, but others don't want it.

Here's my point. Suppose a compiler allows a declaration like this:

struct foo {
char c;
int i;
};
#pragma pack(struct foo)
struct foo obj;
obj.i = 42;

Assume sizeof(int)==4, and int normally requires 4-byte alignment.
Assume that without the #pragma pack, "i" would be allocated at an
offset of 4 bytes, and with the #pragma pack, it would be allocated at
an offset of 1 byte (misaligned).

*If* the compiler supports this, then it had better let me access
obj.i by name without causing my program to crash. It can do whatever
it needs to do to make this happen; I don't much care about the
machine-code details, any more than I care about how it would access
obj.i if it weren't misaligned.

As a programmer, if I use "#pragma pack", I'm *explicitly* asking for
less memory usage at the possible expense of larger and slower code.
That's what it's for. If I don't want to accept the cost, I won't use
the pragma.

If I try to access obj.i and the compiler ignores the misalignment,
generating code that causes my program to blow up, then the compiler
has lied to me. It claims to implement packed structures, but it
really doesn't. And when the compiler sees a reference to obj.i, it
*knows* that it's misaligned.

Ok, maybe the documentation tells me to expect this, but a documented
bug is still a bug. And as I've said before, this wouldn't be a
violation of the standard; it's still a bug.

The same applies if, say, "#pragma STDC PACK" were made a part of a
future C standard. If the implementation or the language lets me
declare a member, it had better let me access it. If it won't let me
access it, it shouldn't let me declare it.

[...]
And in such languages, packing comes with a price.. e.g. what could
have been done via a single load, now require two loads. In a HPC
cluster, you don't want such "packing" on a memory bound programs.

Of course it comes with a price.

I don't know that you wouldn't want such packing in an HPC cluster, or
on any other system; the savings in memory *might* outweigh the costs.
It's going to depend on the application, and it should be up to the
programmer.

Now if you want to argue that "#pragma pack" is a bad idea, I won't
necessarily disagree with you. I don't think I've ever used it
myself, and I don't necessarily advocate adding it to the standard.
I don't object either to providing the feature, or to leaving it out.
I object to implementing it badly.
 
J

James Kuyper

REH said:
Oops, sorry. My poor editing made the above appear as if I am quote
James Kuyper. I apologize James.

No problem. I know how to interpret the indentation levels, so I was
only confused for a few moments; I assume most people will figure it out.
 
T

those who know me have no need of my name

in comp.lang.c i read:
Unaligned accesses to objects, should be banned. It has nothing to do
with portable C code, it might trigger a core dump on non-x86 CPU.

even on x86 systems with the right option enabled.

but i think jacob's point is that people want these "dense" struct's and
that there should be a way to get the compiler to emit the code necessary
to marshall the data. some systems would turn off or on some option while
the program is running, other systems would need the type of handling often
seen in c code for marshalling "plain" structs into and out of arrays of
unsigned char, etc...
 
C

CBFalconer

those said:
even on x86 systems with the right option enabled.

but i think jacob's point is that people want these "dense"
struct's and that there should be a way to get the compiler to
emit the code necessary to marshall the data. some systems
would turn off or on some option while the program is running,
other systems would need the type of handling often seen in c
code for marshalling "plain" structs into and out of arrays of
unsigned char, etc...

This is opinion, but I think he, and they, are mistaken. Code has
quite different objectives today than it used to. There was a time
when memory was extremely precious, and we took almost all options
to economize it. Today, memory (and disk storage) is cheap, and we
use it freely. However nothing has reduced the need for clear and
understandable code.

BTW, please do not strip attributions for material you quote.
 
T

Tor Rustad

Keith said:
[...]
Well, I don't think most system programmers agrees with you, since
this leads to code bloat, hiding of translation details and do hurt
performance.

System programmers who are concerned about code bloat et al can just
not use packed structures.

When you target a specific architecture, the alignment requirements are
know apriori, so system developers do make use of explicit alignment and
knows what they are doing.

Here's my point. Suppose a compiler allows a declaration like this:

struct foo {
char c;
int i;
};
#pragma pack(struct foo)
struct foo obj;
obj.i = 42;

Assume sizeof(int)==4, and int normally requires 4-byte alignment.
Assume that without the #pragma pack, "i" would be allocated at an
offset of 4 bytes, and with the #pragma pack, it would be allocated at
an offset of 1 byte (misaligned).

*If* the compiler supports this, then it had better let me access
obj.i by name without causing my program to crash. It can do whatever
it needs to do to make this happen; I don't much care about the
machine-code details, any more than I care about how it would access
obj.i if it weren't misaligned.

There are really two different cases here, you talk about the usage of
"pack" to minimize padding in a struct, while I talk about the "pack
(n)" usage, which specify alignment(s) of struct members. :)

I must admit that I have never seen the former "pack" usage, and have
never wanted this type of "packing" in my own code. If the memory
available is minimal and you like to minimize the amount of storage used
by structure objects, we can manually re-arrange struct members like this:

struct foo_alt
{
int i; /* typically word aligned */
char c; /* typically byte aligned */
};

and then compile program with size optimization switch on. The
programmer typically need to think about both *data* and *code* size.
The solution above, should minimize the total size, while "pragma pack"
may bloat the *code* size.

Hence, the struct members can be manually sorted by alignment
requirements, like e.g.

struct foo_general
{
/* put type double members here... */
/* put type long members here... */

int i;
/* put more type int members here... */

char c;
/* put more type char members here... */
};

The struct can even be generated by another program. A recent project of
mine, I used this technique to generate a monster struct (among other
stuff):

$ cat iso_bitmap.def

/*
* This file, "iso_bitmap.def", is used by iso_gen_struct.c
* to generate <iso_bitmap_def.h>
*
* Reference: ISO 8583:1993 Table 7A.
*/

/* Name No Type Length Format*/
EXPAND_FIELD(MESSAGE_TYPE_ID, 0, N, 4, FIXED);
EXPAND_FIELD(BITMAPS , 1, B, 16, FIXED);
.....
<lots of other struct members snipped>

The C program using the <iso_bitmap_def.h> header file, make no
assumptions of ordering of members within the struct, or the offsets of
it's struct members. So, if I wanted, I could generate the struct with
members sorted by the alignment requirements.

Hmm.. instead of "pragma pack", it appears to me that a "pragma
align_sort" is more useful, telling compiler there is no need for
ordering the struct members in the order given by the declaration of the
struct.
Of course it comes with a price.

I don't know that you wouldn't want such packing in an HPC cluster, or
on any other system; the savings in memory *might* outweigh the costs.
It's going to depend on the application, and it should be up to the
programmer.

I was above referring to HPC programs, which was run-time memory bound,
(not talking about memory size), and not suggesting that memory size
constraints wasn't important for HPC problems.
Now if you want to argue that "#pragma pack" is a bad idea, I won't
necessarily disagree with you. I don't think I've ever used it
myself, and I don't necessarily advocate adding it to the standard.
I don't object either to providing the feature, or to leaving it out.
I object to implementing it badly.

I object to adding a feature, which do some code-bloat behind the scene
for fix-ups. However, having a directive which inform the compiler that
the struct member ordering can be rearranged (with the most strictly
aligned members first), I'm less negative to, since this shouldn't give
any code bloat for fix-ups, or mis-aligned members.
 
T

Tor Rustad

those said:
in comp.lang.c i read:


even on x86 systems with the right option enabled.

but i think jacob's point is that people want these "dense" struct's and
that there should be a way to get the compiler to emit the code necessary
to marshall the data. some systems would turn off or on some option while
the program is running, other systems would need the type of handling often
seen in c code for marshalling "plain" structs into and out of arrays of
unsigned char, etc...

Well, I usually assume that compiler writers/maintainers know far more C
than myself, but there has been some counter-examples by one person.

OP's lack of attention to detail is striking, instead of

"most compilers provide some way to do this"

it should have been

"common compiler extension ..."


And when OP say:

"Note that under windows you must know this kind of stuff since windows
uses packed structures extensively."

I'm rather sure, he was ignorant to the fact that Windows may crash, when

#pragma pack (1)

is used, which is nothing less than shocking, when coming from someone
specializing on C compiler for this platform.
 
S

santosh

Tor Rustad wrote:

And when OP say:

"Note that under windows you must know this kind of stuff since
windows uses packed structures extensively."

I'm rather sure, he was ignorant to the fact that Windows may crash,
when

#pragma pack (1)

is used, [ ... ]

Windows or the offending program?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
474,432
Messages
2,571,681
Members
48,796
Latest member
Greg L.

Latest Threads

Top