Library bug or my fault?

[dj3vande]

But meanwhile in the real world where efficiency and cost effective SW
development is important ...

....portability matters.

At my day job, we have both code written by standards-and-portability
weenies and code written by whatever-works types.

In terms of efficiency of the code, the standards-and-portability
weenies have a very slight advantage, since they fix their bugs
*before* the optimizer makes subtle bugs less subtle. But that's
barely noticeable at the best of times, and only a small amount of code
actually *needs* to run fast.

In terms of cost-effective development, any difference in productivity
between the standards-and-portability types and the whatever-works
types is completely lost in the effects of understanding of the problem
domain. (We have a lot of variability in the latter (in both the
problem domains and in who understands which ones how well), and it's
pretty much completely independent of the attitudes toward
portability.)


So for single-platform development, there's no detectable difference.
But now we're thinking of switching platforms.

The standards-and-portability weenies have already tested half the code
they've written on the platform we're thinking of switching to, since
that's what standards-and-portability weenies do. (Besides, some types
of problem are a lot easier to debug on that platform than on the one
we currently deploy on, so being able to run the code there already has
a nonnegative ROI.)
The whatever-works types have a whole bunch of code that has never been
compiled or run outside our current toolchain and deployment platform.

Which chunks of code do you think we're more likely to be able to get
running efficiently and cost-effectively on the new platform?


dave
(I don't expect Richard to provide any evidence of having gotten my
point, but it's worth pointing out for other readers.)

 

[jacob navia]

Consider a hypothetical implementation with the following
characteristics:

CHAR_BIT == 8
sizeof(int) == 4 (32 bits)
INT_MIN == -8388608 (-2**23)
INT_MAX == +8388607 (+2**23-1)

Type int has 1 sign bit, 23 value bits, and 8 padding bits.

Storing the result of evaluating any arithmetic expression in an int
object causes the object's padding bits to be set to zero. (This can
be verified by interpreting the object's representation as an array of
unsigned char.) Evaluating the value of an object with non-zero
padding bits causes immediate termination of the program.

Such a system is possible... if you do not care about
practical considerations...

I know of no actual system that has the above characteristics; that's
not the point. I do not claim that such a system is likely to be
built in the future; that's not the point either (though there could
be legitimate reasons, reasons I haven't thought of, to build such a
system). The point is that such a system conforms to the C standard,
and there *could* be some future system that does exhibit the
characteristics I've described.

Yes. Something like that *could* exist.

Do you agree?
Yes.

If so, doesn't that contradict your statement about
padding bits being "masked by the hardware"?

You are not talking about padding bits. You are talking about
control bits that are similar to the parity bits in serial
transmissions. Padding bits exist for alignment reasons.
Parity/control bits/redundancy bits are not there just to take
space, they have redundancy objectives.

If you don't agree, why
not?

See above

I understand that you personally dislike the "regulars", of whom I am
one; as you wrote in another thread, "I am completely opposed to that
people". In replying to this, please try to get past your personal
animosity and respond to what I actually wrote.

I did, but it was quite an effort. Anyway discussions are quite
useless. What bothers me here is that in discussions of people
that ask simple questions, regulars *invent* machines that
*could* exist just to confuse people.

They like (as all pedants) displaying their great "C knowledge",
by inventing possible situations where this or that paragraph of
the standard could apply.

As a matter of fact (of course) they are AGAINST the C99 standard
that they try to disqualify at each post. But they pose as
if they were for Standard C.

That is the AS IF rule!

 

[Keith Thompson]

Such a system is possible... if you do not care about
practical considerations...

As a practical consideration, it's possible that some future system,
or perhaps some relatively exotic system that already exists that
neither of us happens to be familiar with, could have exactly the
characteristics I mentioned above, or something very similar.

I think I may actually have worked on a system that has padding bits
in some integer types. I didn't happen to do anything that was
affected by this, and I didn't check the values from <limits.h> while
I still had access to the system, so I can't be sure. The system was
a Cray T90; I'm fairly sure that it or some of its predecessors used
something like 24 out of 64 bits for some integer types. If I can get
access to a Cray again, I'll check it out.

The T90 is admittedly a fairly old system, and there are probably only
a few of them still in service. But they're not *that* old; the first
were shipped in 1995. And the reason for the odd behavior was a very
strong emphasis on floating-point over integer arithmetic, something
that could very well be a decisive factor for some future systems.

In any case, I actually find it *easier* in most cases to write code
that depends only on what's guaranteed by the standard.

Yes. Something like that *could* exist.


Excellent.


You are not talking about padding bits. You are talking about
control bits that are similar to the parity bits in serial
transmissions. Padding bits exist for alignment reasons.
Parity/control bits/redundancy bits are not there just to take
space, they have redundancy objectives.

Yes, I certainly am talking about padding bits. I asked you to read
C99 6.2.6.2. Please read it again. It explains quite clearly what
padding bits are.

To be clear, parity bits that aren't visible to software are not
padding bits. Padding bits, as defined by the standard, exist only
for integer types; they're bits that are part of the type's
representation (and can therefore be seen if you view the type as an
array of unsigned char) but do not contribute to its value.

Padding *bytes* are used for alignment in structures; they're
mentioned in C99 6.2.6.1.

See above


I did, but it was quite an effort.

It should become easier with practice. Seriously.

Anyway discussions are quite
useless. What bothers me here is that in discussions of people
that ask simple questions, regulars *invent* machines that
*could* exist just to confuse people.

No, we invent machines that could exist to illustrate the wide variety
of machines that are possible, and to encourage programmers to think
about portability when they write code.

They like (as all pedants) displaying their great "C knowledge",
by inventing possible situations where this or that paragraph of
the standard could apply.

As a matter of fact (of course) they are AGAINST the C99 standard
that they try to disqualify at each post. But they pose as
if they were for Standard C.

I am not against the C99 standard. Where have I said that I am?

That is the AS IF rule!

No, it isn't.

 

[Peter Nilsson]

The problem is, the line 38 which copy 3 bytes, starting from
p2, to p1, but the immediately followed memcmp (line 42)
shows the memcpy was not well done.

It's the lines before that signal a problem.

------------------------------------- the minimum sample
--------------------------------------------------
1 #include <stdio.h>
2 #include <string>
3 #include <stdint.h>
4 #include <assert.h>
5
6 struct Foo {
7 � � uint8_t x;
8 � � uint8_t y;
9 � � uint8_t z;
10 � � uint8_t m[3];
11 };
12
13 struct Bar
14 {
15 � � uint8_t m[3];
16 };

These are completely different structs.

yes, so?

So conversion of an address of one (not the first) struct member
into a pointer to another struct is already suspicious.

yes, but there can't be padding at the beginning of a struct

Irrelevant. The m being copied is in the middle of a Foo. Given
that it isn't a Bar, why should a pointer to that be portably
convertable to a pointer to Bar?

hence Bar.m and Bar must be at the same address.

As my suggestion went on to show, there is clearly no problem
copying an array to an array of the same type and size directly.
But for no obvious reason, the code avoids that simple course.

Also uint8_t is probably (almost certainly) an alias (typedef)
for unsigned char and you can safely cast *anything* to
unsigned char

Again, irrelevant. There is no conversion to unsigned char
anywhere in the code quoted above.

I'd say this pretty well had to work (do you have a compiler where
it doesn't work?)

The OP has a compiler where the code doesn't work. I don't know
why the compiler's doing what it's doing, I merely know that, on the
basis of what's been posted, neither the compiler nor the library
can be said to be at fault.

why would foo->m not be correctly aligned?

Because any struct can have alignment requirements that are stricter
than the elements it contains.

Can you state any reasons why an implementation couldn't pad
a 3 byte structure to 4 bytes size and alignment?

 

[Barry Schwarz]

Hi,

Please check the sample code listed in the end of the message. It was
compiled using ARM/Linux cross-compiler and run on an ARM9 target. I
think the problem applies to this group because two cross-compiler
from different vendor result same error. So I guess it is not vendor
specific. If my guess is right, then it means the code itself may get
problem, but I can not figure out where it is.

The problem is, the line 38 which copy 3 bytes, starting from p2, to
p1, but the immediately followed memcmp (line 42) shows the memcpy was
not well done. I am sure the memcpy did not do the job, since if I
provide my own memcpy implemention as below, the error will go
disappear.

void memcpy(void *dest, void *src, size_t n)
{
uint8_t *d = (uint8_t*)dest;

Why do you have a cast here?

uint8_t *s = (uint8_t*)src;

for (size_t i = 0; i < n; ++i)

What is the guarantee that n is never larger that sizeof(uint8_t)?

*d++ = *s++;
}

Could you please check the code as well as the running result and
tell me what wrong with it? Thanks.


------------------------------------- the minimum sample

You need to post your real code using cut and paste. Retyping just
introduces other errors confusing everyone.

3 #include <stdint.h>
4 #include <assert.h>
5
6 struct Foo {
7 uint8_t x;
8 uint8_t y;
9 uint8_t z;
10 uint8_t m[3];
11 };
12
13 struct Bar
14 {
15 uint8_t m[3];
16 };
17
18 void pr(const char *title, const void *block, size_t n)
19 {
20 printf("%s\n", title);
21
22 uint8_t *p = (uint8_t*)block;

This cast removes the const attribute. You should add it back by
fixing the definition of p. Otherwise, there is no point in having in
the function definition at all.

23 for (size_t i = 0; i < n; ++i)
24 printf("0x%02x ", *p++);
25
26 printf("\n");
27 }
28
29 void cp(const Foo *foo)
30 {
31 Bar bar;
32
33 Bar *p1 = &bar;
34 Bar *p2 = (Bar*)(foo->m);
35 pr("before: p1:", p1, 3);
36 pr("before: p2:", p2, 3);
37
38 memcpy(p1, p2, 3);
39 pr("after: p1:", p1, 3);
40 pr("after: p2:", p2, 3);
41
42 if (memcmp(p1, p2, 3) != 0)
43 printf("!!! cp is wrong\n");

After making the obvious tweaks for C89, the code ran exactly as
expected on my system and never printed this message. By any chance
does your actual code have a ; after the if?

44 }
45
46 int main()
47 {
48 Foo foo;
49 foo.x = 1;
50 foo.y = 2;
51 foo.z = 3;
52 foo.m[0] = 0x40;
53 foo.m[1] = 0x19;
54 foo.m[2] = 0x21;
55
56 cp(&foo);
57 cp2(&foo);
58 return 0;
59 }
------------------------------------------------------------------

Below is the running output on an ARM920T board:

before: p1:
0xfc 0x01 0x12
before: p2:
0x40 0x19 0x21
after: p1:
0x40 0x01 0x02
after: p2:
0x40 0x19 0x21
!!! cp is wrong

Remove del for email

 

[Chris Torek]

There has been quite a lot of noise in this thread, but Dik Winter
has, I believe, identified the actual problem:

[I am not sure why the quote marks do not match up as I think the
authors here are correct]

struct Foo {
uint8_t x;
uint8_t y;
uint8_t z;
uint8_t m[3];
};

A compiler can align 'm' any way it wants to, so it may start at any byte
address. And it can require stricter alignment of the struct itself.
Right.

struct Bar
{
uint8_t m[3];
};

A compiler is allowed to require stricter alignment rules for this struct.

Likewise ... and in this case, it does.

Now we hit the interesting bits:

This is a cast that does not necessarily work as you want because
of alignment differences.

The cast happens to work on the ARM, producing a pointer that is
"one byte away" from correct alignment. This would not (and does
not) matter to the compiler in some cases, but...

The next bit of code (which has been snipped here) does a memcpy(),
in effect doing:

memcpy(&bar, p2, 3);

The compiler is allowed to assume that p2 (which can only correctly
point to an actual instance of a "struct Bar") is 4-byte aligned,
and hence emit a "fast and inline" memcpy that does:

load two bytes from address given by p2
store two bytes to first two bytes of "bar"
load third byte from address given by p2+2
store third byte to third byte of "bar"

Here is where things get interesting. On the ARM, if you supply an
unaligned address to a load or store instruction that is loading or
storing more than one byte, the machine IGNORES THE LOW ORDER BITS.
In effect, this "means":

*(short *)&bar = *(short *)((uintptr_t)p2 & ~1);
*((char *)&bar + 2) = *((char *)p2 + 2);

Since p2 has the low order bit set, the hardware's "& ~1" (clear
low bit) has the effect of subtracting 1. So bar.m[0] will actually
take on the value of foo->z and bar.m[1] will take on the value of
foo->m[0], while bar.m[2] will take on the value of foo->m[2].

The lesson here is the same as always: "If you lie to the compiler,
it will get its revenge."

 

[Dik T. Winter]

> I think I may actually have worked on a system that has padding bits
> in some integer types. I didn't happen to do anything that was
> affected by this, and I didn't check the values from <limits.h> while
> I still had access to the system, so I can't be sure. The system was
> a Cray T90; I'm fairly sure that it or some of its predecessors used
> something like 24 out of 64 bits for some integer types. If I can get
> access to a Cray again, I'll check it out.

24 bits was used for 'short' and as a compiler option for 'int'. But
'int' itself was only 48 bits out of 64. And if the upper 16 bits were
not all zero you could get a big surprise when multiplying such things.

(When not all 16 bits of both operands were zero the multiplying unit
would assume that the operands were floating point and operate
accordingly.)

 

[Keith Thompson]

24 bits was used for 'short' and as a compiler option for 'int'. But
'int' itself was only 48 bits out of 64. And if the upper 16 bits were
not all zero you could get a big surprise when multiplying such things.

(When not all 16 bits of both operands were zero the multiplying unit
would assume that the operands were floating point and operate
accordingly.)

Thanks.

There was never a C99 compiler for the Cray T90, and the concepts of
"padding bits" and "trap representations" were introduced by C99.

However, we have demonstrated by example a fairly recent system (still
in production use within the last 10 years) on which integer types can
have bits that don't contribute to the value, but which are not
ignored, resulting in incorrect results if those bits take on certain
values. This admittedly odd design was not implemented for the fun of
it; it was done for perfectly valid reasons.

And it's entirely plausible that something similar could show up in a
future design.

jacob would have us pretend that this just isn't possible. It's
basically the same thing as the old "All the world's a VAX" syndrome,
or its modern equivalent the "All the world's an x86" syndrome.

 

[Antoninus Twink]

My only point was to illustrate that machines, that you claimed only exist
"in the minds of the regulars" exist elsewhere too.

It's really very simple. If someone has a question about programming in
C on a mainframe from the stone-age with trap representations and the
like, or if they're programming an embedded system with 16-bit chars and
half the standard library missing, then all they have to do is make that
clear from the beginning, and then that will be the context for that
thread.

Otherwise, doesn't it make sense to work on the assumption that they're
part of the 99.9% of the world using a modern operating system on modern
hardware? Anywhere apart from in the autistic heart of clc that wouldn't
be controversial in the slightest, merely basic common sense.

 

[jacob navia]

It's really very simple. If someone has a question about programming in
C on a mainframe from the stone-age with trap representations and the
like, or if they're programming an embedded system with 16-bit chars and
half the standard library missing, then all they have to do is make that
clear from the beginning, and then that will be the context for that
thread.

Otherwise, doesn't it make sense to work on the assumption that they're
part of the 99.9% of the world using a modern operating system on modern
hardware? Anywhere apart from in the autistic heart of clc that wouldn't
be controversial in the slightest, merely basic common sense.

Please do not speak about "common sense" here. It is not mentioned
in the C standard of 1989.

 

[santosh]

It's really very simple. If someone has a question about programming
in C on a mainframe from the stone-age with trap representations and
the like, or if they're programming an embedded system with 16-bit
chars and half the standard library missing, then all they have to do
is make that clear from the beginning, and then that will be the
context for that thread.

Otherwise, doesn't it make sense to work on the assumption that
they're part of the 99.9% of the world using a modern operating system
on modern hardware? Anywhere apart from in the autistic heart of clc
that wouldn't be controversial in the slightest, merely basic common
sense.

When a poster makes non-portable assumptions in his source code, it
makes sense to bring them to notice, since tomorrow that code may have
to be ported to systems where those assumptions are no longer valid.

If a poster is absolutely certain that his code will be confined only to
certain systems, then IMO, it makes more sense for that poster to take
full advantage of extensions offered by his set of platforms and
perhaps post in groups specifically set aside for them.

 

[Jean-Marc Bourguet]

Yes, it is not IMPOSSIBLE that somewhere in the planet, there is still some
B6700 running around using floating point for integers... who
knows?

Unisys is still selling mainframes using descendants of this architecture
(as well as mainframes using descendant of Unix 1100, which are ones
complement machine).

How much C programming is done on them, that's another question.

But if you consider the host of strange embedded architectures that exists
out it the wild -- most of them you'll be aware of only if you are in the
company making them -- I'd be surprised if there is currently none of them
with a C compiler and trap integer representations. On another variance
axis, XKL considers usefull to maintain a port of gcc for the PDP-10, a 36
bits, word adressable machine DEC announced the cancellation 25 years ago
(see http://gcc.gnu.org/ml/gcc/2008-04/msg00506.html).

Yours,

 

[jacob navia]

Unisys is still selling mainframes using descendants of this architecture
(as well as mainframes using descendant of Unix 1100, which are ones
complement machine).

How much C programming is done on them, that's another question.

But if you consider the host of strange embedded architectures that exists
out it the wild -- most of them you'll be aware of only if you are in the
company making them -- I'd be surprised if there is currently none of them
with a C compiler and trap integer representations. On another variance
axis, XKL considers usefull to maintain a port of gcc for the PDP-10, a 36
bits, word adressable machine DEC announced the cancellation 25 years ago
(see http://gcc.gnu.org/ml/gcc/2008-04/msg00506.html).

Yours,

That is why we have to tell this to people that wouldn't care less when
they ask a question here. So that we can display our knowledge to them.


UNISYS: They market mainframes. Yes. For instance the MCP "Clear path"
series:

http://www.unisys.com/products/mainframes/mcp__mainframes/index.htm
<quote>
Mid-range MCP systems: Libra Model 400 Server
Libra Model 400 Servers are fully featured, entry and mid-range
mainframe systems for MCP environments. These new servers are based on
the ClearPath Next-Generation Server Architecture and feature quad-core
Intel processors running both MCP and Windows operating systems.
<end quote>

Another product line, the OS2000 series:
<quote>

http://www.unisys.com/products/mainframes/os__2200__mainframes/index.htm
New Next-Generation OS 2200 Servers: Dorado 400 Series

The ClearPath Dorado 400 Series is the first Dorado family of servers
based on the ClearPath Next-Generation Server Architecture. These
entry-level servers run the OS 2200 operating system on Intel® processors.
<end quote>

Those are the two main lines of UNISYS sold TODAY!
-----------------------------------------------------------------------------

As for the unix 1100 that you mention, that product line started as
follows:
1. UNIVAC 1107 introduced in 1962
2. UNIVAC 1108 introduced in 1965
3. UNIVAC 1106 introduced in 1969
4. UNIVAC 1110 introduced in 1970
5. UNIVAC 1100/10 redesignation of UNIVAC 1106 in 1975
6. UNIVAC 1100/20 redesignation of UNIVAC 1108 in 1975
7. UNIVAC 1100/40 redesignation of UNIVAC 1110 in 1975
8. UNIVAC 1100/80 introduced in 1975
9. UNIVAC 1100/80A introduced in 1977
10. UNIVAC 1100/60 introduced in 1979
11. UNIVAC 1100/70 introduced in 1981
12. UNIVAC 1100/90 introduced in 1982

And there it STOPS! 26 YEARS AGO.
---------------------------------------------------------------------------

That is why we have to tell people asking questions here about that. It
is ESSENTIAL that they know the problems they *could* have if they port
their software to an UNISYS mainframe emulator running in their windows
machine!

 

[Jean-Marc Bourguet]

That is why we have to tell this to people that wouldn't care less when
they ask a question here. So that we can display our knowledge to them.

I'm not the one who started to write about that, I'm just adding some
information.

UNISYS: They market mainframes. Yes. For instance the MCP "Clear path"
series:

Another product line, the OS2000 series:

Those are the two main lines of UNISYS sold TODAY!

Yes, but you seem to have missed that the 2200 series is a direct
descendant, still backward compatible, of the 1107 Univac, and the MCP is a
direct descendant, still backward compatible, of the Burrough B6500.

Yours,

 

[santosh]

[ ... ]

That is why we have to tell this to people that wouldn't care less
when they ask a question here. So that we can display our knowledge to
them.

[ ... ]

That is why we have to tell people asking questions here about that.
It is ESSENTIAL that they know the problems they *could* have if they
port their software to an UNISYS mainframe emulator running in their
windows machine!

As long as the Standard recognises the possibility of trap
representations, then maximally portable code has to take it into
account. It doesn't matter whether actual machines exist. It matters
that the language Standard allows for the possibility in it's abstract
machine model. A maximally portable program should be able to run and
produce correct output on a theoretical implementation that implements
every possible feature specified by ISO C.

Given the "quality" of most "C/C++" web forums, you should be glad that
a high quality group like this still exists. After all, you have, on
numerous occasions, had your own questions and misunderstandings
cleared through this group (and comp.std.c).

 

[Bartc]

Antoninus Twink wrote:

When a poster makes non-portable assumptions in his source code, it
makes sense to bring them to notice, since tomorrow that code may have
to be ported to systems where those assumptions are no longer valid.

Would it be possible for someone to design a new computer system that would
break a C program that is otherwise 100% portable today and adheres to
everything in the Standard?

And I mean a general purpose machine not something specifically concocted to
break C.

 

[santosh]

Would it be possible for someone to design a new computer system that
would break a C program that is otherwise 100% portable today and
adheres to everything in the Standard?

Yes. If the system is unable to support a conforming implementation of
ISO C.

However the Committee has taken substantial care in keeping the
requirements of C as low as is feasible, so that ISO C can be
implemented on as broad a variety of machines as possible. Naturally
such an effort is likely to introduce some complexity that many people
will consider as unnecessary.

And I mean a general purpose machine not something specifically
concocted to break C.

I am not aware of "general purpose" machines, but some embedded systems
are unable to support a hosted implementation, and I wouldn't be
surprised to hear of some that cannot support a conforming freestanding
one either, though I don't know of any examples.

But the point is this. It's *very* easy to write C code that has
numerous subtle portability problems. This needn't be done
deliberately, but simply because most C programmers do not (perhaps
understandably) understand all the ins-and-outs of Standard C. In fact
many do not even know that a Standard (and a public draft of it)
exists. I can attest to this personally. My own C code of say five
years ago has many more non-portable constructs and assumptions than
that of today. I credit the improvement to this group. Web forums are
great for help, yes, but in my experience they do not have the kind of
expertise on *Standard* C that clc and csc have. That's why it's worth
maintaining the high levels of standard that this group still has.

 

[Jean-Marc Bourguet]

Would it be possible for someone to design a new computer system that would
break a C program that is otherwise 100% portable today and adheres to
everything in the Standard?

Answering that would need an exhaustive knowledge about existing
implementation that probably nobody has.

And I mean a general purpose machine not something specifically concocted
to break C.

BTW, there is quite a big space between a general purpose machine and
something specifically concocted to break C.

And don't forget the tendency in optimizer to use whatever room the
standard leave them in creative ways.

Yours,

 

[Antoninus Twink]

Yes. If the system is unable to support a conforming implementation of
ISO C.

Wonderful. So now we're meant to worry about portability to hardware
that can't even support an implementation of C. Should we also spend our
time fretting about whether our code is maximally portable enough to
keep working when the hardware melts down in a nuclear winter?

 

[santosh]

On 27 Jul 2008 at 12:56, santosh wrote:

Wonderful. So now we're meant to worry about portability to hardware
that can't even support an implementation of C.

You have extracted my statement from it's context. Here it is, for the
benefit of anyone who might take Antoninus's post seriously:

Bartc wrote:
Would it be possible for someone to design a new computer system that
would break a C program that is otherwise 100% portable today and
adheres to everything in the Standard?

I replied:
Yes. If the system is unable to support a conforming implementation of
ISO C.

<snip>