Strict aliasing and buffer handling

F

Francois Duranleau

Hi,

I am facing a situation where I would like to follow strict aliasing,
but, unless I misunderstand the standard, I can't find a solution
without sacrificing performance, memory size, etc.

The problem is as follows. Essentially, I have a runtime description
of a buffer's layout in memory. The information is essentially a set
of offset (in bytes) with type of data located at that offset. There
is an API I use that needs the data to be sent in that particular
layout. A common method of dealing with such a buffer is something
like this:

- Buffer is allocated like this:

char* buffer = new char[buffer_size];

- Assigning/querying data from that buffer is done with something like
this:

template < typename T >
void set_buffer_data(char* buffer, unsigned int offset, const T& data)
{
T* d = reinterpret_cast<T*>(buffer + offset);
*d = data;
}

template < typename T >
const T& get_buffer_data(const char* buffer, unsigned int offset)
{
const T* d = reinterpret_cast<const T*>(buffer + offset);
return *d;
}

I am volontarily simplifying things to better illustrate the problem,
so never mind alignment and type safety problems, and proper
initialization of the buffer's content with non-POD types.

Now, if I read the standard correctly (3.10-15), this is a violation
of strict aliasing because casting from char* to T* (T != char or
unsigned char) and then accessing the data via that pointer is not
allowed. But then, how can we handle this without sacrificing
performance and memory?

We could do memcpy, e.g.

template < typename T >
void set_buffer_data(char* buffer, unsigned int offset, const T& data)
{
memcpy(buffer + offset, &data, sizeof(T));
}

template < typename T >
void get_buffer_data(const char* buffer, unsigned int offset, T& data)
{
memcpy(&data, buffer + offset, sizeof(T));
}

and hope the compiler optimizes this well. But then, what about non-
PODs? And get_buffer_data now needs to return a copy all the times (I
could live with that, though, as most of the times, the return value
would be assigned to a non-reference variable anyways).

For the record, a concrete example for this problem is when dealing
with OpenGL's uniform buffers (I guess it would be similar with
D3D10+'s constant buffers). At runtime, you get a layout description
of a buffer, and you have to respect that layout to send the data.
 
P

Paul

Francois Duranleau said:
Hi,

I am facing a situation where I would like to follow strict aliasing,
but, unless I misunderstand the standard, I can't find a solution
without sacrificing performance, memory size, etc.

The problem is as follows. Essentially, I have a runtime description
of a buffer's layout in memory. The information is essentially a set
of offset (in bytes) with type of data located at that offset. There
is an API I use that needs the data to be sent in that particular
layout. A common method of dealing with such a buffer is something
like this:

- Buffer is allocated like this:

char* buffer = new char[buffer_size];

- Assigning/querying data from that buffer is done with something like
this:

template < typename T >
void set_buffer_data(char* buffer, unsigned int offset, const T& data)
{
T* d = reinterpret_cast<T*>(buffer + offset);
*d = data;
}

template < typename T >
const T& get_buffer_data(const char* buffer, unsigned int offset)
{
const T* d = reinterpret_cast<const T*>(buffer + offset);
return *d;
}

I am volontarily simplifying things to better illustrate the problem,
so never mind alignment and type safety problems, and proper
initialization of the buffer's content with non-POD types.

Now, if I read the standard correctly (3.10-15), this is a violation
of strict aliasing because casting from char* to T* (T != char or
unsigned char) and then accessing the data via that pointer is not
allowed. But then, how can we handle this without sacrificing
performance and memory?

We could do memcpy, e.g.

template < typename T >
void set_buffer_data(char* buffer, unsigned int offset, const T& data)
{
memcpy(buffer + offset, &data, sizeof(T));
}

template < typename T >
void get_buffer_data(const char* buffer, unsigned int offset, T& data)
{
memcpy(&data, buffer + offset, sizeof(T));
}

and hope the compiler optimizes this well. But then, what about non-
PODs? And get_buffer_data now needs to return a copy all the times (I
could live with that, though, as most of the times, the return value
would be assigned to a non-reference variable anyways).

For the record, a concrete example for this problem is when dealing
with OpenGL's uniform buffers (I guess it would be similar with
D3D10+'s constant buffers). At runtime, you get a layout description
of a buffer, and you have to respect that layout to send the data.

--
Click to expand...
Have you considered placement new ?
Maybe that is another option for you.
 
F

Francois Duranleau

Have you considered placement new ?
Maybe that is another option for you.
Click to expand...

Yes, but it does not solve the strict aliasing issue, because before
assigning, we would need to call the destructor for what was there,
but then we would also need to make a cast to call this destructor,
e.g.

template < typename T >
void set_buffer_value(char* buffer, unsigned int offset, const T&
value)
{
// line below still violates strict aliasing
reinterpret_cast<T*>(buffer + offset)->~T();
new (buffer + offset) T(value);
}
 
F

Francois Duranleau

AFAIK char* is an allowable exception to strict aliasing rules.
Click to expand...

I would like to believe so, but in the standard, it says you can
access the storage of anything via char* or unsigned char*, not the
other way around, which is what bugs me.

There was a discussion about that on Boost developers mailing list
(http://groups.google.com/group/boost-developers-archive/browse_thread/
thread/aa1bd1c1f285f8f8/913ad1a9678ac67e) concerning Boost.Variant
(http://www.boost.org/doc/libs/1_46_1/doc/html/variant.html), which
uses a static char buffer to hold data for various types. A quote from
the thread:

"I thought variant just static_cast<char*> to void* then from void* to
T*. The char* is for the memory buffer in boost::aligned_storage."

which indeed what it does. However, later in the thread, it is also
questioned how this would actually solve the strict aliasing rule
problem, and then there was no answer, except complaints about the
said rule. I tried to look in the standard for specials cases with
static_cast and char*/unsigned char* and void*, but I found nothing of
interest... well, assuming I understood everything where I looked (no
guaranties :) ). I neither read the cover-to-cover 700+ pages.

So I thought there might be some C++ gurus out here with an answer.
 
P

Paul

Leigh Johnston said:
Yeah it bugs me too (I have the same problem with my own variant class, my
own max-size vector class and my own chunk allocator).

What about std::allocator? By default it will be creating/destroying
objects within a char array obtained via new.
Click to expand...
Applying your theory , the char* is pointing to only the first byte,
therefore the rest of the memory is free to be pointed to by anything, no?
:) lol
 
A

Alf P. Steinbach /Usenet

* Francois Duranleau, on 20.06.2011 21:39:
I would like to believe so, but in the standard, it says you can
access the storage of anything via char* or unsigned char*, not the
other way around, which is what bugs me.

There was a discussion about that on Boost developers mailing list
(http://groups.google.com/group/boost-developers-archive/browse_thread/
thread/aa1bd1c1f285f8f8/913ad1a9678ac67e) concerning Boost.Variant
(http://www.boost.org/doc/libs/1_46_1/doc/html/variant.html), which
uses a static char buffer to hold data for various types. A quote from
the thread:

"I thought variant just static_cast<char*> to void* then from void* to
T*. The char* is for the memory buffer in boost::aligned_storage."

which indeed what it does. However, later in the thread, it is also
questioned how this would actually solve the strict aliasing rule
problem, and then there was no answer, except complaints about the
said rule. I tried to look in the standard for specials cases with
static_cast and char*/unsigned char* and void*, but I found nothing of
interest... well, assuming I understood everything where I looked (no
guaranties :) ). I neither read the cover-to-cover 700+ pages.

So I thought there might be some C++ gurus out here with an answer.
Click to expand...

Strict aliasing is a g++ thing, not a standard C++ thing: the standard is The
Wrong Place to look for info about strict aliasing, it's not there.

Standard C++ does however have rules about alignment.

As a practical matter, if you break your system's alignment rules, then in
practice you get extreme inefficiency or a program crash.


Cheers & hth.,

- Alf
 
F

Francois Duranleau

Strict aliasing is a g++ thing, not a standard C++ thing: the standard isThe
Wrong Place to look for info about strict aliasing, it's not there.
Click to expand...

Quoting 3.10-15 from the C++ standard (2003):

"If a program attempts to access the stored value of an object through
an lvalue of other than one of the following
types the behavior is undefined):
— the dynamic type of the object,
— a cv-qualified version of the dynamic type of the object,
— a type that is the signed or unsigned type corresponding to the
dynamic type of the object,
— a type that is the signed or unsigned type corresponding to a cv-
qualified version of the dynamic type of the object,
— an aggregate or union type that includes one of the aforementioned
types among its members (including, recursively, a member of a
subaggregate or contained union),
— a type that is a (possibly cv-qualified) base class type of the
dynamic type of the object,
— a char or unsigned char type."

So what I am reading here?
Standard C++ does however have rules about alignment.

As a practical matter, if you break your system's alignment rules, then in
practice you get extreme inefficiency or a program crash.
Click to expand...

Of course.
 
A

Alf P. Steinbach /Usenet

* Francois Duranleau, on 20.06.2011 22:29:
Quoting 3.10-15 from the C++ standard (2003):

"If a program attempts to access the stored value of an object through
an lvalue of other than one of the following
types the behavior is undefined):
— the dynamic type of the object,
— a cv-qualified version of the dynamic type of the object,
— a type that is the signed or unsigned type corresponding to the
dynamic type of the object,
— a type that is the signed or unsigned type corresponding to a cv-
qualified version of the dynamic type of the object,
— an aggregate or union type that includes one of the aforementioned
types among its members (including, recursively, a member of a
subaggregate or contained union),
— a type that is a (possibly cv-qualified) base class type of the
dynamic type of the object,
— a char or unsigned char type."

So what I am reading here?
Click to expand...

A flawed attempt at enumerating the ways, which attempt was/is presumably the
perceived rationale for g++ "strict aliasing".

Of course.
Click to expand...

Well, that's all, if your data is correct representation of the object.

Just tell g++ to shut up. :)


Cheers & hth.,

- Alf
 
J

Joshua Maurice

* Francois Duranleau, on 20.06.2011 22:29:













A flawed attempt at enumerating the ways, which attempt was/is presumablythe
perceived rationale for g++ "strict aliasing".



Well, that's all, if your data is correct representation of the object.

Just tell g++ to shut up. :)
Click to expand...

The C standards committee disagrees with you good sir.
 
K

Kai-Uwe Bux

Alf said:
* Francois Duranleau, on 20.06.2011 22:29:

A flawed attempt at enumerating the ways, which attempt was/is presumably
the perceived rationale for g++ "strict aliasing".
Click to expand...

What is the flaw?

I don't think the flaw is obvious (it isn't to me). As evidence (not that it
isn't flawed, but that's it is not _obviously_ flawed) I would offer that an
adapted version of the list made it into n3291 as 3.10/10. So, it appears,
the committee did not see a flaw; or is there a core issue about this that I
am unaware of? or is the flaw just fixed by passing to glvalues etc.?


Best,

Kai-Uwe Bux
 
J

Joshua Maurice

Hi,

I am facing a situation where I would like to follow strict aliasing,
but, unless I misunderstand the standard, I can't find a solution
without sacrificing performance, memory size, etc.

The problem is as follows. Essentially, I have a runtime description
of a buffer's layout in memory. The information is essentially a set
of offset (in bytes) with type of data located at that offset. There
is an API I use that needs the data to be sent in that particular
layout. A common method of dealing with such a buffer is something
like this:

- Buffer is allocated like this:

char* buffer = new char[buffer_size];

- Assigning/querying data from that buffer is done with something like
this:

template < typename T >
void set_buffer_data(char* buffer, unsigned int offset, const T& data)
{
    T* d = reinterpret_cast<T*>(buffer + offset);
    *d = data;

}

template < typename T >
const T& get_buffer_data(const char* buffer, unsigned int offset)
{
    const T* d = reinterpret_cast<const T*>(buffer + offset);
    return *d;

}

I am volontarily simplifying things to better illustrate the problem,
so never mind alignment and type safety problems, and proper
initialization of the buffer's content with non-POD types.

Now, if I read the standard correctly (3.10-15), this is a violation
of strict aliasing because casting from char* to T* (T != char or
unsigned char) and then accessing the data via that pointer is not
allowed. But then, how can we handle this without sacrificing
performance and memory?

We could do memcpy, e.g.

template < typename T >
void set_buffer_data(char* buffer, unsigned int offset, const T& data)
{
    memcpy(buffer + offset, &data, sizeof(T));

}

template < typename T >
void get_buffer_data(const char* buffer, unsigned int offset, T& data)
{
    memcpy(&data, buffer + offset, sizeof(T));

}

and hope the compiler optimizes this well. But then, what about non-
PODs? And get_buffer_data now needs to return a copy all the times (I
could live with that, though, as most of the times, the return value
would be assigned to a non-reference variable anyways).

For the record, a concrete example for this problem is when dealing
with OpenGL's uniform buffers (I guess it would be similar with
D3D10+'s constant buffers). At runtime, you get a layout description
of a buffer, and you have to respect that layout to send the data.
Click to expand...

Ok, if you need to convert an actual char buffer to a POD struct, then
I think you're right and you're screwed - you would need to copy the
data byte by byte into an object of dynamic type POD struct.

If you need to go the other way, then a simple cast suffices. You are
allowed to read a POD object through a char lvalue.

Optionally, do you need code portability? gcc -fno-strict-aliasing may
be for you.
 
A

Alf P. Steinbach /Usenet

* Joshua Maurice, on 20.06.2011 23:37:
The C standards committee disagrees with you good sir.
Click to expand...

I rather doubt that the C standards committee are discussing my C++ Usenet
postings (or even my C Usenet postings).

But perhaps relevant quotes from the C standard could be useful to the OP, in
order to relate to the g++ compiler?


Cheers & hth.,

- Alf
 
A

Alf P. Steinbach /Usenet

* Kai-Uwe Bux, on 20.06.2011 23:38:
What is the flaw?
Click to expand...

Just as an example that the above list does not cover everything, the intro to
classes says that if class A is a POD, and its first member is a double, say,
then if you have a pointer to an A then you can reinterpret_cast it to pointer
to double to access that first member. It's for C compatibility.

And that's not covered by the list, i.e. there's at least one engine stutter:
the list says it's UB, later on in the standard it's said to be well-defined. ;-)

I don't know exactly what the flaw or set of flaws is, in the sense of not
knowing exactly why an engine stutters (i.e., cause). Perhaps just the attempt
to cover everything. It has been discussed at length, many times, as I recall
without any good consensus, but it hould be possible to find those discussions.

I don't think the flaw is obvious (it isn't to me). As evidence (not that it
isn't flawed, but that's it is not _obviously_ flawed) I would offer that an
adapted version of the list made it into n3291 as 3.10/10. So, it appears,
the committee did not see a flaw; or is there a core issue about this that I
am unaware of? or is the flaw just fixed by passing to glvalues etc.?
Click to expand...

The committee is not perfect, nor does it have the capacity to deal with everything.

I think there's a lot of flawed stuff migrating straight over into the new standard.


Cheers & hth.,

- Alf
 
J

Joshua Maurice

* Joshua Maurice, on 20.06.2011 23:37:




I rather doubt that the C standards committee are discussing my C++ Usenet
postings (or even my C Usenet postings).

But perhaps relevant quotes from the C standard could be useful to the OP, in
order to relate to the g++ compiler?
Click to expand...

From some of the recent defect reports and their resolutions which
I've been reading, it's clear that the C standards committee does
intend the simple reading of the strict aliasing rules, that you may
not access an object with a differently typed lvalue, unless that
lvalue is char or unsigned char. Of course, it's playing hell with
unions, and then there's my defect (which hasn't made it to defect
status yet) about userspace general purpose memory pooling allocators
in that comp.std.c++ thread. (I even tried to bring this over to
comp.std.c, and the discussion went nowhere fast for 70+ posts.) I
hope someone who can do something about it saw one of the two
discussions, and fixes it the right way.
 
I

Ian Collins

"Francois Duranleau"<[email protected]> ha scritto nel messaggio

i don't know if this is right, it is only one try it is something about:
----------
#include<stdint.h>
#include<stdlib.h>
#include<iostream.h>

// macro for types
#define u64 uint64_t
#define u32 uint32_t
#define u16 uint16_t
#define u8 uint8_t
#define uns unsigned

// macro for function
#define P printf
Click to expand...

NO!
 
P

Paul

Leigh Johnston said:
You think you are being funny and/or clever but the only thing that is
laughable is your technical competence. Get a life and stop trolling
here.
Click to expand...
But according to you with :
char* p = new char[100];
p only points to the first char. There are 99 unpointed-to chars.


There is nothing wrong with being incorrect now and then but you refuse to
accept your errors.
So I don't think I am being funny , I actually think you are funny. :)
 
P

Paul

Leigh Johnston said:
Leigh Johnston said:
AFAIK char* is an allowable exception to strict aliasing rules.

I would like to believe so, but in the standard, it says you can
access the storage of anything via char* or unsigned char*, not the
other way around, which is what bugs me.

There was a discussion about that on Boost developers mailing list
(http://groups.google.com/group/boost-developers-archive/browse_thread/

thread/aa1bd1c1f285f8f8/913ad1a9678ac67e) concerning Boost.Variant
(http://www.boost.org/doc/libs/1_46_1/doc/html/variant.html), which
uses a static char buffer to hold data for various types. A quote
from
the thread:

"I thought variant just static_cast<char*> to void* then from void*
to
T*. The char* is for the memory buffer in boost::aligned_storage."

which indeed what it does. However, later in the thread, it is also
questioned how this would actually solve the strict aliasing rule
problem, and then there was no answer, except complaints about the
said rule. I tried to look in the standard for specials cases with
static_cast and char*/unsigned char* and void*, but I found nothing
of
interest... well, assuming I understood everything where I looked (no
guaranties :) ). I neither read the cover-to-cover 700+ pages.

So I thought there might be some C++ gurus out here with an answer.

Yeah it bugs me too (I have the same problem with my own variant
class, my own max-size vector class and my own chunk allocator).

What about std::allocator? By default it will be creating/destroying
objects within a char array obtained via new.

Applying your theory , the char* is pointing to only the first byte,
therefore the rest of the memory is free to be pointed to by
anything, no?
:) lol

You think you are being funny and/or clever but the only thing that is
laughable is your technical competence. Get a life and stop trolling
here.
Click to expand...
But according to you with :
char* p = new char[100];
p only points to the first char. There are 99 unpointed-to chars.
Click to expand...

Correct, p points the the first element of the array; it does not point to
the second, third, fourth or any other other element of the array. Read
the standard about what new[] actually returns.
Click to expand...
It amazes me how you manage to be so irritating yet amusing at the same
time. LOL.
 
F

Francois Duranleau

* Kai-Uwe Bux, on 20.06.2011 23:38:



Just as an example that the above list does not cover everything, the intro to
classes says that if class A is a POD, and its first member is a double, say,
then if  you have a pointer to an A then you can reinterpret_cast it topointer
to double to access that first member. It's for C compatibility.

And that's not covered by the list, i.e. there's at least one engine stutter:
the list says it's UB, later on in the standard it's said to be well-defined. ;-)
Click to expand...

Well, actually, it is covered by this point:

"- an aggregate or union type that includes one of the aforementioned
types among its members (including, recursively, a member of a
subaggregate or contained union)"

although not very clearly.
 
A

Alf P. Steinbach /Usenet

* Francois Duranleau, on 21.06.2011 14:03:
Well, actually, it is covered by this point:

"- an aggregate or union type that includes one of the aforementioned
types among its members (including, recursively, a member of a
subaggregate or contained union)"

although not very clearly.
Click to expand...

You have that backwards. In the example above, an `A` object is accessed as a
`double`. A `double` is not an aggregate or union.

But presumably you would not have posted the article I'm replying to now, if you
agreed with that.

However, even if it seems to you to be the other way around, consider that the
standard's list only works one way, while later on in the standard, a two way
conversion is defined -- one of those 2 directions is necessarily missing.


Cheers & hth.,

- Alf
 
F

Francois Duranleau

* Francois Duranleau, on 21.06.2011 14:03:





You have that backwards. In the example above, an `A` object is accessed as a
`double`. A `double` is not an aggregate or union.

But presumably you would not have posted the article I'm replying to now,if you
agreed with that.

However, even if it seems to you to be the other way around, consider that the
standard's list only works one way, while later on in the standard, a twoway
conversion is defined  --  one of those 2 directions is necessarily missing.
Click to expand...

Indeed, I misread your example, sorry. What was I thinking.

Do you know where in the standard where we can find that two way
conversion definition?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

char and strict aliasing 4
Union and strict aliasing 4
Weird things when calculating the size for a mergesort-buffer 0
warning of breaking strict-aliasing rules 9
strict-aliasing?? 5
Strict aliasing rule: pointer to void vs. pointer to char and transitivity 3
String buffer overruns? 8
Strict Pointer Aliasing Question 2

Members online

Total: 18 (members: 1, guests: 17)
Robots: 224

Forum statistics

Threads
473,764
Messages
2,569,566
Members
45,041
Latest member
RomeoFarnh

Latest Threads

Top