Encoding Return Address and Stack Pointer

[Casey Hawthorne]

Making the stack area non-executable will not help against stack based
buffer overruns, since the return address could be changed to an
executable area of memory.

A solution could be that the C compiler is setup so that when a
function call is made, it is set up to encode the return address
placed on the stack with the stack pointer (SP) and possibly some fast
encryption technique.
Then if the return address is altered, when the return address is
unencoded/decrypted and found to be invalid, the application would
stop with an invalid return address message sent back to the O/S.

I realize this would eat up quite a few cycles, so it would be better
if it was done in hardware.

Tail Call Optimization (TCO) would still work, since no additional
stack frames are pushed on the stack, so the return address would not
be altered.

 

[Gene]

Making the stack area non-executable will not help against stack based
buffer overruns, since the return address could be changed to an
executable area of memory.

A solution could be that the C compiler is setup so that when a
function call is made, it is set up to encode the return address
placed on the stack with the stack pointer (SP) and possibly some fast
encryption technique.
Then if the return address is altered, when the return address is
unencoded/decrypted and found to be invalid, the application would
stop with an invalid return address message sent back to the O/S.  

I realize this would eat up quite a few cycles, so it would be better
if it was done in hardware.

Tail Call Optimization (TCO) would still work, since no additional
stack frames are pushed on the stack, so the return address would not
be altered.

It would be far simpler to keep return addresses on a separate stack
that's inaccessible from any pointer within a data frame.

 

[BGB / cr88192]

Making the stack area non-executable will not help against stack based
buffer overruns, since the return address could be changed to an
executable area of memory.

A solution could be that the C compiler is setup so that when a
function call is made, it is set up to encode the return address
placed on the stack with the stack pointer (SP) and possibly some fast
encryption technique.
Then if the return address is altered, when the return address is
unencoded/decrypted and found to be invalid, the application would
stop with an invalid return address message sent back to the O/S.

I realize this would eat up quite a few cycles, so it would be better
if it was done in hardware.

Tail Call Optimization (TCO) would still work, since no additional
stack frames are pushed on the stack, so the return address would not
be altered.

this is OT here but:
Linux uses the trick of randomizing load addresses, locations for allocated
memory, ... in this way, overflowing the stack and getting ar the return
address will be unable to readily produce usable results, since on each run
of an app the memory layout is different.


even if the return address/... were mangled, one would still need a
randomized feature for it to be effective, and it would cost in terms of
performance, ...

randomized load addresses are, thus, much cheaper, just not as ideal on
Windows since it would then requiring rebasing DLLs on load, which reduces
Windows ability to use shared memory for them.

a possible option (if implemented in the OS), would be to essentially ignore
the default load address, and instead re-assign the DLL to some randomized
address on a per-boot basis (or periodically for longer-running systems).
this could still allow a moderate level of memory sharing (since the DLLs
would be largely rebased in a manner consistent between running processes).

similarly, it could actually improve the situation considering how often
DLL's tend to use the "default" load address, it could actually cause them
to be rebased more consistently.

some level of randomization WRT the heap layout, locations of the stack, ...
also makes sense.

the one downside is that with 32-bit processes, this does notably increase
the risk of unusably fragmenting the address space if full randomness is
allowed, but an "ordered random" strategy is less likely to produce good
levels of chaos.

but at least it could shuffle them around in memory, which may be worthwhile
in its own right.

a related thick could be "turbulent indeterminism" where, in cases where a
full random pattern is not usable, things are fudged such that some level of
"chaos" is added, such that no 2 runs (and maybe no 2 builds) have exactly
the same memory layout.


or such...