Forced inclusion of "unused" objects

[Noah Roberts]

I'm trying to come up with a way to keep the compiler from optimizing
away objects that it doesn't think are used. I'm using the creation of
static objects to cause behavior to occur before the main() function.

So say I have function `int f();` and inside doit.cpp I have:

namespace {
int x = f();
}

and that's it.

Normally, if doit.cpp is within the program being compiled (at least
with VS2005+) x exists, and since it does f() is called before main,
which is actually what I want to happen.

However, now I'm trying to move doit.cpp into a library and I'm having
no luck at all getting it to happen.

Is there a way to force the compiler into keeping x around?

I don't really need something portable if the only answer is compiler
specific. I'm using MSVC10. I'd prefer something standard though.

 

[Noah Roberts]

I'm trying to come up with a way to keep the compiler from optimizing
away objects that it doesn't think are used. I'm using the creation of
static objects to cause behavior to occur before the main() function.

So say I have function `int f();` and inside doit.cpp I have:

namespace {
int x = f();
}

and that's it.

Normally, if doit.cpp is within the program being compiled (at least
with VS2005+) x exists, and since it does f() is called before main,
which is actually what I want to happen.

However, now I'm trying to move doit.cpp into a library and I'm having
no luck at all getting it to happen.

Is there a way to force the compiler into keeping x around?

I don't really need something portable if the only answer is compiler
specific. I'm using MSVC10. I'd prefer something standard though.

I've found my issue and apparently it's a compiler "feature". The
linker eliminates object files that don't contain anything used,
regardless of side effects. Documented switches that you'd expect to
turn this off (like /OPT:NOREF) do not. There doesn't appear to be a
fix; if I want to force behavior to occur before main() it simply can't
be done this way--I'll have to find a way to "use" some symbol in that
object file.

 

[Alf P. Steinbach /Usenet]

* Noah Roberts, on 08.12.2010 18:44:

I'm trying to come up with a way to keep the compiler from optimizing
away objects that it doesn't think are used. I'm using the creation of
static objects to cause behavior to occur before the main() function.

So say I have function `int f();` and inside doit.cpp I have:

namespace {
int x = f();
}

and that's it.

Normally, if doit.cpp is within the program being compiled (at least
with VS2005+) x exists, and since it does f() is called before main,
which is actually what I want to happen.

However, now I'm trying to move doit.cpp into a library and I'm having
no luck at all getting it to happen.

Is there a way to force the compiler into keeping x around?

I don't really need something portable if the only answer is compiler
specific. I'm using MSVC10. I'd prefer something standard though.

You need to use something from the relevant translation unit.

Whatever, but a function call is the common device.


Cheers & hth.,

- Alf

 

[Bo Persson]

I've found my issue and apparently it's a compiler "feature". The
linker eliminates object files that don't contain anything used,
regardless of side effects. Documented switches that you'd expect
to turn this off (like /OPT:NOREF) do not. There doesn't appear to
be a fix; if I want to force behavior to occur before main() it
simply can't be done this way--I'll have to find a way to "use"
some symbol in that object file.

That's the general idea of a library - put together a bunch of object
files and have the linker pull out the ones you need. Otherwise every
program would contain a full copy of the entire standard library!


Bo Persson

 

[Joshua Maurice]

That's the general idea of a library - put together a bunch of object
files and have the linker pull out the ones you need. Otherwise every
program would contain a full copy of the entire standard library!

For static linking yes, but two qualms.

First, for windows / unix-like systems, don't most people link to a
shared lib (aka dll) libc nowadays anyway? (I don't know much about
other systems.)

Second, this is due to what many consider a bug in the standard. There
is an allowance in the standard which says that the namespace scope
initialization of a translation unit may be delayed until first use.
This is a piss-poor attempt at making shared libs aka dlls standard
complying. In practice, it's basically impossible for a compiler +
linker to actually delay the initialization a finite amount of time
without programmer intervention, such as during the creation of a
shared lib.

Unfortunately, some people got the idea that this is a cool allowance
for an optimization like the OP suggests. I'm not actually sure if
that's what the OP is hitting, but it may be. Last I heard only a
couple of obscure systems were doing it, not gcc.

Also, this doesn't automatically preclude your use case of statically
linking to a large static lib, and you only want to pull in the used
parts. That's perfectly fine. If the unused parts lack side effects,
and the compiler + linker is able to prove it, then it's fine to
remove it under the "as if" rule. I honestly don't know how hard such
analysis is. I know the Microsoft Windows Visual Studios compiler +
linker does this analysis to some degree. (Specifically, I think 2003,
2005, and/or 2008 don't report a missing member function definition if
that class is never used. Another annoying inconvenience when porting
to unix-like systems.)

Finally, is this use case all that common? How many large static libs
are there out there which you would static link to it and only want to
use a small subset of its functionality? I'd /guess/ that either you
control the lib and can split it up, or it's some third party lib
think that's not under your control, and that sounds like it would be
a shared lib.

 

[Stuart Golodetz]

For static linking yes, but two qualms.

First, for windows / unix-like systems, don't most people link to a
shared lib (aka dll) libc nowadays anyway? (I don't know much about
other systems.)

Second, this is due to what many consider a bug in the standard. There
is an allowance in the standard which says that the namespace scope
initialization of a translation unit may be delayed until first use.
This is a piss-poor attempt at making shared libs aka dlls standard
complying. In practice, it's basically impossible for a compiler +
linker to actually delay the initialization a finite amount of time
without programmer intervention, such as during the creation of a
shared lib.

Unfortunately, some people got the idea that this is a cool allowance
for an optimization like the OP suggests. I'm not actually sure if
that's what the OP is hitting, but it may be. Last I heard only a
couple of obscure systems were doing it, not gcc.

Also, this doesn't automatically preclude your use case of statically
linking to a large static lib, and you only want to pull in the used
parts. That's perfectly fine. If the unused parts lack side effects,
and the compiler + linker is able to prove it, then it's fine to
remove it under the "as if" rule. I honestly don't know how hard such
analysis is. I know the Microsoft Windows Visual Studios compiler +
linker does this analysis to some degree. (Specifically, I think 2003,
2005, and/or 2008 don't report a missing member function definition if
that class is never used. Another annoying inconvenience when porting
to unix-like systems.)

Finally, is this use case all that common? How many large static libs
are there out there which you would static link to it and only want to
use a small subset of its functionality? I'd /guess/ that either you
control the lib and can split it up, or it's some third party lib
think that's not under your control, and that sounds like it would be
a shared lib.

Personally I'd tend to statically link to third party libraries where
possible unless there was a compelling reason not to -- I'm not from the
school of thought that thinks inflicting DLL hell on users is desirable
Many of the third party libraries I've seen provide static as well as
shared versions, and there are often good alternatives to those that don't.

Cheers,
Stu

 

[Joshua Maurice]

Personally I'd tend to statically link to third party libraries where
possible unless there was a compelling reason not to -- I'm not from the
school of thought that thinks inflicting DLL hell on users is desirable
Many of the third party libraries I've seen provide static as well as
shared versions, and there are often good alternatives to those that don't.

True. I just have a knee-jerk reaction against allowances in the
standard that change program behavior silently and in a platform
dependent manner.

 

[James Kanze]

That's the general idea of a library - put together a bunch of object
files and have the linker pull out the ones you need. Otherwise every
program would contain a full copy of the entire standard library!

[/QUOTE]

That's the *definition* of a library. (Despite the name, DLL's
are not libraries, but object files.) If you want a specific
object file, you link the object file, or give the linker some
other explicit command to include it.

For static linking yes, but two qualms.

For *libraries* (as opposed to object files like .obj or .dll
under Windows, .o or .so under Unix).

First, for windows / unix-like systems, don't most people link to a
shared lib (aka dll) libc nowadays anyway? (I don't know much about
other systems.)

There are two issues: dynamically linking to the system is
generally necessary if you want to interface to different
versions of the system; on the other hand, you don't want to
dynamically link to anything that might not be present on the
system (isn't bundled). As far as I know, libc (the system
interface and the C runtime) has been bundled with every Unix
ever delivered, and most modern Unix don't even provide it as
a static library. In the case of Windows, I'm less sure, but it
wouldn't surprise me if the system interface and the C runtime
were separate libraries (which makes logical sense), and only
the system interface were systematically bundled.

Second, this is due to what many consider a bug in the standard. There
is an allowance in the standard which says that the namespace scope
initialization of a translation unit may be delayed until first use.
This is a piss-poor attempt at making shared libs aka dlls standard
complying. In practice, it's basically impossible for a compiler +
linker to actually delay the initialization a finite amount of time
without programmer intervention, such as during the creation of a
shared lib.

I'm not sure what you mean by the "this is due". I agree that
the permission for delayed initialization is a bug in the
standard---I've argued strongly against it more than once. But
I don't see how it affects anything here; no compiler uses this
freedom.

What is certain is that the compiler will only initialize
objects in modules which are part of the program. How you
specify which modules are part of the program is implementation
defined, but it would be a very strange (and highly undesirable)
implementation which included a module just because it happened
to appear in a library.

Unfortunately, some people got the idea that this is a cool allowance
for an optimization like the OP suggests. I'm not actually sure if
that's what the OP is hitting, but it may be. Last I heard only a
couple of obscure systems were doing it, not gcc.

I think HP experimented with it at one time. But no current
compiler does it.

Also, this doesn't automatically preclude your use case of statically
linking to a large static lib, and you only want to pull in the used
parts. That's perfectly fine. If the unused parts lack side effects,
and the compiler + linker is able to prove it, then it's fine to
remove it under the "as if" rule.

Whether the "unused" parts have side effects or not is
irrelevant. They aren't part of the program. By the definition
of "library".

I honestly don't know how hard such
analysis is. I know the Microsoft Windows Visual Studios compiler +
linker does this analysis to some degree. (Specifically, I think 2003,
2005, and/or 2008 don't report a missing member function definition if
that class is never used. Another annoying inconvenience when porting
to unix-like systems.)

Unix systems generally behave in the same manner. The C++
standard requires it: a function that isn't used need not be
defined.

Finally, is this use case all that common? How many large static libs
are there out there which you would static link to it and only want to
use a small subset of its functionality?

Most of them, I'd guess. It's pretty rare to use everything in
a library.

 

[Stuart Golodetz]

True. I just have a knee-jerk reaction against allowances in the
standard that change program behavior silently and in a platform
dependent manner.

Indeed -- I entirely agree with you on that score

 

[Joshua Maurice]

That's the *definition* of a library.  (Despite the name, DLL's
are not libraries, but object files.)  If you want a specific
object file, you link the object file, or give the linker some
other explicit command to include it.


For *libraries* (as opposed to object files like .obj or .dll
under Windows, .o or .so under Unix).


There are two issues: dynamically linking to the system is
generally necessary if you want to interface to different
versions of the system; on the other hand, you don't want to
dynamically link to anything that might not be present on the
system (isn't bundled).  As far as I know, libc (the system
interface and the C runtime) has been bundled with every Unix
ever delivered, and most modern Unix don't even provide it as
a static library.  In the case of Windows, I'm less sure, but it
wouldn't surprise me if the system interface and the C runtime
were separate libraries (which makes logical sense), and only
the system interface were systematically bundled.


I'm not sure what you mean by the "this is due".  I agree that
the permission for delayed initialization is a bug in the
standard---I've argued strongly against it more than once.  But
I don't see how it affects anything here; no compiler uses this
freedom.

What is certain is that the compiler will only initialize
objects in modules which are part of the program.  How you
specify which modules are part of the program is implementation
defined, but it would be a very strange (and highly undesirable)
implementation which included a module just because it happened
to appear in a library.


I think HP experimented with it at one time.  But no current
compiler does it.


Whether the "unused" parts have side effects or not is
irrelevant.  They aren't part of the program.  By the definition
of "library".


Unix systems generally behave in the same manner.  The C++
standard requires it: a function that isn't used need not be
defined.


Most of them, I'd guess.  It's pretty rare to use everything in
a library.

As I said else-thread, I just have a knee-jerk reaction to allowances
in the standard which allow the implementation to silently change the
behavior of the program in a platform dependent manner, such as
optimizing away unused translation units at whim, optimizing away
infinite loops, and so on.

If the term "static library" is defined in such a way, then that seems
like a perfectly reasonable way to go about things. I have no problem
with that, well besides less than ideal documentation and education on
the topic.

However, in the commercial code of my company, I was concerned enough
about this allowance to remove unused translation units that I moved
some namespace scope initialization code from one file to another file
which I knew was referenced externally. It's a really bad thing in the
standard which should be removed. Hopefully it will be if and when we
ever get dll aka shared "object" support.

 

[James Kanze]

On Dec 9, 4:17 am, James Kanze <[email protected]> wrote:

[...]

As I said else-thread, I just have a knee-jerk reaction to allowances
in the standard which allow the implementation to silently change the
behavior of the program in a platform dependent manner, such as
optimizing away unused translation units at whim, optimizing away
infinite loops, and so on.

I agree with you there (and you should add silently suppressing
a copy), but that's not what's happening in this case. The
issue here is that you're not including the translation unit in
the final program.

If the term "static library" is defined in such a way, then that seems
like a perfectly reasonable way to go about things. I have no problem
with that, well besides less than ideal documentation and education on
the topic.

The term "library" (static or otherwise) has always been defined
like that, ever since I can remember (and that goes back some).
As Pete said, when you go to a library, you don't have to take
out all of the books in it just to read one.

With regards to documentation and education: one could argue
that Microsoft has mislead people calling their dynamically
linked objects DLLs (where the last L stands for
library)---under Unix, they're .so (where the last o stands for
object). But the distinction really only becomes important when
static initializers are involved, and I suspect the name
predates that.

However, in the commercial code of my company, I was concerned
enough about this allowance to remove unused translation units
that I moved some namespace scope initialization code from one
file to another file which I knew was referenced externally.
It's a really bad thing in the standard which should be
removed. Hopefully it will be if and when we ever get dll aka
shared "object" support.

I'm not sure what you mean by "bad thing" in the standard. The
standard is simply conforming to the universal existing
practice, and in fact, leaves the entire issue "implementation
defined". (Everything concerning how you invoke the compiler is
"implementation defined". There is an implication, however, in
the statement "Library components are linked to satisfy external
references to functions and objects not defined in the current
translation." (§2.1/9)

The support in the standard for dynamic linking is on hold for
the moment; hopefully, we'll get it sometime (in the form of
modules). But it certainly won't change anything fundamental
here: how you specify which modules are part of your program
will remain implementation defined, and when in a library,
modules (or components) will only be linked "to satify external
references to functions and objects not defined in the current
translation".

The obvious solution is to link the object files, rather than
putting them into libraries.

 

[Joshua Maurice]

On Dec 9, 4:17 am, James Kanze <[email protected]> wrote:

    [...]

As I said else-thread, I just have a knee-jerk reaction to allowances
in the standard which allow the implementation to silently change the
behavior of the program in a platform dependent manner, such as
optimizing away unused translation units at whim, optimizing away
infinite loops, and so on.

I agree with you there (and you should add silently suppressing
a copy), but that's not what's happening in this case.  The
issue here is that you're not including the translation unit in
the final program.

I'm bias-ed. I recognize that silently suppressing a copy does change
program semantics, but it's been there since the beginning for me, and
it's a much more sensible allowance than the other named allowances.
(I'll avoid the full rant which I recently made in both comp.std.c++
and comp.lang.c++.moderated. The rant did specifically mention copy
constructor elision as a direct parallel, although one I support.)

The term "library" (static or otherwise) has always been defined
like that, ever since I can remember (and that goes back some).
As Pete said, when you go to a library, you don't have to take
out all of the books in it just to read one.

With regards to documentation and education: one could argue
that Microsoft has mislead people calling their dynamically
linked objects DLLs (where the last L stands for
library)---under Unix, they're .so (where the last o stands for
object).  But the distinction really only becomes important when
static initializers are involved, and I suspect the name
predates that.


I'm not sure what you mean by "bad thing" in the standard.  The
standard is simply conforming to the universal existing
practice, and in fact, leaves the entire issue "implementation
defined".  (Everything concerning how you invoke the compiler is
"implementation defined".  There is an implication, however, in
the statement "Library components are linked to satisfy external
references to functions and objects not defined in the current
translation." (§2.1/9)

Sorry. I wasn't clear enough. The whole time I was referring to C++03
standard - "3.6.2 Initialization of non-local objects / 3". AFAIK: In
practice, it's basically impossible for a compiler, linker, and
implementation to delay the dynamic initialization of some namespace
scope objects past main but before an actual first use of the
translation unit - barring specific user guidance such as the creation
of a shared object. It's a token gesture at making shared objects
standard conformant. However, someone saw this as a cool allowance to
optimize away "unreferenced" but still linked translation units. That
is the bug in the standard with which I was annoyed. Because I was
afraid of being bitten by that, I moved a namespace scope object whose
entire job was to initialize a callback from its own cpp file to an
already existing cpp file to ensure that some broken-but-conforming
linker wouldn't optimize it away.

 

[James Kanze]

On Dec 9, 4:17 am, James Kanze <[email protected]> wrote:

[...]

As I said else-thread, I just have a knee-jerk reaction to allowances
in the standard which allow the implementation to silently change the
behavior of the program in a platform dependent manner, such as
optimizing away unused translation units at whim, optimizing away
infinite loops, and so on.

I agree with you there (and you should add silently suppressing
a copy), but that's not what's happening in this case. The
issue here is that you're not including the translation unit in
the final program.

I'm bias-ed. I recognize that silently suppressing a copy does change
program semantics, but it's been there since the beginning for me,

The way libraries work is even older.

and
it's a much more sensible allowance than the other named allowances.
(I'll avoid the full rant which I recently made in both comp.std.c++
and comp.lang.c++.moderated. The rant did specifically mention copy
constructor elision as a direct parallel, although one I support.)

But I more or less agree here. The suppression of copies
doesn't cause me any worries, except on theoretical grounds.
The other suppressions do.

[...]

Sorry. I wasn't clear enough. The whole time I was referring to C++03
standard - "3.6.2 Initialization of non-local objects / 3". AFAIK: In
practice, it's basically impossible for a compiler, linker, and
implementation to delay the dynamic initialization of some namespace
scope objects past main but before an actual first use of the
translation unit - barring specific user guidance such as the creation
of a shared object. It's a token gesture at making shared objects
standard conformant.

It's a misguided gesture at making dynamic linking standard
conformant. To be conformant, an implementation must behave "as
if" everything was linked before any user written code is
executed. This automatically excludes any explicit dynamic
linking (calls to dlopen or LoadLibrary). And if implicit
dynamic linking loads all of the dependencies on program start
up, it more or less fulfills the requirement to behave "as if".

However, someone saw this as a cool allowance to
optimize away "unreferenced" but still linked translation units.

Who? I know several people who have expressed the opinion that
the current wording does allow this, but I know of no compiler
which actually exploits it.

That is the bug in the standard with which I was annoyed.
Because I was afraid of being bitten by that, I moved
a namespace scope object whose entire job was to initialize
a callback from its own cpp file to an already existing cpp
file to ensure that some broken-but-conforming linker wouldn't
optimize it away.

I use the technique you're trying to use extensively, and I've
never had any problems, either with static or dynamic
linking---neither on Unix platforms (Solaris or Linux) nor under
Windows. The important thing has always been to ensure that the
object file with the static object is part of the process: this
is done either by linking the object file explicitly, either
into the executable or into a dynamic object which is explicitly
linked.

 

[Joshua Maurice]

    [...]
As I said else-thread, I just have a knee-jerk reaction to allowances
in the standard which allow the implementation to silently change the
behavior of the program in a platform dependent manner, such as
optimizing away unused translation units at whim, optimizing away
infinite loops, and so on.
I agree with you there (and you should add silently suppressing
a copy), but that's not what's happening in this case.  The
issue here is that you're not including the translation unit in
the final program.

I'm bias-ed. I recognize that silently suppressing a copy does change
program semantics, but it's been there since the beginning for me,

The way libraries work is even older.

and
it's a much more sensible allowance than the other named allowances.
(I'll avoid the full rant which I recently made in both comp.std.c++
and comp.lang.c++.moderated. The rant did specifically mention copy
constructor elision as a direct parallel, although one I support.)

But I more or less agree here.  The suppression of copies
doesn't cause me any worries, except on theoretical grounds.
The other suppressions do.

    [...]

Sorry. I wasn't clear enough. The whole time I was referring to C++03
standard - "3.6.2 Initialization of non-local objects / 3". AFAIK: In
practice, it's basically impossible for a compiler, linker, and
implementation to delay the dynamic initialization of some namespace
scope objects past main but before an actual first use of the
translation unit - barring specific user guidance such as the creation
of a shared object. It's a token gesture at making shared objects
standard conformant.

It's a misguided gesture at making dynamic linking standard
conformant.  To be conformant, an implementation must behave "as
if" everything was linked before any user written code is
executed.  This automatically excludes any explicit dynamic
linking (calls to dlopen or LoadLibrary).  And if implicit
dynamic linking loads all of the dependencies on program start
up, it more or less fulfills the requirement to behave "as if".

However, someone saw this as a cool allowance to
optimize away "unreferenced" but still linked translation units.

Who?  I know several people who have expressed the opinion that
the current wording does allow this, but I know of no compiler
which actually exploits it.

I forget. As I said, I thought someone on these newsgroups said some
obscure implementation was doing it.

I use the technique you're trying to use extensively, and I've
never had any problems, either with static or dynamic
linking---neither on Unix platforms (Solaris or Linux) nor under
Windows.  The important thing has always been to ensure that the
object file with the static object is part of the process: this
is done either by linking the object file explicitly, either
into the executable or into a dynamic object which is explicitly
linked.

Indeed. I was very wrong about libraries. I admit fault. I was
ignorant and confusing the issue with what I feel is a very badly
worded, buggy even, "C++03 standard - 3.6.2 Initialization of non-
local objects / 3".

Also, thank you. It's good to know that you haven't encountered an
implementation broken in this regard.