Reducing build times

[Stuart MacMartin]

Looking for tricks for speeding up builds...

We have an application with about 970 of our own classes of various
sizes plus a fair number of routines. Over the past year the
compile/link time has been slowing down on PC (MFC dev studio 6) and on
linux. Link time on PC seems to depend on RAM - a new desktop machine
with 2Gb of RAM compiles and links significantly faster than a three
year old laptop with 1/2 Gb of RAM but same processor speed.

Over the past year or so we've been using more templates and more STL.
We also have more classes, but I don't think the size of the project
fully accounts for the slowdown.

Tricks we use:
- On PC, precompiled header file includes often used things like
stingray include files; on Unix this is bypassed (although I understand
gcc now supports it?)
- We appropriately minimize cascading include files (member variables
are typically pointers, except where judiciously chosen not to be)
- I've started hiding some implementations further by pimpl, especially
dialogs used where the full dialog support classes aren't otherwise
needed anyway

Q: Are there any other tricks?
Q: Can proper use of namespaces help?
Q: Is there an issue with stl or templates with respect to build times?
Q: We haven't really seen the need to split the project up into
libraries. Can this help (and if so, why?)

Stuart

 

[Panjandrum]

Looking for tricks for speeding up builds...

Q: Are there any other tricks?

Forward declaration of classes. You can forward declare all classes
except for data members. Also return value types and types passed by
value to functions.

Q: Can proper use of namespaces help?

Don't see how.

Q: Is there an issue with stl or templates with respect to build times?

Of course. Templates are 'header only'! Templates need to be included,
parsed, and instantiated.

Q: We haven't really seen the need to split the project up into
libraries. Can this help (and if so, why?)

If you can split it. Code that is not designed for modularity tends to
be momolithic and intermingled. BTW, the best book in this case still
is John Lakos 'Large-Scale C++ Software Design'.

 

[Allan Bruce]

Looking for tricks for speeding up builds...

We have an application with about 970 of our own classes of various
sizes plus a fair number of routines. Over the past year the
compile/link time has been slowing down on PC (MFC dev studio 6) and on
linux. Link time on PC seems to depend on RAM - a new desktop machine
with 2Gb of RAM compiles and links significantly faster than a three
year old laptop with 1/2 Gb of RAM but same processor speed.

Over the past year or so we've been using more templates and more STL.
We also have more classes, but I don't think the size of the project
fully accounts for the slowdown.

Tricks we use:
- On PC, precompiled header file includes often used things like
stingray include files; on Unix this is bypassed (although I understand
gcc now supports it?)
- We appropriately minimize cascading include files (member variables
are typically pointers, except where judiciously chosen not to be)
- I've started hiding some implementations further by pimpl, especially
dialogs used where the full dialog support classes aren't otherwise
needed anyway

Q: Are there any other tricks?
Q: Can proper use of namespaces help?
Q: Is there an issue with stl or templates with respect to build times?
Q: We haven't really seen the need to split the project up into
libraries. Can this help (and if so, why?)

Stuart

you can pop a pragma for your compiler to say 'build this class only once'.
This is not very portable though, in VS6 the follwing does it:
#pragma once

Allan

 

[Paul Bilnoski]

you can pop a pragma for your compiler to say 'build this class only once'.
This is not very portable though, in VS6 the follwing does it:
#pragma once

Allan

The '#pragma once' directive is the same as using '#ifdef' include
guards. It doesn't look at classes, just keeps the entire file from
beign sent to the compiler more than once.
--Paul

 

[Stuart MacMartin]

We do equivalent protection within our include files using
#if !defined(FILE_H)
#define FILE_H
...
#endif

Also for "cascaded includes" I meant we don't include other files
in the include files unless we really have to mention that class
by value. We try to use forward declarations of classes
in the include files wherever possible.

Stuart

 

[Manfred]

Looking for tricks for speeding up builds...

We have an application with about 970 of our own classes of various
sizes plus a fair number of routines. Over the past year the
compile/link time has been slowing down on PC (MFC dev studio 6) and on
linux. Link time on PC seems to depend on RAM - a new desktop machine
with 2Gb of RAM compiles and links significantly faster than a three
year old laptop with 1/2 Gb of RAM but same processor speed.

Over the past year or so we've been using more templates and more STL.
We also have more classes, but I don't think the size of the project
fully accounts for the slowdown.

Tricks we use:
- On PC, precompiled header file includes often used things like
stingray include files; on Unix this is bypassed (although I understand
gcc now supports it?)
- We appropriately minimize cascading include files (member variables
are typically pointers, except where judiciously chosen not to be)
- I've started hiding some implementations further by pimpl, especially
dialogs used where the full dialog support classes aren't otherwise
needed anyway

Q: Are there any other tricks?
Q: Can proper use of namespaces help?
Q: Is there an issue with stl or templates with respect to build times?
Q: We haven't really seen the need to split the project up into
libraries. Can this help (and if so, why?)

Stuart

Have you had a look at:

http://ccache.samba.org/

I have only heard of it but never used it myself, but it looks
interesting.

Manfred

 

[James]

Looking for tricks for speeding up builds...

We have an application with about 970 of our own classes of various
sizes plus a fair number of routines. Over the past year the
compile/link time has been slowing down on PC (MFC dev studio 6) and on
linux. Link time on PC seems to depend on RAM - a new desktop machine
with 2Gb of RAM compiles and links significantly faster than a three
year old laptop with 1/2 Gb of RAM but same processor speed.

Over the past year or so we've been using more templates and more STL.
We also have more classes, but I don't think the size of the project
fully accounts for the slowdown.

Tricks we use:
- On PC, precompiled header file includes often used things like
stingray include files; on Unix this is bypassed (although I understand
gcc now supports it?)
- We appropriately minimize cascading include files (member variables
are typically pointers, except where judiciously chosen not to be)
- I've started hiding some implementations further by pimpl, especially
dialogs used where the full dialog support classes aren't otherwise
needed anyway

Q: Are there any other tricks?
Q: Can proper use of namespaces help?
Q: Is there an issue with stl or templates with respect to build times?
Q: We haven't really seen the need to split the project up into
libraries. Can this help (and if so, why?)

Here's the problem: every time you include a template header
(including any STL header lke <string> or <vector>), the compiler must
generate code for it. This is fundamentally different from including a
header to an external library you link to, since in that case the
library is compiled separately. So essentially if you have 10 small
files that #include <vector> and <string> and instantiate
vector<string>, you are compiling your "string" and "vector" libraries
10 times over, completely redundantly if you are using the same
template arguments each time. The linker then has to load all the
object files you compiled and eliminate the duplicate template code,
which can cause it to use a lot of extra memory. The best solution
I've found to this problem is to compile many source files together in
1 c++ file. If you have 10 source files named "src1.cpp" to
"src10.cpp", generate a file called "src_all.cpp" that consists of

#include "src1.cpp"
#include "src2.cpp"
....
#include "src10.cpp"

In my experience this single file will compile almost as fast as any 1
individual file if they all include all the same template headers. I
disabled precompiled headers and dont use them.

 

[Stuart MacMartin]

When I build on a new machine its a lot faster than when I build on an
old one. Same processor speed, but the new one has 2Gb RAM and the old
1/2 Gb. So maybe this has something to do with paging.

Are there tricks to reduce the amount of memory the compiler/linker
needs (specifically with VS6, but a general question)

Stuart

 

[Phlip]

We have an application with about 970 of our own classes of various
sizes plus a fair number of routines.

Do you have one class per .h file? There's usually no need to do that.

Does every .cpp file wind up including every .h file?

MFC dev studio 6

If the Class Wizard generated all that, it will follow a stereotypically bad
pattern. That will make fixes easy to roll out.

Tricks we use:
- On PC, precompiled header file includes often used things like
stingray include files; on Unix this is bypassed (although I understand
gcc now supports it?)

The "StdAfx.h" system (on PC, folks) is brute-force, and should contain
every .h that doesn't change frequently.

Q: Are there any other tricks?

Research the Dependency Inversion Principle. C++ makes big projects
reachable by allowing code that logically depends on interfaces to not
physically depend on all the types used in and behind the interface. This is
the logical design technique that Pimpl is an emergency physical design hack
to simulate.

Then read (but don't worship) /Large Scale C++ Software Design/ by John
Lakos.

 

[Stuart MacMartin]

Do you have one class per .h file? There's usually no need to do that.

Your experience obviously differs from mine.

We did some experiments with several compilers and found that splitting
the include files up did not significantly increase the compile time
when all files depend on the same files. However, reducing the
dependencies and noise within a compile unit does affect build times.
Furthermore, the more classes in an include file the more your
dependencies are cascaded and the more collisions you have between
developers maintaining source. While I agree that several leaf level
classes in the implementation of a subsystem can often be in one
include file, as a general rule I find it best to limit what goes into
any include file, and my experience leads me to the opposite conclusion
from you.

Does every .cpp file wind up including every .h file?

No. We do have a few more cascaded dependencies than I'd like, but in
general this is under control.

If the Class Wizard generated all that, it will follow a stereotypically bad pattern.

Agreed. I personally don't like the code class wizard generates.

The "StdAfx.h" system (on PC, folks) is brute-force, and should contain every .h that doesn't change frequently.

True to a point. Eventually adding include files that rarely change
slows down builds. I have a feeling this is where our problem lies:
too much in stdafx.h. I think I have to split up our project into
separate libraries so that certain big subsystems needed by only 1/3 of
our files are only included for those 1/3 and not the rest.

Research the Dependency Inversion Principle

Never heard of this, at least by name. I'll look into it. I'll also
take a look at Lakos' book, but I'm not expecting to find much I don't
already know.

Stuart

 

[msalters]

James schreef:

Here's the problem: every time you include a template header
(including any STL header lke <string> or <vector>), the compiler must
generate code for it.

How? When you include <string> you get std::basic_string< T > but
without

This is fundamentally different from including a
header to an external library you link to, since in that case the
library is compiled separately.

So essentially if you have 10 small
files that #include <vector> and <string> and instantiate
vector<string>, you are compiling your "string" and "vector" libraries
10 times over, completely redundantly if you are using the same
template arguments each time. The linker then has to load all the
object files you compiled and eliminate the duplicate template code,
which can cause it to use a lot of extra memory. The best solution
I've found to this problem is to compile many source files together in
1 c++ file.

Too bad that it you change 1 cpp, you're really recompiling all cpp
files.
With PCH you share the parsing of <string> and <vector> and best of all
<windows.h>, but you don't have to recompile file1.cpp if file2.cpp
changes.
Still leaves the linker problem, but that can be solved to. I've not
had
problems there - my current project has gone from 12 minutes to 3 using
PCH (with two PCHs for different parts of the program), yet link time
is
still seconds.

The best argument to your boss should still be comparing programmer
prices
with memory prices. So what if it takes 2Gb? That's cheap enough.

Regards,
Michiel Salters

 

[Billy N. Patton]

Looking for tricks for speeding up builds...

We have an application with about 970 of our own classes of various
sizes plus a fair number of routines. Over the past year the
compile/link time has been slowing down on PC (MFC dev studio 6) and on
linux. Link time on PC seems to depend on RAM - a new desktop machine
with 2Gb of RAM compiles and links significantly faster than a three
year old laptop with 1/2 Gb of RAM but same processor speed.

Over the past year or so we've been using more templates and more STL.
We also have more classes, but I don't think the size of the project
fully accounts for the slowdown.

Tricks we use:
- On PC, precompiled header file includes often used things like
stingray include files; on Unix this is bypassed (although I understand
gcc now supports it?)
- We appropriately minimize cascading include files (member variables
are typically pointers, except where judiciously chosen not to be)
- I've started hiding some implementations further by pimpl, especially
dialogs used where the full dialog support classes aren't otherwise
needed anyway

Q: Are there any other tricks?
Q: Can proper use of namespaces help?
Q: Is there an issue with stl or templates with respect to build times?
Q: We haven't really seen the need to split the project up into
libraries. Can this help (and if so, why?)

Stuart

Reduce file io;
ex:
my_stuff.h :
#ifndef MY_STUFF_H
#define MY_STUFF_H

class my_stuff
{
};

#endif

my_stuff.cxx :
#ifndef _CPP_IOSTREAM
#include <iostream>
#endif
#ifndef _CPP_STRING
#include <string>
#endif
#ifndef _CPP_VECTOR
#include <vector>
#endif
#ifndef MY_STUFF_H
#include "my_stuff.h"
#endif

every place you do an include, either in a .h or a .cxx file do the
ifndef check. It keep the compiler from haveing to open up every file
to check for the ifndef.

Look at the top of the system include file. Most all have #ifndef
_SOMETHING. True this keep the file from loading but the compiler still
had to opent it and begin parsing.

--
___ _ ____ ___ __ __
/ _ )(_) / /_ __ / _ \___ _/ /_/ /____ ___
/ _ / / / / // / / ___/ _ `/ __/ __/ _ \/ _ \
/____/_/_/_/\_, / /_/ \_,_/\__/\__/\___/_//_/
/___/
Texas Instruments ASIC Circuit Design Methodology Group
Dallas, Texas, 214-480-4455, (e-mail address removed)

 

[Ben Pope]

every place you do an include, either in a .h or a .cxx file do the
ifndef check. It keep the compiler from haveing to open up every file
to check for the ifndef.

Look at the top of the system include file. Most all have #ifndef
_SOMETHING. True this keep the file from loading but the compiler still
had to opent it and begin parsing.

Don't most commercial compilers spot that pattern, and if everything useful is guarded, remember that, and not open the file?

Ben