Merits and uses of static vs. dynamic libraries

[Qu0ll]

When should one create a static library as opposed to a dynamic library and
what are the relative merits of each type?

I can see immediately that having one (perhaps large) static library would
be advantageous in the sense that you needn't worry about keeping track of
multiple dependencies so it could be appropriate if you are developing a
class library.

What are the other pros and cons?

--
And loving it,

-Qu0ll (Rare, not extinct)
_________________________________________________
(e-mail address removed)
[Replace the "SixFour" with numbers to email me]

 

[Ian Collins]

When should one create a static library as opposed to a dynamic library
and what are the relative merits of each type?

The answer depends on your platform.

 

[SG]

Dynamic libraries are a later innovation, the biggest selling point being
the byte-saving both on disk and in memory as shared libraries can be
potentially shared by many applications at the same time. OTOH, it
becomes much more difficult to keep track the versions and ensure that
each application uses the right shared libraries, these problems are
often regarded as "the DLL hell" in the Windows world. To avoid this,
each application often uses its own set of semi-standard DLL-s packaged
together with the application, thus negating most of any memory
comsumption effect which it might have had.

Another selling point would be IMHO:
You can update a shared library (i.e. bug fixes -- assuming binary
compatibility) without the need to recompile every application that
uses it.

Cheers!
SG

 

[Bo Persson]

Another selling point would be IMHO:
You can update a shared library (i.e. bug fixes -- assuming binary
compatibility) without the need to recompile every application that
uses it.

And this is where "the DLL hell" starts.

As some of the applications are only tested - and working - with the
older versions of the library, how do we know if they are also working
with the supposedly fixed version?

On the other hand, if the applications are statically linked to the
library, you will have to get a separate update for each of them, for
the same bug.

A catch-22, I guess.


Bo Persson

 

[James Kanze]

When should one create a static library as opposed to a
dynamic library and what are the relative merits of each type?

If you have to ask the question, you probably shouldn't create a
dynamic library. Generally, use static libraries when you can,
dynamic libraries when you have to.

I can see immediately that having one (perhaps large) static
library would be advantageous in the sense that you needn't
worry about keeping track of multiple dependencies so it could
be appropriate if you are developing a class library.

What are the other pros and cons?

Dynamic libraries facility linking in the wrong version of
something,. They also make deployment more complex (unless
they're bundled with the system, like libc under Unix).

Dynamic libraries allow choosing the actual library at runtime.
It's possible to link in dynamic libraries which weren't even
written when your code was deployed. Plugins are good examples
of where dynamic libraries are essential.

 

[James Kanze]

"Qu0ll" <[email protected]> kirjutas:

Not really C++ specific, and shared libraries are not touched
by the current C++ standard, but anyway, here it goes...

Dynamic libraries are a later innovation,

They predate C++ by a large margin. (I believe Multix supported
dynamic linking, and other systems of that period certainly
did.)

the biggest selling point being the byte-saving both on disk
and in memory as shared libraries can be potentially shared by
many applications at the same time.

That was a major motivation for the early GUI systems (Sun OS,
etc.), where the GUI library often represented 90% of the size
of an executable. It's rarely relevant today.

It's especially irrelevant for libraries you (the average
programmer) write, which will only be used in a couple of
applications (which will typically require different versions of
the library anyway).

OTOH, it becomes much more difficult to keep track the
versions and ensure that each application uses the right
shared libraries, these problems are often regarded as "the
DLL hell" in the Windows world. To avoid this, each
application often uses its own set of semi-standard DLL-s
packaged together with the application, thus negating most of
any memory comsumption effect which it might have had.

To summarize, shared libraries should be used only in the
following situations IMO:

* standard implementation-provided libraries with fixed or
at least back-compatible binary interface, like kernel32.dll
or libm.so.
* implementing optionally loadable modules extending the
application functionality at run-time.

Windows won't pick up symbols in the root when linking a shared
library. (At least, that's my impression.) Which means that
any functionality which should be shared between the root and a
dynamically loaded object, or between different dynamically
loaded objects, must itself be in a DLL. (Maybe there are
options around this. And I think that's really what's behind
some of the examples you gave that I cut.)

 

[James Kanze]

Another selling point would be IMHO:
You can update a shared library (i.e. bug fixes -- assuming
binary compatibility) without the need to recompile every
application that uses it.

That's probably the biggest argument against it. You end up
with programs that have never been tested in the actual version
the client is using. It's a valid argument for things like the
system API (which is usually a bundled dynamic object), or the
data base interface (your code can access several different
versions of Oracle, depending on what the client has installed),
but very few people are working at that level, where it makes
sense.

 

[Qu0ll]

When should one create a static library as opposed to a dynamic library
and what are the relative merits of each type?

I can see immediately that having one (perhaps large) static library would
be advantageous in the sense that you needn't worry about keeping track of
multiple dependencies so it could be appropriate if you are developing a
class library.

What are the other pros and cons?

Thanks to all those who replied. It looks like static is the way to go in
most circumstances.

--
And loving it,

-Qu0ll (Rare, not extinct)
_________________________________________________
(e-mail address removed)
[Replace the "SixFour" with numbers to email me]

 

[SG]

I agree with Bo Persson and James Kanze here

Yes, it's more of a catch-22. But I woulnd't consider it to be that
bad (may really depend on the OS and kind of library).

ability to update single
DLL-s in the customer installation creates more problems than solves. To
get such a thing working one needs quite complicated automatic updater,
which would synchronize the customer installation to the last thoroughly
tested combination of DLL-s.

I was more thinking along the lines of *nix, /usr/lib/ and open source
development. If a buffer overlow bug in zlib is found & fixed and the
new version stays binary-compatible I don't want to have to download
and/or recompile Gimp (or any other program that makes use of this
library).


Cheers!
SG

 

[peter koch]

Yes, it's more of a catch-22.  But I woulnd't consider it to be that
bad (may really depend on the OS and kind of library).


I was more thinking along the lines of *nix, /usr/lib/ and open source
development.  If a buffer overflow bug in zlib is found & fixed and the
new version stays binary-compatible I don't want to have to download
and/or recompile Gimp (or any other program that makes use of this
library).

I agree with you - it really depends on the type of application you
write and the type of code in the library. For stuff like zlib and the
OS-near code, a DLL is the way to go.
Other cases where dynamic libraries are warranted is where your
application consists of a group of processes that share common code,
and the cases where your application has an abstract interface that is
more easily supplied (and updated) as a group of DLLs.
Other groups should prefer dynamic libraries.

/Peter

 

[James Kanze]

I was more thinking along the lines of *nix, /usr/lib/ and
open source development. If a buffer overlow bug in zlib is
found & fixed and the new version stays binary-compatible I
don't want to have to download and/or recompile Gimp (or any
other program that makes use of this library).

In theory, this should be true. In practice, I've found that at
least under Linux, an awful lot of the libraries do change the
binary interface when they change versions, so any given
executable only works with one version, and when you update one
of the libraries, you generally have to update all of the
programs which use it as well.

 

[James Kanze]

On Apr 13, 2:37 pm, peter koch <[email protected]> wrote:

[...]

One additional caveat that I've found with shared libraries
and C++ in particular is that it's easy to make a change that
requires recompilation of all clients of the library but not
to be aware of it.

Anytime you change anything in a header file, all client code
must be recompiled. Formally, if all you change are comments,
you're OK, and there are a few other things you can change, but
you can't change a single token in a class definition. And in
practice, the simple rule is: header file changed=>all client
code must be recompiled. (Which is the way makefiles work,
anyway.)

This is why in large projects, developers can't normally
check-out header files for edition.

I was in a medium sized group (about 30 people) that used
shared libraries extensively, and we had many such problems.
Adding a new virtual method, for instance, may not seem like a
breaking change

Changing anything in a class definition is a breaking change.
Maybe not always in practice, but according to the standard.
You can't add (or remove) a member, even private, without having
to recompile all of the client code. (Why do you think the
compilation firewall idiom is so widely used?)

 

[Ian Collins]

In theory, this should be true. In practice, I've found that at
least under Linux, an awful lot of the libraries do change the
binary interface when they change versions, so any given
executable only works with one version, and when you update one
of the libraries, you generally have to update all of the
programs which use it as well.

That assumes you upgrade the library to a new version, rather than patch
the version you built against. Shifting interfaces is one of the joys
of opensource.

 

[Ian Collins]

That's probably the biggest argument against it. You end up
with programs that have never been tested in the actual version
the client is using. It's a valid argument for things like the
system API (which is usually a bundled dynamic object), or the
data base interface (your code can access several different
versions of Oracle, depending on what the client has installed),
but very few people are working at that level, where it makes
sense.

How about a patch for a bug in the platform's C++ runtime or standard
library?

 

[Ian Collins]

In theory, this should be true. In practice, I've found that at
least under Linux, an awful lot of the libraries do change the
binary interface when they change versions, so any given
executable only works with one version, and when you update one
of the libraries, you generally have to update all of the
programs which use it as well.

That assumes you upgrade the library to a new version, rather than patch
the version you built against. Shifting interfaces is one of the joys
of opensource.

 

[Ian Collins]

In theory, this should be true. In practice, I've found that at
least under Linux, an awful lot of the libraries do change the
binary interface when they change versions, so any given
executable only works with one version, and when you update one
of the libraries, you generally have to update all of the
programs which use it as well.

That assumes you upgrade the library to a new version, rather than patch
the version you built against. Shifting interfaces is one of the joys
of opensource.

 

[Andreas Dehmel]

On Mon, 13 Apr 2009 13:10:40 -0700 (PDT), Tony Strauss wrote:

[...]

One additional caveat that I've found with shared libraries and C++ in
particular is that it's easy to make a change that requires
recompilation of all clients of the library but not to be aware of
it. I was in a medium sized group (about 30 people) that used shared
libraries extensively, and we had many such problems. Adding a new
virtual method, for instance, may not seem like a breaking change
intuitively because the existing interfaces are unchanged, but it is
(because the layout of the virtual table may change).

It's always good to know what you're doing ;-).

Our experience is that shared libs have huge advantages when developing large
projects. In these you tend to have a reasonable association of developers
with modules, so as long as people don't change the _exported_ library
interface all they need to do is build "their" shared lib(s) and not bother
having to link a huge program with every change.

We ran into massive link time problems with our app and a previous version
of MSVC (several minutes, even with a "hot" file cache; don't even ask about
the size of the linker process...). Problems which could be alleviated
considerably by introducing DLLs. The MSVC version we're currently using
seems to cope better with the linking stage (probably weak symbol handling
for the most part), but the time savings of just linking a DLL rather than
a 40MB behemoth are and always will be massive. I don't know how well this
translates to Unix-style shared libs, though, since they're closer to
static libs (whereas DLLs are closer to executables). Still worth a try
IMHO.



Andreas

 

[James Kanze]

On Apr 13, 10:10 pm, Tony Strauss

On Apr 13, 2:37 pm, peter koch
<[email protected]> wrote:

[...]

One additional caveat that I've found with shared
libraries and C++ in particular is that it's easy to make
a change that requires recompilation of all clients of the
library but not to be aware of it.

Anytime you change anything in a header file, all client
code must be recompiled. Formally, if all you change are
comments, you're OK, and there are a few other things you
can change, but you can't change a single token in a class
definition. And in practice, the simple rule is: header
file changed=>all client code must be recompiled. (Which is
the way makefiles work, anyway.)

You make it sound so simple , but at the particular place
where I was working, the rules did not end up being as clear
as you're laying out (naturally).

The rules I just explained have nothing to do with where you
might work. They're part of the language. Something called the
one definition rule, section 3.2 of the standard.

The only simplification I made was the last sentence: any change
in the header requires all client code to be recompiled. In
practice, that's what you get anyway, at least with build
systems (make) that I'm familiar with. The only way you can
avoid it is to play tricks with the time stamps, and that's a
sure road to problems.

The system was large (on the order of 1,000,000 lines of code)
and consisted of many programs (about 150), many instances of
which were running on a machine at any given moment. The
build system was recursive, and Makefiles in a large recursive
build system often will *NOT* recompile all code when a header
file changes.

Then there's something wrong with them. All of the systems I'm
familiar with have some means of automatically maintaining the
dependencies, and you're definitely better off using them.

For instance, if you have this file layout and a recursive
build system (which is very evil but, alas, very common):

There's nothing wrong with recursive build systems---they
correspond to the project organization. But like everything
else, they have to be done right.

Makefile


a common pattern is to make a change to the library, build the
library, and then separately build the program.

That's not recursive build. But that shouldn't be a problem.
Obviously, the changes in the library are checked in at the same
time the changes in the headers, and become visible
simultaneously (atomically). Otherwise, of course, nothing
works.

Rebuilding the library does not cause the program to be
rebuilt.

Should it? The important thing is that the program has a
consistent view of the library (including its headers). That
view may not correspond to the latest state, but that shouldn't
matter (unless you're dealing with shared memory, or some such).

Imagine now this build system scaled to the large system that
I described; it's never going to be clear what needs to be
rebuilt when developing and a top level build probably takes
too long to do for each change when testing. While everyone
knew that doing something like changing a header file required
recompiling all dependencies, it was difficult to do in
practice because of the size of the system.

Test deployment also created issues. Imagine that six
programs rely on a library, of which four always are running.
If a developer is making changes to the library for one of the
programs and moves a new version of that program and of the
shared library to a shared test environment (but does not move
new versions of the other programs), then the other programs
will break if they are bounced at some later time (perhaps by
a different developer). Had we been using static libraries,
the existing programs on the machine would not have been
broken by moving in the new program.

You don't use dynamic linking in such cases.

Using C with shared libraries is a little easier, because you
can add functions to a module and preserve compatibility
(though of course you cannot change the size or layout of a
structure safely).

Nor the signature of a function. Nor in many cases the value of
a constant.

 

[joshuamaurice]

My brain must have been asleep when I wrote my prior post, because
both recursive and non-recursive build systems have issues with shared
libraries.  In retrospect, I'm not sure why I got on my 'recursive
build systems' are evil rant and confused the point that I was trying
to make.  To illustrate the issues that shared libraries can introduce
with any build system, consider:

Makefile

library1/Makefile
library1/library1.h        => library1.so
library1/library1.cpp

program1/Makefile
program1/program1.cpp        => program1 (which depends on
library1.so)

program2/Makefile
program2/program2.cpp        => program2 (which depends on
library1.so)

Leaving aside SCM and just talking about a local working copy of the
source tree, suppose a developer:
1.) Changes library1.
2.) Rebuilds library1 (cd library1 && make)
3.) Rebuilds program1 (cd program1 && make)

Shared libraries introduce the complication that changing and
rebuilding library1 invalidates any programs that depend on library1
(program2 may no longer work correctly).  You're right that this is
not a flaw in the build system, but it does put a burden on the
programmer to remember to rebuild program2 before moving it.  If the
system in question had been using static libraries, this rebuild of
program2 would not be necessary (assuming that the functional change
to library1 was not required in program2).  Since, however, the
example system is using shared libraries, the programmer is
responsible for remembering to rebuild any programs that depends on
library1 before he moves library1.  At my former workplace, this
proved to be a drag on development efficiency (there never were any
problems in production because the system as a whole was built and
moved to production).

Couldn't you just specify for all things which depend on shared
libraries an actual dependency in make? If a shared library changes,
then all 'exe's and shared libraries which depend on it would be
rebuilt. Quite easy to automate this with make.

 

[James Kanze]

On Apr 14, 8:12 pm, Tony Strauss <[email protected]>
wrote:

On Apr 13, 10:10 pm, Tony Strauss
On Apr 13, 2:37 pm, peter koch
[...]
One additional caveat that I've found with shared
libraries and C++ in particular is that it's easy to make
a change that requires recompilation of all clients of the
library but not to be aware of it.
Anytime you change anything in a header file, all client
code must be recompiled. Formally, if all you change are
comments, you're OK, and there are a few other things you
can change, but you can't change a single token in a class
definition. And in practice, the simple rule is: header
file changed=>all client code must be recompiled. (Which is
the way makefiles work, anyway.)
You make it sound so simple , but at the particular place
where I was working, the rules did not end up being as clear
as you're laying out (naturally).

The rules I just explained have nothing to do with where you
might work. They're part of the language. Something called the
one definition rule, section 3.2 of the standard.

I was talking about the feasibility of ensuring that all
client code was recompiled when a header changed, not about
the correctness of doing so. The feasibility does of course
depend on various 'local' factors.

It has to be feasible, since it is necessary. If the local
factors don't make if feasible, then you can't use C++. The
standard is quite clear about this.

[...]

This isn't the forum to discuss this, but I have to disagree.
I think that recursive builds are inherently flawed (although,
of course, they can be perfectly functional). You already may
well have read this and come to your own conclusions, but I
tend to agree with much of what Peter Miller says in his
classic paper on non-recursive build
systems:http://aegis.sourceforge.net/auug97.pdf

I've read it. The author doesn't know anything about software
engineering; recursive make doesn't work well if the project is
poorly engineered. But then, nor does anything else. (Read the
list of problems in section 2. They all start from a
fundamental assumption that the system wasn't designed properly.
That in fact, it wasn't designed at all---I've worked in a lot
of different companies, at different maturity levels, but even
the worse never had these problems.)

My brain must have been asleep when I wrote my prior post,
because both recursive and non-recursive build systems have
issues with shared libraries.

Yes. The problem is shared libraries (or more correctly,
dynamicly linked objects---they're not really libraries, and
they don't have to be shared), not recursive make. The answer
to this problem is to use static linking. Dynamic linking
increases the management overhead---basically, the "header
files" (the interface specification) of a dynamic library must
be fixed at the start, and anytime it's changed (which should be
very, very rarely, if ever), *everything* must be recompiled.
Or you design some sort of version management into the system.
(I'm not sure how this works on other systems, but Unix does
support this somewhat; if you link against libxxx.so.4.3, that's
the dynamic library it will load. Even if there is a
libxxx.so.4.4 available.)

In retrospect, I'm not sure why I got on my 'recursive build
systems' are evil rant and confused the point that I was
trying to make. To illustrate the issues that shared
libraries can introduce with any build system, consider:

library1/Makefile
library1/library1.h => library1.so
library1/library1.cpp

program1/Makefile
program1/program1.cpp => program1 (which depends on
library1.so)

program2/Makefile
program2/program2.cpp => program2 (which depends on
library1.so)

Leaving aside SCM and just talking about a local working copy
of the source tree,

There shouldn't be one. A good version management system acts
as a file server, presenting different views to each programmer.
But it doesn't matter.

suppose a developer:
1.) Changes library1.
2.) Rebuilds library1 (cd library1 && make)
3.) Rebuilds program1 (cd program1 && make)

Shared libraries introduce the complication that changing and
rebuilding library1 invalidates any programs that depend on library1
(program2 may no longer work correctly).

If the changes in the library don't affect the header files,
there should be no problem. If they do, then the version number
changes; programs linked against the old version will continue
to load it (at least under Unix).

You're right that this is not a flaw in the build system, but
it does put a burden on the programmer to remember to rebuild
program2 before moving it. If the system in question had been
using static libraries, this rebuild of program2 would not be
necessary (assuming that the functional change to library1 was
not required in program2).

If you've versioned the shared objects, and haven't deleted the
old ones, the rebuild won't be necessary either.

Since, however, the example system is using shared libraries,
the programmer is responsible for remembering to rebuild any
programs that depends on library1 before he moves library1.
At my former workplace, this proved to be a drag on
development efficiency (there never were any problems in
production because the system as a whole was built and moved
to production).

It is usual to do a complete, clean build before moving into
production (and often, once a week, over the weekend). If for
no other reason than to not have to deliver several versions of
each shared library.