Books for learning how to write "big" programs

D

duli

Hi:
I would like recommendations for books (in any language, not
necessarily C++, C, python) which have walkthroughs for developing
a big software project ? So starting from inception, problem
definition, design, coding and final delivery on a single theme
or application.

Most of the code I have written and books that I have read deal with
toy programs and I am looking for something a bit more
comprehensive. For example, maybe a complete compiler written in C++
for some language, or a complete web server or implementing
..net libraries in some language (just a few examples of the scale of
things I am interested in learning).

Thanks!
Duli.
 
M

Martin P. Hellwig

duli said:
Hi:
I would like recommendations for books (in any language, not
necessarily C++, C, python) which have walkthroughs for developing
a big software project ? So starting from inception, problem
definition, design, coding and final delivery on a single theme
or application.

Most of the code I have written and books that I have read deal with
toy programs and I am looking for something a bit more
comprehensive. For example, maybe a complete compiler written in C++
for some language, or a complete web server or implementing
.net libraries in some language (just a few examples of the scale of
things I am interested in learning).

Thanks!
Duli.

Hmm I think it has not so much to do with the programming language or
the complexity itself. Might be a good thing to look for stuff like
project management, Release Management and Software Life Cycle.

A good starting point might be this:
http://en.wikipedia.org/wiki/Software_development_process

In my opinion, bigger projects are like smaller projects excepts you
have about 90% overhead on documentation before you even write a single
line of code, of course you can try without but that usually ends in
failure of the project.
 
P

Puppet_Sock

In my opinion, bigger projects are like smaller projects excepts you
have about 90% overhead on documentation before you even write a single
line of code, of course you can try without but that usually ends in
failure of the project.

With the caution that the small-big division here should be
round about the size of project that you can do in one day,
and can hold the entire spec in your head without notes.

And that the percentage may not be quite as high as 90%.
But it should be well over half. Actual coding on most
complicated commercial projects isn't usually as much as
50 percent of the total effort. And the larger the app,
the more time you need to spend in docs, design, testing,
verification, and other such issues.

As several of my bosses have stated: The job is not over
till the paperwork is done.

There are a few language specific issues for large apps.
Probably you can get most of the benefit from such books
as _Effective C++_ and the other books by Scott Meyers
in the Effective line. Don't forget to look for the CD
version of the first two books.

Also, I see that John Lakos seems to have updated his book.

Large-Scale C++ Software Development by John S. Lakos
(Paperback - Mar 28 2009)

But it does not seem to be possible to order it from
Amazon as yet. I don't know if this is just the old
text (which lost stars in reviews due to being dated)
or a new text with updated coverage.
Socks
 
M

Mike Driscoll

Hi:
I would like recommendations for books (in any language, not
necessarily C++, C, python) which have walkthroughs for developing
a big software project ? So starting from inception, problem
definition, design, coding and final delivery on a single theme
or application.

Most of the code I have written and books that I have read deal with
toy programs and I am looking for something a bit more
comprehensive.  For example, maybe a complete compiler written in C++
for some language, or a complete web server or implementing
.net libraries in some language (just a few examples of the scale of
things I am interested in learning).

Thanks!
Duli.

The "Programming Python 3rd Ed." by Lutz has a fairly extensive email
program both as a CLI and a GUI (Tkinter) program. He has various
other real programs too, but they're much smaller.

Other than that, I'd recommend joining a project programmed in Python.
Search SoureForge for them...

Mike
 
I

Ian Collins

Martin said:
In my opinion, bigger projects are like smaller projects excepts you
have about 90% overhead on documentation before you even write a single
line of code, of course you can try without but that usually ends in
failure of the project.
I though it was the over emphasis on paper and lack of working code that
has caused the failure of too many projects.
 
J

James Kanze

Martin P. Hellwig wrote:
I though it was the over emphasis on paper and lack of working
code that has caused the failure of too many projects.

If all you produce is paper, I don't think that the project can
be considered a success. On the other hand, anything that
hasn't been written down and reviewed isn't there. I certainly
wouldn't go with "90% overhead on documentation"; I would argue
that any "overhead" on documentation is a sign of a poor
process. Documentation, in various formats, does appear as a
side effect of other aspects of a good process, however: you
can't do a design review unless there is a design, written down
somewhere; you can't do code review unless there is code,
written down somewhere (and the code has to be readable, which
could also be considered documentation); you can't review the
completeness of the unit tests unless there is something which
says what the unit tests test. And you're really wasting your
time if you write a single line of code without knowing why, or
what it should do---and what isn't written down, isn't known.
(That doesn't mean you need a classical external document. Just
that whatever you were thinking when you wrote the code is
written down somewhere, in some form or another.)

From experience, I find that about 60% of the time in a project
will be spent in design, 30% in coding (including coding the
tests), and about 10% in debugging. But of course, it varies
greatly. And what you count in which phase may vary---if you're
using a library you're not familiar with and which isn't well
documented, you'll almost certainly end up writing some
"experiments" to see how it behaves: is that "coding", "design",
or yet something else?

I don't think that there's that much difference between big
projects and small projects in this respect. The difference
between the two is how you manage larger numbers of people on
the bigger project. In other words, not the "documentation",
but how you communicate it between developers, and how you
ensure that everyone is on the same wave length.
 
M

Matthias Buelow

James said:
And you're really wasting your
time if you write a single line of code without knowing why, or
what it should do---and what isn't written down, isn't known.

This might be true for some projects but certainly not in the general
case -- sometimes, the goals to achieve are somewhat blurred or might
not yet be known in entirety and have yet to be discovered. In such
situations, explorative programming is (imho) more effective than trying
to pen it down on paper beforehand.
 
I

Ian Collins

Matthias said:
This might be true for some projects but certainly not in the general
case -- sometimes, the goals to achieve are somewhat blurred or might
not yet be known in entirety and have yet to be discovered. In such
situations, explorative programming is (imho) more effective than trying
to pen it down on paper beforehand.

Indeed. I'm just embarking on that voyage of discovery with my current
client. With eight stakeholders, all with slightly different and
somewhat vague ideas what they want, it should be an interesting journey.
 
M

Michele Simionato

Hi:
I would like recommendations for books (in any language, not
necessarily C++, C, python) which have walkthroughs for developing
a big software project ? So starting from inception, problem
definition, design, coding and final delivery on a single theme
or application.

Take any Python Web framework, study the source and ask questions.
Also, keep reading this newsgroup, there are plenty of discussions
about (good) design practices.

Michele Simionato
 
J

James Kanze

This might be true for some projects but certainly not in the
general case -- sometimes, the goals to achieve are somewhat
blurred or might not yet be known in entirety and have yet to
be discovered. In such situations, explorative programming is
(imho) more effective than trying to pen it down on paper
beforehand.

Yes, but you still have to decide what you're exploring. You
don't just write random code; you write code to fulfill some
specific specification. Whether you're sure that it's the right
specification or not is separate issue---in practice, I'd say
that it's even more important to document what the code is doing
if you're not sure about the specification.
 
J

James Kanze

I think there's an important difference, but it depends on
just what a "small project" is. For relatively new
programmers, a small project is something they've worked on by
themselves, something simple enough that they can hold all of
the documentation in their mind without writing it down.

Part of my point was that you don't in fact know anything that
isn't written down. You just think you do.

Normally, no professional project should be done by just one
person, but if you don't have the choice, I find that writing
something down forces me to be clearer about it, and to actually
think it through. In my own case, at least, I don't seem to be
able to get away with as much "hand waving" when I'm writing
things down.
 
I

Ian Collins

James said:
Yes, but you still have to decide what you're exploring. You
don't just write random code; you write code to fulfill some
specific specification.

As specified by your unit tests. The requirements from the customer are
often very vague, especially in the web application world. In some
domains, the best way to get detailed requirements from a customer is to
start building the application. Once they see something, they then
realise exactly what they want it to do.

The quality of requirements varies from the exact (say a safety critical
device driver) to the virtually nonexistent (I started what turned out
to be a one year web project with a single page sketch). We have to be
able to adapt our development process to match.
Whether you're sure that it's the right
specification or not is separate issue---in practice, I'd say
that it's even more important to document what the code is doing
if you're not sure about the specification.
Indeed, half the battle is getting the customer to agree (or if they are
game, write) acceptance tests. In the vague and woolly world of web
based applications, tools like FitNesse have evolved to help users write
tests.
 
I

Ian Collins

James said:
Part of my point was that you don't in fact know anything that
isn't written down. You just think you do.
That's a good point.
Normally, no professional project should be done by just one
person, but if you don't have the choice, I find that writing
something down forces me to be clearer about it, and to actually
think it through. In my own case, at least, I don't seem to be
able to get away with as much "hand waving" when I'm writing
things down.
As we are all individuals, the form of the writing down that works best
varies from person to person. I prefer to express my thoughts as code,
in the form of tests. Some prefer UML diagrams, others prefer use cases
or even good old functional requirements.

The most important thing, as you say, is that you do write something
down before you write any application code.
 
W

Walter Roberson

James Kanze said:
Part of my point was that you don't in fact know anything that
isn't written down. You just think you do.

And of course as soon as more than one programmer gets involved,
what one thinks they know might not be the same as what the other
person thinks is known about the same topic.

Just this afternoon I asked a co-worker to put an interface in writing,
specifically so as to "lock it in", making it clearer that if the
interface changes after this point that I have to be notified of
that.

The interface itself was relatively simply described... or at least
it was until I replied back and asked about a bunch of corner-cases
that have shown up in our work from time to time.

The responses to several of my questions might be "No, we don't do that".
But those "No's" will form part of the interface description, parts
that were left undifferentiated may become at least partly defined
by our now saying that we will NOT use particular fields in
particular ways.

It is exactly as you said: until something is written down (and, ideally,
reviewed), we just *think* we know it.
 
J

James Kanze

As specified by your unit tests.

And how do you specify your unit tests?
The requirements from the customer are often very vague,
especially in the web application world. In some domains, the
best way to get detailed requirements from a customer is to
start building the application. Once they see something, they
then realise exactly what they want it to do.

Certainly. But you do have to specify what you're going to
code. That doesn't mean that the specification is cast in
stone, but that you have a trace of what you've done, and that
you know what you've done. (For certain things, it's probably
acceptable to specify "whatever"---but you should at least know,
and have written down, that it doesn't matter, and that whatever
happens to come up as a result of the implementation is fine.
And if the customer later declares that it is important, then
you put it in writing, and do code review and add tests to
ensure that it behaves like that in all situations, and after
all further modifications.)
The quality of requirements varies from the exact (say a
safety critical device driver) to the virtually nonexistent (I
started what turned out to be a one year web project with a
single page sketch). We have to be able to adapt our
development process to match.

I'm not sure that we agree on what is meant by "specifications".
In a very real sense, all projects start out as a vague idea,
which is fleshed in little by little. And all successful
projects end up with a working system, with code. The only
issue is how much of this is done exclusively by the client, how
much exclusively by the development team, and how much in
collaboration.

It's true that in many of my applications, an important part of
the specification has been an RFC (or several); if you're
writing a RADIUS server (for dynamic allocation of IP
addresses), you don't "experiment" to find out what the RADIUS
protocol should be.
Indeed, half the battle is getting the customer to agree (or
if they are game, write) acceptance tests. In the vague and
woolly world of web based applications, tools like FitNesse
have evolved to help users write tests.

In a visual application, I can imagine that having the customer
"sign off" screen shots could be an important part of the
specification process; if you have a tool which allows the
customer to actually generate the screen shots himself, so much
the better. That still leaves a number of questions open: in a
web application, for example, different levels of logging are
doubtlessly required. You can't really expect the customer to
write, or even understand, test programs which verify that the
logs are correct.
 
J

James Kanze

James Kanze wrote:

[...]
As we are all individuals, the form of the writing down that
works best varies from person to person. I prefer to express
my thoughts as code, in the form of tests. Some prefer UML
diagrams, others prefer use cases or even good old functional
requirements.

Tests can be part of it; I'm open to all forms of
"documentation", or "writing down". A significant part of the
contract can sometimes even be expressed in the function
signature. Typically, however, a test represents enough code
that I would want something more abstract. And a test doesn't
really work well as documentation for pre-conditions (e.g. this
pointer may not be null).
The most important thing, as you say, is that you do write
something down before you write any application code.

I said "any code", not "any application code". IMHO, the same
thing applies to test code. On the other hand, by code, I
really mean executable code. Things like function signatures
are definitly an important part of documentation.

I remember one project where there was an absolute rule that
every function must be preceded by Doxygen like comments, with a
comment for each parameter, and a short statement saying what it
did, at the least. Which led to things like:

// Sets the administrative state.
//
// \param newAdministrativeState
// The new administrative state.
// -----------------------------------------------------------
void setAdministrativeState(
AdministrativeState const& newAdministrativeState ) ;

At least in the base class, I think that the function signature
is already more than enough, and management was responsive
enough that when presented with this example, they changed the
rule. (In the derived classes, seting the administrative state
could, and often did, have significant side effects.)
 
I

Ian Collins

James said:
And how do you specify your unit tests?

That's pretty close to asking how do you specify your specifications?
Unit tests can be used as executable requirements.
Certainly. But you do have to specify what you're going to
code. That doesn't mean that the specification is cast in
stone, but that you have a trace of what you've done, and that
you know what you've done.

Tests do that for you. They are concise, executable and always up to date.
I'm not sure that we agree on what is meant by "specifications".

I think we are close on a meaning, but differ on the execution, if you
excuse the pun.
In a visual application, I can imagine that having the customer
"sign off" screen shots could be an important part of the
specification process; if you have a tool which allows the
customer to actually generate the screen shots himself, so much
the better. That still leaves a number of questions open: in a
web application, for example, different levels of logging are
doubtlessly required. You can't really expect the customer to
write, or even understand, test programs which verify that the
logs are correct.
Depending on the application, that falls in the unit tests, or the
customer acceptance tests, or both. Sometimes the client simply doe not
care (I have been asked "what logs?"). In other cases, the
organisation's IT department will provide the requirements and tests.
 
I

Ian Collins

Paavo said:
My English might not be the best, but this phrase "customers are game"
should AFAIK mean that the customers will be hunted down and killed :)
In a couple of cases, I wouldn't argue with that!
 
N

noagbodjivictor

Hi:
I would like recommendations for books (in any language, not
necessarily C++, C, python) which have walkthroughs for developing
a big software project ? So starting from inception, problem
definition, design, coding and final delivery on a single theme
or application.

Most of the code I have written and books that I have read deal with
toy programs and I am looking for something a bit more
comprehensive.  For example, maybe a complete compiler written in C++
for some language, or a complete web server or implementing
.net libraries in some language (just a few examples of the scale of
things I am interested in learning).

Thanks!
Duli.

So much text, and no suggestion!
OP, this very message board suggested me not long ago: Software tools
in Pascal. Yes it's pascal, but Pascal is not that hard. I have read
through chapter 3, and so far I have learned:
How to implement strings (yes we do that in C already :) )
How to write a regular expression parser
How to write the famous C "#include" (this was not available in the
old pascal they were using"

There is so much more, and at the end of the book, there will be a
text editor. This book has already changed how I view programming. I
just hoped there was a version with up to date programming languages.
Freepascal has already most of the stuff implemented, which infuriates
me sometimes because I have to name my strings mystring :)

Also there is "Beautiful Code", where the greatest programmers have
put together the code the loved most in their entire life (with full
code). I heard the creator of C has discussed his spam detector (the
one in Practice of programming) in it. The creator of Ruby has also
some input in it.
This is definitely a book I should get.
 
J

James Kanze

So much text, and no suggestion!
OP, this very message board suggested me not long ago:
Software tools in Pascal.

That might have been me; I have recommended the book in the past
(and still do). It's an excellent book (and it's really in C,
disguised as Pascal.) It's not about programming large
applications, however, and doesn't begin to address the issues
raised when you have, say, 50 people working on the project.
Yes it's pascal, but Pascal is not that hard. I have read
through chapter 3, and so far I have learned:
How to implement strings (yes we do that in C already :) )
How to write a regular expression parser
How to write the famous C "#include" (this was not available in the
old pascal they were using"
There is so much more, and at the end of the book, there will
be a text editor. This book has already changed how I view
programming.

It does that. Compared to books teaching a language, it does
teach how to write programs that do something useful.
Independently of the language---I once used it as a course book
for a course in C, since the way the authors write Pascal is
really C in disguise.

If you're interested, try converting some of the programs to
C++. The only real issue is what classes to define (if
any---some of the earlier code is so simple that adding classes
wouldn't be justified). If you have questions, post them here.
I just hoped there was a version with up to date programming
languages. Freepascal has already most of the stuff
implemented, which infuriates me sometimes because I have to
name my strings mystring :)

Part of adopting the code to C would be, IMHO, using std::string
and std::vector rather than the code they propose. There's
enough other good stuff in there that isn't supported directly
by the C++ library.
Also there is "Beautiful Code", where the greatest programmers
have put together the code the loved most in their entire life
(with full code). I heard the creator of C has discussed his
spam detector (the one in Practice of programming) in it. The
creator of Ruby has also some input in it.
This is definitely a book I should get.

I'm not familiar with it. When you spoke of "large" programs,
however, I immediately thought a million or so lines. Reading
back, I think what you meant wasn't so much what a professional
would consider "large", but simply something which was large
enough to do some realistic work. If that's the case, you've
probably found the best book around. For something more C++
oriented (but which doesn't present full projects), Barton and
Nackman's "Scientific and Engineering C++" is quite good (but
beware that it is also quite dated---the ideas are still very,
very good, but the C++ isn't the same language we call C++
today). I'd also recommend Robert Martin's "Designing Object
Oriented C++ Applications Using the Booch Method", but beware
that it is very dated, since not only has C++ greatly evolved
since it was written, but Booch has given way to UML.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top