I mentioned this little side project I had here quite a while ago in\ncomp.lang.c++. The context was me asking for an incrementally correct\nbuild system.\n\nTo explain, let me define some terms.\n\nA build system tool - is a distributed, customizable tool that let's\nyou create a build system. Ex: GNU make.\n\nA build system - is a particular build system, usually built on top of\nsome build tool. This is the in-fact implementation that does your\nbuild. GNU make is a build tool, which can be used to create a build\nsystem by writing a makefile.\n\nAn incremental build - is a build on top of a pre-exsting build, where\nthere have been some source code changes between the old build and\nnow. Usually such builds do not rebuild everything and instead use\nsome "intelligence" to build less than the full clean rebuild, but\nenough to get results /as if/ it did a full clean rebuild.\n\nA correct incremental build - is an incremental build which produces\nresults "equivalent" to that of a hypothetical full clean rebuild.\n\nAn incrementally correct build system - is a build system which can\nonly produce incrementally correct builds. That is, there is no source\ncode changes which a developer may do, inadvertently or purposefully,\nwhich can result in the build system doing an incrementally incorrect\nbuild.\n\nThe key part to realize about this requirement is that the build\nsystem scripts, such as makefiles, are themselves source code under\nthis model. At least, I claim that they ought to be considered as\nsource code for the purposes of determining if your incremental build\nsystem is incrementally correct.\n\nNow, at this point, there is a blurring of lines between build system\ntool and build system. When the makefiles are both source code and\nbuild system, it becomes hard to distinguish between them. I'm merely\ntrying to say that I want a tool in which it is exceptionally hard, if\nnot impossible short of maliciousness, to "break" the build system\nsuch that it might produce an incrementally incorrect build.\n\nNow, why do I want this? I hate waiting for hours each day doing a\nbuild. Yes, I know that the source code base in question has other\nproblems if it takes that long to build, but unfortunately I am not\nthe king of the company, and so I have to make due. Besides, even\nspending 10 minutes on a full clean rebuild is 10 minutes\n(potentially) wasted, if another better solution exists. And even more\nimportantly, 4 hours spent tracking down a bug which was simply the\nresult of a broken incremental build is 4 hours wasted.\n\nI claim that it's not that hard. However, as far as I can tell, very\nfew people have attempted such a thing. In the C++ world at least,\nmost ended with something that was "good enough" that caught 95% of\nthe cases. A good example AFAIK is the Recursive Make Considered\nHarmful solution, found here:\n[URL]http://miller.emu.id.au/pmiller/books/rmch/[/URL]\nIt's good, but it still misses cases. Depending on the exact details\nof implementation, it won't do a rebuild if you change compiler or\nlinker options, such as a new preprocessor define. It won't relink an\nexecutable if you delete one of its associated cpp source files. It\nwon't recompile an object file if you introduce a new header file\nwhich hides a previously included header file on the include path.\n\nI'm not presumptuous enough to claim that what I'm about to link to is\nfully incrementally correct, but AFAIK it's much further along than\nanything else I've ever found, and it has the good potential to get\nthere.\n\nDo note that you have to build trust on trust. To take a side trip,\nsecurity works by trust. You have to have some trusted agent in order\nto have secure communication online. In a similar token, the\nincremental correctness of a build system cannot be guaranteed if\nsomeone "accidentally" modifies some of the make source code badly,\nand installs it locally. The incremental correctness of a build system\nis thus contingent on some facts, like on the correctness of some core\nbuild system executables. It would also be dependent on the\ncorrectness of the compiler and other such things.\n\nMy aim is simply to catch as much as I can, focusing on common\ndeveloper activities, such as changing preprocessor defines, adding\nfiles, removing files, and ultimately using source control to sync to\na new revision of the code, which will include changes of which he is\nnot aware, and the incremental build system ought to just "work".\n\nSo, I'm experimenting around with ideas now. My initial plan was to do\na make clone. What I have is somewhat close to a drop in replacement\nto GNU make 3.81. It's missing a lot of the bells and whistles - for\nexample it doesn't have a job server, nor a bunch of built-in\nfunctions. I also hate the tab syntax, and I would be loathe to add\nthat back in. However, it has user definable functions, the same basic\nscripting language, a bunch of the built-in functions, and simple and\nrecursive variables.\n\nIn fact, it has enough functionality in similar that I am prepared to\neven do some speed comparisons between the two, my make, infamake, and\nGNU make. On a standard Linux box, infamake parses makefiles about 20\ntimes faster (not 20%, but 20x). On a standard Linux box, infamake\nalso does an "all from all" build about 8 times faster (not 8%, but\n8x) than GNU make. An "all from clean" build invoking g++ on basically\nempty cpp files runs about just as fast with my infamake and GNU make\n- stuck on IO I presume. I have included the generators of these\ntests, so that other people can look at them and confirm, or point out\nweaknesses in my testing approach.\n\nWhile I lack a lot of the functionality of GNU make, I believe that I\nhave enough of it that the missing features would not noticeably\nchange the results of these tests.\n\nWith a little work, I could put the rule syntax with all of its evil\ntabs in, and make it an actual drop-in replacement for GNU make (as\nlong as you don't do things like job control, vpath, etc.).\n\n[URL]https://sourceforge.net/projects/infamake/files/[/URL]\n\n(Documentation is sorely lacking, I know. It comes with a bunch of\nunit tests, which should make it clear if you spend a lot of time with\nit. I'll try to add some documentation sometime soon if anyone\ncares.)\n\nSo, why did I rewrite it instead of fixing GNU make? A couple of\nreasons. First, it was a fun exercise. Second, I didn't want to deal\nwith the existing C source code. Third, I felt that the original make\nway of doing incremental builds is fundamentally broken, so I wanted\nto diverge a bit from that legacy.\n\nLet me explain why on that third point, why I think the GNU make\napproach is fundamentally broken. GNU make, and in fact make in\ngeneral, uses a directed acyclic graph of nodes to control which\nportions of the build are skipped on an incremental build. The nodes\nusually correspond to files. This works out well in a lot of cases,\nbut it fails spectacularly on some common developer actions. As I\nmentioned before, example include adding files, deleting files,\nchanging build commands. None of this is well captured by looking at a\nDAG where the nodes are existing files. There are some clever hacks to\nget around some of these issues, namely the .d file generation of\nRecursive Make Considered Harmful, but it doesn't catch all of them.\n(The src/tests2 folder includes a bunch of tests I run against my own\ninfamake built in C++ rule to ensure that it produces correct\nresults.)\n\nMoreover, this problem is even worse for Java. Fortunately /\nunfortunately, my company also uses Java, and apart from Eclipse's\ncompiler (which is almost fully incrementally correct for Java, but\nnot quite, AFAIK), there isn't anything even resembling an good\nattempt an incremental Java builds. Part of my motivation is to create\na build system which handles C++ and Java, and incrementally\ncorrectly, and with fast builds. None of this is uploaded right now,\nbut I have a worked out implementation lying around that I just need\nto merge in.\n\nCurrently, the build logic is done largely in C++. I did this for\nspeed, and honestly because I'm still not quite sure if there's a good\nway to generalize it to something easy to use in make's scripting\nlanguage. I wanted the full power of C++ when doing the incremental C+\n+ build logic. Perhaps I can fix that later. I don't know.\n\nSo where do I go form here? I don't know. Just thought I'd post this\nup, as I made some pretty large claims a while ago on comp.lang.c++,\nand felt I might as well back them up. Took a bit longer than expected\nfor legal to get back to me to open source it.\n\nSo, any comments?