Incremental Java Compile

[Arne Vajhøj]

So, I ask again: what if the generator changes, which it does
"somewhat" frequently? I'd like to do a build in that case.

But it should not. The team that maintains that generator should
make multiple changes, test them carefully and then release
them to the other teams.

Again, I do not hold sufficient sway, and we're dealing with a product
with a code level published API which wasn't well designed, so we've
coded ourselves into a corner, so to speak.

Stuck with certain API's is a very common thing. But that does
not necessarily mean that thousands of classes change all the
time or that the generator tool change all the time.

Arne

 

[markspace]

I think they missed the important
part of the memo: that it only works when the code is decoupled,
modular, and relatively stable and well defined interfaces instead of
the ~25,000 source file mess we have now.

I've heard that misery loves company, so here you go:

<http://en.wikipedia.org/wiki/Big_ball_of_mud>

"Programmers in control of a big ball of mud project are strongly
encouraged to study it and to understand what it accomplishes, and to
use this as a loose basis for a formal set of requirements for a
well-designed system that could replace it. Technology shifts – such as
client-server to web-based or file-based to database-based – may provide
good reasons to start over from scratch."

 

[Joshua Cranmer]

I am fixing it. I am not whining. I was asking for help on how to do
it. I have asked for real solutions to the real problems I am facing
writing it, such as how to get a list of class files per compiled java
file as if I called javac once per java file in the dir.

final static int constants (or other constant types that ldc works on)
are directly hardcoded into the class file. It is therefore impossible
to read a classfile and tell you which Java classes would have to change
for it to need to be recompiled.

 

[Arne Vajhøj]

I think it could.

As I recall it then the conclusion was that correct handling
of constants (static final) required source-

Arne

 

[Arne Vajhøj]

IF you use ANT, you don't need to bother with this. The time in a
traditional compile is mostly loading Javac.exe. With ANT it gets
compiled only once. Further JAVAC looks at dates of *.java and
*.class files and avoids most unnecessary recompilation.

Well - the safe way is to clean before building.

Arne

 

[Mike Schilling]

I wish I knew. I just got an email today from the serialization team
asking "What's with this error?" I "hacked" the C++ Maven plugin to
report "<> has detected visual studios warning <>, deletion of a
pointer to an incomplete type. This is formally undefined behavior
according to the C++ spec. Fix it." Apparently changes are still
ongoing.


As such a developer, perhaps, but when I mess up, I break the mainline
build, and because the build on the automated build machine, or
private perforce branch build machine, can take the better part of a
day, it's sometimes hard to isolate down who broke it, and especially
when ML is broken this leaves people in a bind. Currently we lock ML
on such events. Rollback is possible. Devops is floating that idea
around at the moment.

You only break the mainline build if you check in code based on doing that
incorrectly. I'm not suggesting that.

Done. Perforce is so awesome for the record.

It is. They do a hell of a job (and I don't say that just because I know a
lot of the folks there.)

Sounds like integration hell. We do have separate teams working on
their own little view for weeks or a month or two on end, and each
team has their own private branch in perforce which is integrated
roughly weekly with mainline.

It shouldn't be hell, especially with a good tool like Perforce helping with
any merges that result. Though if there's a lot of churn in the code and
everything uses everything else, yeah, it'll be harder than if things were
stable and well-organized.

 

[Mike Schilling]

As I recall it then the conclusion was that correct handling
of constants (static final) required source-

You mean analysis of the source is required, because no traces of the
constant use are left in the class file? You know, I assumed that such
constants would be listed in the constant pool, but (having tried a simple
test) I see that that's wrong (and you're correct.). Bad decision.

 

[Mike Schilling]

final static int constants (or other constant types that ldc works on)
are directly hardcoded into the class file.

And no trace of their origin is written to the class file.

It is therefore impossible
to read a classfile and tell you which Java classes would have to
change for it to need to be recompiled.

Because the designers of Java didn't consider that important, or the
necessary information would have been written to the class file.

 

[Joshua Maurice]

Because the designers of Java didn't consider that important, or the
necessary information would have been written to the class file.

And because such information alone is insufficient to do a correct
incremental compile. See the paper in my opening post "Ghost
Dependencies" or some such.

It would go a long way to helping me make it correct though if javac
spat out "loading class X when compiling java file Y" for all such
pairs.

 

[Joshua Maurice]

You only break the mainline build if you check in code based on doing that
incorrectly.  I'm not suggesting that.

Interesting idea. "Don't break the mainline build!" says the managers.
Unfortunately, if they're unable to build to a documented interface,
and the whole build takes hours, if not longer, on their own computer,
then it's hard in practice to not break it.

Just saying.

 

[Mike Schilling]

Interesting idea. "Don't break the mainline build!" says the managers.
Unfortunately, if they're unable to build to a documented interface,
and the whole build takes hours, if not longer, on their own computer,
then it's hard in practice to not break it.

Stay isolated longer. Do the full build and test less often.

 

[Arne Vajhøj]

You mean analysis of the source is required, because no traces of the
constant use are left in the class file? You know, I assumed that such
constants would be listed in the constant pool, but (having tried a simple
test) I see that that's wrong (and you're correct.). Bad decision.

If I remember correct then const in C# is the same way.

Arne

 

[Roedy Green]

Did you even read any of my other posts in this thread? Ant's
incremental compile is woefully incorrect, so incorrect as to be near
useless on an automated build machine.

In the early days the company I worked for a company that had a guy
who did nothing but tweak compile scripts using traditional MAKE-like
logic. They were INCREDIBLY slow compared with ANT. The errors
Javac/ANT makes in deciding what to recompile make little difference
compared to the massive speedup of loading Javac.exe only once. They
are also insignificant compared with jar and zip time.


If you fiddle non-private static finals, remember to do a clean
compile of the universe. Other than that, for all practical purposes,
ANT works.

 

[Joshua Maurice]

On Mon, 3 May 2010 11:36:43 -0700 (PDT), Joshua Maurice


In the early days the company I worked for a company that  had a guy
who did nothing but tweak compile scripts using traditional MAKE-like
logic. They were INCREDIBLY slow compared with ANT. The errors
Javac/ANT makes in deciding what to recompile make little difference
compared to the massive speedup of loading Javac.exe only once. They
are also insignificant compared with jar and zip  time.

If you fiddle non-private static finals, remember to do a clean
compile of the universe.  Other than that, for all practical purposes,
ANT works.

No, no, and no. Perhaps you'll listen this time.

First, my solution only loads javac once. It would be silly to do
otherwise. Moreover, calling javac once per java file is quite slow,
and I was asking for a way around that to get fast, incrementally
correct java compiles. However, even with calling javac once per java,
I still outperformed Maven for a full clean build.

Also, do you have any numbers at all to support your proposition that
jar-ing and zip-ing is the time sucker? Any sources, your own or
otherwise? For a sizable portion of my company's product, with my new
system without incremental dependency analysis, I was able to compile
~3,000 java files in ~3 min. The jar-ing of the resultant class files
took ~15 seconds, and ~8 seconds of that was simply my build system
overhead. It seems that jar-ing is \much\ faster than java compilation
for standard hardware. At least, is it with the -O option to jar, the
"do not compress" option, which should be the standard option during
development.

Finally, no. Ant does not work all of the time for all practical
purposes. I can list off numerous times from the last month where our
"streaming build" because it uses such poor dependency analysis
techniques. It required manual intervention by devops to do a clean to
get it to start passing, and I can assure you static finals were not
the dominant cause. (Though, admittingly, Ant probably does a better
job than Maven.)

However, even then, static finals are part of the language, and I want
an automated build which can actually do automated builds. The build
machine has no (easy) way to determine if a static final was changed
or not, or other (potentially obscure) scenarios which Ant would fail
on. When I as a developer update to a new revision / changelist using
perforce, I do not have an (easy) way to check if there was a change
to a static final, so I would have to do a clean build. This is not
acceptable if there's a feasible alternative.

Also, QA now will never take such a dubiously correct build after
having been burned many times by incrementally incorrect builds,
wasting days of stress testing because the build was "incorrectly"
done.

I will not continue if you just repeat these unfounded and inaccurate
assessments, especially if you do not cite any sources at all, even
your own. For example, I have asked of your own timing numbers for
your own builds for java compilation vs jar -0 times, and you have yet
to provide any.

 

[Lew]

No, no, and no. Perhaps you'll listen this time.

And perhaps you won't be so damn rude next time. What the hell?

You have consistently rejected every piece of good advice, given complete
nonsense excuses for doing so, and thrown mud in the face of people who try to
help you. What a piece of work!

 

[Joshua Maurice]

And perhaps you won't be so damn rude next time.  What the hell?

You have consistently rejected every piece of good advice, given complete
nonsense excuses for doing so, and thrown mud in the face of people who try to
help you.  What a piece of work!

And you have done the same droning, repeating the same untruths which
I have called out, and repeated them thrice in this thread. Such
untruths include:

1- Jar-ing takes longer than java compilation. Correct: No it doesn't,
at least not always, and I would wager not often judging from the
actual numbers before me for my company's code base.

2- Halfway incremental works always in practice. Again, I have lots of
evidence from the automated build in my own company that no, it
doesn't.

3- Everyone immediately assumes that I'm going to implement it
stupidity in make, invoking a separate JVM for each different jar-dir,
possibly per java file. I have said numerous times that I would not do
this, and this is not what I want. When people mention this, it is a
straw man argument. It is a great disservice to me.

I have been so rude because they have been rude to me first, except
they were more insidious about it.

PS: I do agree that we need to componentize. I disagree that
incrementally correct builds are useless after that.

 

[Joshua Maurice]

And you have done the same droning, repeating the same untruths which
I have called out, and repeated them thrice in this thread. Such
untruths include:

Sorry for the collective "you" there. I recognize that it was not
literally "you" who have said everything. Freudian slip. It should
read "him", or as a more dangerous general "them".

 

[Lew]

I have been so rude because they have been rude to me first, except
they were more insidious about it.

Plonk.

 

[Joshua Maurice]

Plonk.

Ok sir. As you will. I tried to have a decent and civil conversation
about a technical detail - the ability to get the list of used class
files per java file in java compilation. Instead of discussing my
topic of interest, I was told half-truths which are repeated ad
nauseum, and downright lies and misinformation.

I am sorry sir that we will no longer have an intelligent discourse,
or any discourse, but neither will I take such abuse lying down. I
have asked numerous times for any such evidence that jar-ing indeed
takes longer than java compilation, as I posit it does not and present
evidence, and certain people have suggested the opposite for quite a
while now. It does tend to grate on one's nerves.

 

[Mike Schilling]

PS: I do agree that we need to componentize. I disagree that
incrementally correct builds are useless after that.

I've worked in systems roughly as large as yours (tens of thousands of
source file) which were layered, so that each seperately compiled subsystem
had at most a few hundreds of files. At that point, there's no particular
advantage to avoiding clean builds.

During development, a developer works on a small set of subsystems. He
knows when he's changing interfaces rather than implementations, and at that
point can afford the clean build.

The automated build-and-test might spend an hour or so on the clean build,
but that's a small fraction of the time the tests take.