speeding up javac

[Aryeh M. Friedman]

I have a very large project that would take to long to recompile stuff
that does not need to be recompiled. For various reasons using ANT
is not possible (doesn't play well enough with other languages). I
looked into Jikes but that is no good because our code is 1.5+. I
tried javac with -g:none and -server and had no improvement at all.
Is there any other way to speed stuff up and still compile things one
class at a time?

 

[JC]

I have a very large project that would take to long to recompile stuff
that does not need to be recompiled.   For various reasons using ANT
is not possible (doesn't play well enough with other languages).  I
looked into Jikes but that is no good because our code is 1.5+.   I
tried javac with -g:none and -server and had no improvement at all.
Is there any other way to speed stuff up and still compile things one
class at a time?

Maybe use Make? Or compile only the Java parts with ANT, but use
something else for the rest of the code that ANT is causing you
problems with?

HTH,
Jason

 

[Aryeh M. Friedman]

Maybe use Make? Or compile only the Java parts with ANT, but use
something else for the rest of the code that ANT is causing you
problems with?

HTH,
Jason

I use cook which is much smarter then make and subtasking to an other
build program would ruin many of the best features of cook such as
full project scanning.

 

[Mike Schilling]

I have a very large project that would take to long to recompile
stuff
that does not need to be recompiled. For various reasons using ANT
is not possible (doesn't play well enough with other languages). I
looked into Jikes but that is no good because our code is 1.5+. I
tried javac with -g:none and -server and had no improvement at all.
Is there any other way to speed stuff up and still compile things
one
class at a time?

If ANT's minimal dependency management is sufficient for your project,
it wouldn't be difficult to write a Java program that reproduces it
and could be called from any make-like build system. In fact, you'd
only have to slightly modify the code that ANT's javac task uses.

 

[Aryeh M. Friedman]

If ANT's minimal dependency management is sufficient for your project,
it wouldn't be difficult to write a Java program that reproduces it
and could be called from any make-like build system.  In fact, you'd
only have to slightly modify the code that ANT's javac task uses.

I already did this to some extent when it selects what to compile (I
wrote a little program to scan imports to make this decision)... yes I
know it is a little dumb since it needs an explicit import for
everything with no wild cards.

 

[Andrew Thompson]

I have a very large project that would take to long ...

Idea. Get a faster box or be more patient.

 

[Aryeh M. Friedman]

Idea.  Get a faster box or be more patient.

Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])

 

[Aryeh M. Friedman]

On Jan 2, 5:05 pm, "Aryeh M. Friedman" <[email protected]>
wrote:

Idea.  Get a faster box or be more patient.

Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])

Forgot to mention without the import scanner I mentioned it is 45 mins
(that is with javac `find . -name '*.java'` not compiling one at a
time I never bothered to time that but it would likely be well over 2
hrs

 

[Andrew Thompson]

Not to bragging ..

While you are 'not bragging' please don't forget
to trim sigs. from replies. Not all of us have
broadband-on-demand. TIA.

 

[RedGrittyBrick]

Idea. Get a faster box or be more patient.

Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])

That does indeed sound insane. How can javac's speed be an issue? Do you
have a single .java file containing a titanic amount of code? Are you
asking javac to compile .java files that don't need compiling? (if so why?)

Either you are doing something rather unusual or my knowledge of Java is
incomplete in some interesting way - please enlighten me.

 

[Aryeh M. Friedman]

Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])

That does indeed sound insane. How can javac's speed be an issue? Do you
have a single .java file containing a titanic amount of code? Are you
asking javac to compile .java files that don't need compiling? (if so why?)

Either you are doing something rather unusual or my knowledge of Java is
incomplete in some interesting way - please enlighten me.

About 80% of that time is taken up by the import scanner (needs to
read about 5,000 files) and the start/stop time for javac.... part of
the issue (and there is no progrmatic way around this I can think of)
is if class Y depends on X and X was recompiled then Y is also
recompiled (make and all it's variants also do this and do a much
worse job then cook does at it)

 

[Knute Johnson]

Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])

It sounds like you should have chunked this job up into some libraries.
I read below that it has 5000 files. Assuming the files are of some
reasonable size that means the program is probably well over a million
lines of code. That's a lot of code for anybody for one program.

Just a suggestion but why don't you change 20 lines of code and compile
once an hour? Just keeping track of where you are working in a program
that size must take some time.

 

[JC]

On Jan 2, 5:05 pm, "Aryeh M. Friedman" <[email protected]>
wrote:
I have a very large project that would take to long ...
Idea.  Get a faster box or be more patient.
--
Andrew Thompsonhttp://pscode.org/
Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])

That does indeed sound insane. How can javac's speed be an issue? Do you
have a single .java file containing a titanic amount of code? Are you
asking javac to compile .java files that don't need compiling? (if so why?)

Either you are doing something rather unusual or my knowledge of Java is
incomplete in some interesting way - please enlighten me.

About 80% of that time is taken up by the import scanner (needs to
read about 5,000 files)

What is the "import scanner"? Can you not run it if imports haven't
changed, or do you have to run it every time you compile a small
change?

and the start/stop time for javac....

The clear solution here is to reduce the number of times javac starts
and stops. If you have to compile a single source file at a time, then
you can't really do anything here. However, if you have the ability to
generate a list of files that need recompiled, javac can accept lists
of files:

http://java.sun.com/j2se/1.4.2/docs/tooldocs/windows/javac.html

And you can eliminate start/stop overhead by generating the list first
then running javac only once.

part of
the issue (and there is no progrmatic way around this I can think of)
is if class Y depends on X and X was recompiled then Y is also
recompiled (make and all it's variants also do this and do a much
worse job then cook does at it)

There is no good programmatic way around this, you are correct. The
compiler would have to track information about what parts of X were
relevant to Y, then analyze your code changes to determine if any of
those relevant parts had changed, and only recompile Y if they have.
This is not trivial, and javac does not do it. Note, however, that
certain design techniques can significantly reduce interdependency
between classes; e.g. copious use of interfaces, more specific classes
(e.g. if Y depends on one part of X that is frequently changing, and Z
depends on another part of X, perhaps it is more appropriate to split
X into two more specific classes), anything that increases the
modularity and isolation of your code will help reduce dependencies.


Jason

 

[JC]

On Jan 2, 6:07 am, RedGrittyBrick <[email protected]>
wrote:

Aryeh M. Friedman wrote:
On Jan 2, 5:05 pm, "Aryeh M. Friedman" <[email protected]>
wrote:
I have a very large project that would take to long ...
Idea.  Get a faster box or be more patient.
--
Andrew Thompsonhttp://pscode.org/
Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])
That does indeed sound insane. How can javac's speed be an issue? Do you
have a single .java file containing a titanic amount of code? Are you
asking javac to compile .java files that don't need compiling? (if so why?)
Either you are doing something rather unusual or my knowledge of Java is
incomplete in some interesting way - please enlighten me.

About 80% of that time is taken up by the import scanner (needs to
read about 5,000 files)

What is the "import scanner"? Can you not run it if imports haven't
changed, or do you have to run it every time you compile a small
change?

and the start/stop time for javac....

The clear solution here is to reduce the number of times javac starts
and stops. If you have to compile a single source file at a time, then
you can't really do anything here. However, if you have the ability to
generate a list of files that need recompiled, javac can accept lists
of files:

 http://java.sun.com/j2se/1.4.2/docs/tooldocs/windows/javac.html

I'm sorry, the correct link is:

http://java.sun.com/javase/6/docs/technotes/tools/windows/javac.html

Jason

 

[Aryeh M. Friedman]

On Jan 2, 6:07 am, RedGrittyBrick <[email protected]>
wrote:

Aryeh M. Friedman wrote:
On Jan 2, 5:05 pm, "Aryeh M. Friedman" <[email protected]>
wrote:
I have a very large project that would take to long ...
Idea.  Get a faster box or be more patient.
--
Andrew Thompsonhttp://pscode.org/
Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])
That does indeed sound insane. How can javac's speed be an issue? Do you
have a single .java file containing a titanic amount of code? Are you
asking javac to compile .java files that don't need compiling? (if so why?)
Either you are doing something rather unusual or my knowledge of Java is
incomplete in some interesting way - please enlighten me.

About 80% of that time is taken up by the import scanner (needs to
read about 5,000 files)

What is the "import scanner"? Can you not run it if imports haven't
changed, or do you have to run it every time you compile a small
change?

and the start/stop time for javac....

The clear solution here is to reduce the number of times javac starts
and stops. If you have to compile a single source file at a time, then
you can't really do anything here. However, if you have the ability to
generate a list of files that need recompiled, javac can accept lists
of files:

 http://java.sun.com/j2se/1.4.2/docs/tooldocs/windows/javac.html

And you can eliminate start/stop overhead by generating the list first
then running javac only once.

part of
the issue (and there is no progrmatic way around this I can think of)
is if class Y depends on X and X was recompiled then Y is also
recompiled (make and all it's variants also do this and do a much
worse job then cook does at it)

There is no good programmatic way around this, you are correct. The
compiler would have to track information about what parts of X were
relevant to Y, then analyze your code changes to determine if any of
those relevant parts had changed, and only recompile Y if they have.
This is not trivial, and javac does not do it. Note, however, that
certain design techniques can significantly reduce interdependency
between classes; e.g. copious use of interfaces, more specific classes
(e.g. if Y depends on one part of X that is frequently changing, and Z
depends on another part of X, perhaps it is more appropriate to split
X into two more specific classes), anything that increases the
modularity and isolation of your code will help reduce dependencies.

Jason

I did this last night and it cut the time in half but still too slow
(namely I rewrote the import scanner to be a shell script instead of a
java program):
#!/bin/tcsh

foreach i (`grep '^import' $1|grep -v '[*];$'|grep -v '^import java'|
cut -f2 -d' '|cut -f1 -d';'|tr '.' '/'`)
echo src/$i.java
end

exit 0

I also rewrote the cookbook (the cook equiv of a makefile) as a
prototype (still needs much more logic about compiling non-Java and
other jugglings) as:

manifest = [collect find src -name "*.java" -print];

all: [fromto src/%0%1.java obj/%0%1.class [match_mask %0%1.java
[manifest]]]
{
echo ISU all built;
}

obj/%0%1.class: src/%0%1.java
set mkdir
{
echo Compiling src/%0%1.java;

javac -d obj -sourcepath src [resolve src/%0%1.java];
}

Note: I will be placing a call to the above script in to this later
and expect an other 5 to 10% increase is speed. But the overall
issue still remains

 

[Lew]

It sounds like you should have chunked this job up into some libraries.
  I read below that it has 5000 files.  Assuming the files are of some
reasonable size that means the program is probably well over a million
lines of code.  That's a lot of code for anybody for one program.

I've been on Java projects that large. Having such a project all in
one compile is a mistake; don't blame the compiler.

Knute is right - break the project up into libraries and your problem
will go away. That's what the projects did where I worked. I can
rebuild code in a minute or so on a much weaker workstation than the
OP described.

This is more than just a compilation issue. In a monolithic structure
such as the OP implies they have, testing is nigh impossible.
Everything depends on everything. How can one ever do effective
regression testing on changes?

By separating the program into independent modules, the state space
adds rather than multiplies as project size grows. For example, if
the project has four parts, A, B, C and D, with respectively 3, 4, 5
and 7 states, the monolithic approach has to test and verify 3 * 4 * 5
* 7 = 420 states. Separation of the project into four libraries means
it only has 3 + 4 + 5 + 7 = 19 states to test and verify.

The fault, dear Brutus, lies not in our compilers but ourselves.

 

[Tom Anderson]

On Jan 2, 5:05 pm, "Aryeh M. Friedman" <[email protected]>
wrote:

I have a very large project that would take to long ...
Idea.  Get a faster box or be more patient.

Not to bragging but quad core running at 2.58 GHz and 4 GB of RAM with
a sata 300 drive should be enough... and as far as patience goes I
think 5 mins to recompile for a 1 one line change is insane
(exspecially when we do this on a quite regular basis [say 20 times an
hour])

That does indeed sound insane. How can javac's speed be an issue? Do you
have a single .java file containing a titanic amount of code? Are you
asking javac to compile .java files that don't need compiling? (if so why?)

Either you are doing something rather unusual or my knowledge of Java is
incomplete in some interesting way - please enlighten me.

About 80% of that time is taken up by the import scanner (needs to read
about 5,000 files) and the start/stop time for javac.... part of the
issue (and there is no progrmatic way around this I can think of) is if
class Y depends on X and X was recompiled then Y is also recompiled
(make and all it's variants also do this and do a much worse job then
cook does at it)

We had a discussion about this a while ago. The conclusion was that this
was a fairly hard problem.

Idea number one would be to ditch your custom framework, and use Eclipse
JDT:

http://www.eclipse.org/jdt/core/index.php

That's an incremental compiler - it does all the dependency tracking and
recompilation itself. It does a minimal amount of work, and is thoroughly
tested.

Idea number two would be to (a) cache the results of your import scans, so
you don't need to do them afresh each time (checking the timestamp of the
source file against that of the cache to see if it does need redoing, of
course) and (b) store the cache in such a way that you can quickly look up
dependents of source files that have changed (eg by keeping two files per
source file - one listing files it depends on, and one listing files which
depend on it; when you recompile a file, you recompute the dependencies,
and use any differences between the fresh list and the list in the file to
drive changes to the depended-on-by lists of the other files - does that
make sense?).

tom

 

[Roland de Ruiter]

While you are 'not bragging' please don't forget
to trim sigs. from replies. Not all of us have
broadband-on-demand. TIA.

Please use the proper sig block separator: "-- ", i.e. dash dash space
(followed by a new line).

 

[Andrew Thompson]

...
Please use the proper sig block separator: "-- ", i.e. dash dash space
(followed by a new line).

Please realize that *no* poster from GG
can use the correct sig. delimiter, since
the GG WITUN chews up the trailing ' '
from '-- '.

<suggestion>
Deal with it, or block the offending posters
and stop whining.
</suggestion>

 

[Andrew Thompson]

On Jan 3, 11:16 pm, "Peter Duniho" <[email protected]>
wrote:
....

By the way, no one is forced to use the travesty that is the Google Groups  
Usenet portal.  

Great! What do you suggest for someone
posting from an internet cafe, or the local
library, who does not have the right nor
ability to install a news client, nor hunt
around for free providers of usenet archives?

JavaKB? Tried it, as many problems as the
GG WITUN.

(Some of you people have it too lucky - force
you to use someone else's computer through a
low bandwidth modem and I reckon it would wipe
that silly grin off your face.)