Dependency resolution in Java builds

[Markus Gessner]

Hello all!

Having recently started working with Java, coming from C++, I am now
taking part in a Java project, the build of which is managed by Maven.

I frequently run into a very basic situation exemplified by the two
follwing classes, each of which is implemented in its own file:

----------------------------------------------------------------------
public interface Foo
{
void method1();
}

public class FooImpl implements Foo
{
void method1() { ; }
}
----------------------------------------------------------------------

After compiling, I make the follwing change, i.e., only change the
interface:

----------------------------------------------------------------------
public interface Foo
{
void method1();
void method2();
}

public class FooImpl implements Foo
{
void method1() { ; }
}
----------------------------------------------------------------------

When building, I see only a recompilation of Foo.java, which
indicates, that only a simple comparison of the respective timestamps
of *.java and *.class files is undertaken. From my C++-work, based on
gcc/make, I am accustomed to a setup, which in the analogous scenario
for C++ detects the dependence of FooImpl on Foo and initiates a
recompilation of FooImpl as well (in this example leading to a compile
time error).

Moreover, from the Java guys i frequently here something like "when in
doubt, clean the project and rebuild".

Although this is a matter of the build tools rather than the language, I
would like to ask the experienced Java developers out here: is this to
be put up with, or are there build mechanisms obviating this, concisely
taking into account the interdependence of classes (like "gcc -M" in a
make-based build system for C++).

Greetings
Markus

 

[Joshua Maurice]

Hello all!

Having recently started working with Java, coming from C++, I am now
taking part in a Java project, the build of which is managed by Maven.

I frequently run into a very basic situation exemplified by the two
follwing classes, each of which is implemented in its own file:

----------------------------------------------------------------------
public interface Foo
{
  void method1();

}

public class FooImpl implements Foo
{
  void method1() { ; }}

----------------------------------------------------------------------

After compiling, I make the follwing change, i.e., only change the
interface:

----------------------------------------------------------------------
public interface Foo
{
  void method1();
  void method2();

}

public class FooImpl implements Foo
{
  void method1() { ; }}

----------------------------------------------------------------------

When building, I see only a recompilation of Foo.java, which
indicates, that only a simple comparison of the respective timestamps
of *.java and *.class files is undertaken. From my C++-work, based on
gcc/make, I am accustomed to a setup, which in the analogous scenario
for C++ detects the dependence of FooImpl on Foo and initiates a
recompilation of FooImpl as well (in this example leading to a compile
time error).

Moreover, from the Java guys i frequently here something like "when in
doubt, clean the project and rebuild".

Although this is a matter of the build tools rather than the language, I
would like to ask the experienced Java developers out here: is this to
be put up with, or are there build mechanisms obviating this, concisely
taking into account the interdependence of classes (like "gcc -M" in a
make-based build system for C++).

I've been looking for the same thing.

In short, all commonly available build systems suck. It's just a
difference of degree. That includes the idiomatic solution of
"Recursive Make Considered Harmful" with gnu make and gcc -M on c++
code. Ex: Try removing a cpp file, and see if your link output gets
rebuilt. See what happens when you add a new header file which hides
another header file on an include search path.

So, the modern commonly available Java build systems handle even less
of these dependencies than the common "Recursive Make Considered
Harmful" handles. I've been working on my actually incrementally
correct build system on the side which does /correct/ incremental
Java, but don't expect anything soon. (It also relies on Sun JVM's
compiler interface, which isn't Java standard but Sun standard, so
don't expect it to work on other platforms which don't have the Sun
JVM. However, the output bytecode is portable and can run
everywhere.)

I think you'll just have to make do with full clean builds. That's
also the advice I've heard from basically all Java people to whom I've
talked on this subject.

Builds is one of the most under-appreciated aspects of programming.

 

[Arne Vajhøj]

Having recently started working with Java, coming from C++, I am now
taking part in a Java project, the build of which is managed by Maven.

I frequently run into a very basic situation exemplified by the two
follwing classes, each of which is implemented in its own file:

----------------------------------------------------------------------
public interface Foo
{
void method1();
}

public class FooImpl implements Foo
{
void method1() { ; }
}
----------------------------------------------------------------------

After compiling, I make the follwing change, i.e., only change the
interface:

----------------------------------------------------------------------
public interface Foo
{
void method1();
void method2();
}

public class FooImpl implements Foo
{
void method1() { ; }
}
----------------------------------------------------------------------

When building, I see only a recompilation of Foo.java, which
indicates, that only a simple comparison of the respective timestamps
of *.java and *.class files is undertaken. From my C++-work, based on
gcc/make, I am accustomed to a setup, which in the analogous scenario
for C++ detects the dependence of FooImpl on Foo and initiates a
recompilation of FooImpl as well (in this example leading to a compile
time error).

Moreover, from the Java guys i frequently here something like "when in
doubt, clean the project and rebuild".

Although this is a matter of the build tools rather than the language, I
would like to ask the experienced Java developers out here: is this to
be put up with, or are there build mechanisms obviating this, concisely
taking into account the interdependence of classes (like "gcc -M" in a
make-based build system for C++).

The clean and rebuild is the standard solution.

Arne

 

[markspace]

When building, I see only a recompilation of Foo.java, which
indicates, that only a simple comparison of the respective timestamps
of *.java and *.class files is undertaken.

I have it a quick test here, and it worked fine for me. ant detected
that the subclass file needed to be recompiled. Javac told me the
subclass did not implement the interface, and everything stopped there.

Maybe something with your build.xml file? Perhaps if you showed us
exactly what was happening, we could help you.

 

[Lew]

Builds is one of the most under-appreciated aspects of programming.

Second to deployment, which is subordinate to operations, which subsumes the
aforementioned.

Generally, programmers as a class are woefully ignorant of
build/deployment/ops matters. I've seen this repeatedly in
multi-million-dollar projects. One such had radically different builds on
developer machines from those on test/production machines. You could pass all
unit tests on the developer machine and fail on the test box, for example.

Another symptom is framework plethora. Let's just throw our
RichFaces/Acegi/Echo2/Spring/Spring-JDBC/Spring-Aspect/Spring-OMG!/ all over
our XHTML over various disparate JMS scaffolds between localhost processes.
No, that's not a rant, that's a prototypic description of more than one
real-world scenario, both large and small projects. The exact frameworks in
question vary (I mentioned ones I've encountered in various combination) but
the arity is similar. I forbore to mention the classloader and library
incompatibilities between JARS loaded by unchecked programmers and versions
provided by the application server.

On a recent project I had the opportunity to get elbow-deep into Maven builds,
Hudson continuous integration and deployments to a Java EE server. Boy howdy!

On a less recent project, I worked for about six months with the ops guys, in
a town I'll call Deploymentville. Their opinion of the project's programming
team, about twenty miles up the highway in Programmerton? "We like to get
those Programmerton guys over here for about six months," one ops guru told
me, grinning wickedly. "They go back ... /changed/."

--
Lew
Ceci n'est pas une fenêtre.
..___________.
|###] | [###|
|##/ | *\##|
|#/ * | \#|
|#----|----#|
|| | * ||
|o * | o|
|_____|_____|
|===========|

 

[Markus Gessner]

Hello!

First of all, please excuse my typos of yesterday, it was a bit late
here in Europe.

In short, all commonly available build systems suck. It's just a
difference of degree. That includes the idiomatic solution of
"Recursive Make Considered Harmful" with gnu make and gcc -M on c++
code. Ex: Try removing a cpp file, and see if your link output gets
rebuilt. See what happens when you add a new header file which hides
another header file on an include search path.

This may happen, but in my everyday work I never came across such
situations. We even had a medley of metacompilers (moc, proc ...), and
the handling of the additional dependencies introduced by them also
always worked. You only had to explicitly add source files to some
*.pro-files, the makefiles were generated from, but that I happily
submitted to, everything else just working.

To sum it up -- this setup, devised a lot of years ago and never
changed, just worked, I almost never noticed it, which unobtrusiveness
IMHO is the best one can say about a tool. It was just "issue a command
and sit back".

So, the modern commonly available Java build systems handle even less
of these dependencies than the common "Recursive Make Considered
Harmful" handles. I've been working on my actually incrementally
correct build system on the side which does /correct/ incremental
Java, but don't expect anything soon. (It also relies on Sun JVM's
compiler interface, which isn't Java standard but Sun standard, so
don't expect it to work on other platforms which don't have the Sun
JVM. However, the output bytecode is portable and can run everywhere.)

This is a very laudable undertaking, but the fact of its necessity says
a lot about the common regard of this complex, as you note further
below.

I think you'll just have to make do with full clean builds. That's
also the advice I've heard from basically all Java people to whom I've
talked on this subject.

At least now I won't do too much digging, although in this thread
markspace wrote about a different experience, which he made with Ant
(not Maven, I am stuck with, alas).

Builds is one of the most under-appreciated aspects of programming.

Having just begun working with Java, I of course cannot really
appreciate its merits, but it seems, that it would be impossible to
implement the needs of a lot of business oriented applications without
its standards (Java EE), and some frameworks do have a certain charme
(e.g., JAXB and Hibernate). Nevertheless, the build environment I
experience gives me the overall impression of a car, nicely fitted with
teak panels, a GPS system and a minibar, but which has to be cranked
every mile or so. Says a somewhat frustrated mind, who prefers the
tramway

Markus

 

[Markus Gessner]

Hello markspace!

I have it a quick test here, and it worked fine for me. ant detected
that the subclass file needed to be recompiled. Javac told me the
subclass did not implement the interface, and everything stopped
there.

Thank you for your input and the work done for it! This sounds
encouraging, although it seems to contradict the experiences given by
the other posters.

Maybe something with your build.xml file? Perhaps if you showed us
exactly what was happening, we could help you.

It is a host of pom.xml-files, so this would not be feasible, but I
greatly appreciate your kind offer!

My question was aimed at getting an overall appreciation of the
situation and the feasibility of looking for solutions.

Cheers
Markus

 

[Lew]

Having just begun working with Java, I of course cannot really
appreciate its merits, but it seems, that it would be impossible to
implement the needs of a lot of business oriented applications without
its standards (Java EE), and some frameworks do have a certain charme
(e.g., JAXB and Hibernate). Nevertheless, the build environment I
experience gives me the overall impression of a car, nicely fitted with
teak panels, a GPS system and a minibar, but which has to be cranked
every mile or so. Says a somewhat frustrated mind, who prefers the
tramway

OH, come on! Really?

I've worked on some hairy large projects and smaller ones, too. The time to
build an application as affected by this issue is never a factor, unless you
have improperly factored the dependencies, and that would be a problem in any
language.

Java's dependencies, at least when using Ant (I agree Maven is a bitch), are
usually resolvable for incremental builds without issue. When they aren't,
it's because of the unique features of Java (it compiles taking
already-compiled libraries into account, sort of combining compilation and
linking if you're thinking C[++]-ishly). In those situations, you do a clean
build and wait an extra minute. Big effing deal!

If your clean builds take hours, that's not Java's fault, that's yours.

--
Lew
Ceci n'est pas une fenêtre.
..___________.
|###] | [###|
|##/ | *\##|
|#/ * | \#|
|#----|----#|
|| | * ||
|o * | o|
|_____|_____|
|===========|

 

[Lew]

It is a host of pom.xml-files, so this would not be feasible, but I
greatly appreciate your kind offer!

The highly-touted features of Maven, namely the auto-resolution of
dependencies across the Internet, are confusing enough. Add the pom.xml
spaghetti perpetrated by the same numb-nuts who write spaghetti code and your
situation gets pathetic indeed.

I'm on a project now with Maven builds that we inherited from such
incompetents. I've been part of an effort that's so far taken eight months to
resolve the build issues. It's a combination of bad project organization and
Maven abuse (with a liberal dose of framework [Spring, et al.] abuse). The
trouble is you change something in one module (with its own pom.xml) and it
breaks something in another related module. And that one only had ten
pom.xmls at its peak, down to seven thanks to our efforts.

Best of luck.

--
Lew
Ceci n'est pas une fenêtre.
..___________.
|###] | [###|
|##/ | *\##|
|#/ * | \#|
|#----|----#|
|| | * ||
|o * | o|
|_____|_____|
|===========|

 

[Joshua Maurice]

Hello markspace!



Thank you for your input and the work done for it! This sounds
encouraging, although it seems to contradict the experiences given by
the other posters.

Ant's dependency analysis will handle a lot more cases than javac, but
it will still miss a large and important set of incremental changes.
Do not trust it.

 

[Joshua Maurice]

Hello!

First of all, please excuse my typos of yesterday, it was a bit late
here in Europe.



This may happen, but in my everyday work I never came across such
situations. We even had a medley of metacompilers (moc, proc ...), and
the handling of the additional dependencies introduced by them also
always worked. You only had to explicitly add source files to some
*.pro-files, the makefiles were generated from, but that I happily
submitted to, everything else just working.

To sum it up -- this setup, devised a lot of years ago and never
changed, just worked, I almost never noticed it, which unobtrusiveness
IMHO is the best one can say about a tool. It was just "issue a command
and sit back".

I think I'll stick with what I said - it's just a difference of
degree. Once in a blue moon, you could have a new header file which
hides another header file on an include search path, and an
incremental build will produce different results than a full clean
build. Also, as I said, (or perhaps meant to say?), the solution
advocated in "Recursive Make Considered Harmful" is still one of the
best solutions publicly available for C and C++.

Moreover, many of the ideas and concepts presented in that paper are
invaluable. It's what kicked off my quest to become a build expert.

I merely feel that the paper doesn't go far enough. I want a build
system so that, given the correctness of the build system executable,
it is impossible to make source code deltas /or build code/ deltas so
that the incremental build will produce different results than a full
clean rebuild. That, I think, is the critical difference between what
I want and all commonly available build systems. A pom.xml, a
makefile, an ant file, etc., are all executable code, written in an
interpreted domain specific language. I want a build language where
it's impossible to code the wrong thing. As this language is
restricted entirely to builds, I think that such a thing is quite
feasible and practicable. I'm working on it on and off in my spare
time.

 

[Joshua Maurice]

It is a host of pom.xml-files, so this would not be feasible, but I
greatly appreciate your kind offer!

The highly-touted features of Maven, namely the auto-resolution of
dependencies across the Internet, are confusing enough.  Add the pom.xml
spaghetti perpetrated by the same numb-nuts who write spaghetti code and your
situation gets pathetic indeed.

I'm on a project now with Maven builds that we inherited from such
incompetents.  I've been part of an effort that's so far taken eight months to
resolve the build issues.  It's a combination of bad project organization and
Maven abuse (with a liberal dose of framework [Spring, et al.] abuse).  The
trouble is you change something in one module (with its own pom.xml) and it
breaks something in another related module.  And that one only had ten
pom.xmls at its peak, down to seven thanks to our efforts.

I've been experiencing Maven hell for a couple years now. I think
we're over 1000 poms in a build which developers are expected to do
before each checkin. The automated build machine does that 1000 pom
build daily (or more) as well. Maven is the main driver of the build.
The build includes Java, C++, various in-house code generators, /and/
some Eclipse plugin build stuff which is cobbled together with some
rather poor scripts. Feel my pain.

 

[Tom Anderson]

Moreover, from the Java guys i frequently here something like "when in
doubt, clean the project and rebuild".

A process which, for most people, is pretty fast - seconds to minutes.
Much faster than in C, certainly. I suspect that the speed of clean builds
has meant that there isn't much demand for incremental ones, and hence we
don't have any ability to do them.

tom

 

[Tom Anderson]

Builds is one of the most under-appreciated aspects of programming.

I'll drink to that. Indeed, when working on build stuff, i often get the
urge to do so before lunch.

tom

 

[Lew]

I've been experiencing Maven hell for a couple years now. I think
we're over 1000 poms in a build which developers are expected to do
before each checkin. The automated build machine does that 1000 pom
build daily (or more) as well. Maven is the main driver of the build.
The build includes Java, C++, various in-house code generators, /and/
some Eclipse plugin build stuff which is cobbled together with some
rather poor scripts. Feel my pain.

That would be much easier if your project were organized into independent
libraries with some smaller portion dependent thereupon. I'm sure you know
that, and it's likely impossible with such a large fubared code base, but
others reading this might take a lesson.

--
Lew
Ceci n'est pas une fenêtre.
..___________.
|###] | [###|
|##/ | *\##|
|#/ * | \#|
|#----|----#|
|| | * ||
|o * | o|
|_____|_____|
|===========|

 

[Joshua Maurice]

That would be much easier if your project were organized into independent
libraries with some smaller portion dependent thereupon.  I'm sure you know
that, and it's likely impossible with such a large fubared code base, but
others reading this might take a lesson.

Correct on every count.

They're still going to try to un-fubar it by componentizing it. Guess
I'll see how that goes.

 

[Tom Anderson]

Second to deployment, which is subordinate to operations, which subsumes
the aforementioned.

Generally, programmers as a class are woefully ignorant of
build/deployment/ops matters. I've seen this repeatedly in
multi-million-dollar projects. One such had radically different builds
on developer machines from those on test/production machines. You could
pass all unit tests on the developer machine and fail on the test box,
for example.

It's simply stunning that people let this happen in this day and age.

Even more stunning, we did it on our current project. There are some
necessary differences between the dev and server builds (exploded vs
packed EARs, for example), but the differences between the build processes
for them are far greater than that difference requires. I didn't make the
decision to make them so different; someone else did a long time ago. But
that difference has persisted and grown on my watch, so there's blood on
my hands.

Perhaps all that is required for bad builds to triumph is for good
programmers to do nothing.

Another symptom is framework plethora. Let's just throw our
RichFaces/Acegi/Echo2/Spring/Spring-JDBC/Spring-Aspect/Spring-OMG!/ all
over our XHTML over various disparate JMS scaffolds between localhost
processes. No, that's not a rant, that's a prototypic description of
more than one real-world scenario, both large and small projects. The
exact frameworks in question vary (I mentioned ones I've encountered in
various combination) but the arity is similar.

Isn't that just a question of shovelling JARs onto the classpath and then
jarring up the right things? Or do these frameworks have more involved
build requirements?

I forbore to mention the classloader and library incompatibilities
between JARS loaded by unchecked programmers and versions provided by
the application server.

There is that.

On a less recent project, I worked for about six months with the ops
guys, in a town I'll call Deploymentville. Their opinion of the
project's programming team, about twenty miles up the highway in
Programmerton? "We like to get those Programmerton guys over here for
about six months," one ops guru told me, grinning wickedly. "They go
back ... /changed/."

There is this very nebulous thing called The Devops Movement, which is not
an antagonist from a James Bond novel, but some sort of theory that people
with experience of operations should be involved in development:

http://en.wikipedia.org/wiki/DevOps

Seems lke a good idea.

tom

 

[Arne Vajhøj]

The highly-touted features of Maven, namely the auto-resolution of
dependencies across the Internet, are confusing enough.

You can set it up to use your own repository.

Add the pom.xml
spaghetti perpetrated by the same numb-nuts who write spaghetti code and
your situation gets pathetic indeed.

I'm on a project now with Maven builds that we inherited from such
incompetents.

It is my impression that Maven is a 90% tool. It make life easier for
the 90% easiest cases - cases where it is possible to have
the project follow the Maven philosophy. For the remaining
10% the same features that make it easier for the normal cases
make it real hard. So if the project is tricky, then you
are probably better of with Ant that is intended as
a custom build script tool.

Arne

 

[Mike Schilling]

A process which, for most people, is pretty fast - seconds to minutes.
Much faster than in C, certainly. I suspect that the speed of clean builds
has meant that there isn't much demand for incremental ones, and hence we
don't have any ability to do them.

I'd guess you're right.

 

[Lew]

Isn't that just a question of shovelling JARs onto the classpath and then
jarring up the right things? Or do these frameworks have more involved build
requirements?

I wish it was that easy. There are at least two orders of problem. The first
is that the frameworks often pull in different versions of the same libraries.
The other is that the frameworks interact with each other in, ahem, emergent
and unexpected ways. On top of those, sometimes different frameworks have
overlapping functionality, such as Spring/JDBC and Hibernate, making it
difficult to implement a coherent strategy for such functionality.

On a recent project with Maven builds, we had a parent build ("foomain") and
several subsidiary modules ("foofrontend", "foologic", "foopersistence", ...).
Historically different versions of the same frameworks were specified for
the different modules. Yet when you pulled the dependencies into the
"foomain" pom.xml in common the build broke. Turns out some of the
dependencies transitively depend on the same libraries, but different versions
also, some of which are transitive dependencies for other stuff, also
different versions.

I tried "shoveling JARs". Nope. It's subtler than that.

--
Lew
Ceci n'est pas une fenêtre.
..___________.
|###] | [###|
|##/ | *\##|
|#/ * | \#|
|#----|----#|
|| | * ||
|o * | o|
|_____|_____|
|===========|