The curse of constant fields

J

Juha Laiho

I got recently bitten by something I consider to be a rather nasty
feature in the Java language, and decided to write up about it.

I had to do a couple of simple changes to a relatively shaky and
not-so-testable codebase, so I chose to do the absolute minimum
required: change initialization values for a couple of String fields
in a class, compile the changed class, and run the code manually to
observe that I had achieved the desired change.

What I didn't pay attention to initially was that the strings were
declared final. So, I changed the values, compiled the changed
class and replaced old classfile with new. Time for the test run,
and what I see is that the program still shows the original values.

A quick search through the source code shows that there is no other
component in the program containing a copy of these values except
the one I already changed - so, I'm starting to worry whether even
the version control for the code is out of synch with the actual
compiled code. This feeling did strenghten quite a lot when I found
out that another class (which directly uses the values of the fields
I changed) contains the old values of the fields. So, has there been
some other version of the _other_ class that at some point did
itself contain the fields that now need to be changed? I rummage
around, but do not find any such version of the source code for
the offending class. At a whim, I recompile the offending class
from source code fresh out of the version control, to observe any
possible differences in the compilation results. The results seem
pretty much similar, except that now the newly compiled class file
contains the changed value for the fields - which were declared
and initialized IN A JAVA SOURCE FILE FOR ANOTHER CLASS! (apologies
for raising my voice)

So, problem solved and programmer confused. Off I go to take a look
at the JLS, and find the explanation there in chapter 13.4.9:
: If a field is a constant variable (§4.12.4), then deleting the
: keyword final or changing its value will not break compatibility
: with pre-existing binaries by causing them not to run, but they
: will not see any new value for the usage of the field unless they
: are recompiled. This is true even if the usage itself is not
: a compile-time constant expression (§15.28)

(with other things hinting to this same behaviour in a number of
places, among others in ch. 3.10.5)

In a word, ouch. A lesson learned.
 
A

Arne Vajhøj

Patricia said:
Juha said:
I got recently bitten by something I consider to be a rather nasty
feature in the Java language, and decided to write up about it.

I had to do a couple of simple changes to a relatively shaky and
not-so-testable codebase, so I chose to do the absolute minimum
required: change initialization values for a couple of String fields
in a class, compile the changed class, and run the code manually to
observe that I had achieved the desired change.
...

You have my sympathy for your frustration, but I would call this story
"The curse of manual builds".

Back in the 1970's, long before Java was invented, I worked on a large
project that avoided reassembling all the code by having a team of
programmer techs who constructed the list of modules that needed to be
assembled due to each change. The programmer techs did their best, but
it was a *very* error prone procedure.

Since then, I have seen complaints, for several languages, of the form
"I did a manual build, and got into trouble due to [some aspect of the
programming language or environment]".

I am convinced that the only viable choices are rebuilding everything
for every change or using language-appropriate automatic dependency
analysis.

And tools like ant make it easy.

Arne
 
R

Roedy Green

In a word, ouch. A lesson learned.

Rule of thumb, any time you change the values of any non-private
constants, do a clean compile.

I got nailed with that myself the other day, with build numbers that
failed to increment.
--
Roedy Green Canadian Mind Products
http://mindprod.com
PM Steven Harper is fixated on the costs of implementing Kyoto, estimated as high as 1% of GDP.
However, he refuses to consider the costs of not implementing Kyoto which the
famous economist Nicholas Stern estimated at 5 to 20% of GDP
 
M

Mike Schilling

Arne said:
Patricia said:
Juha said:
I got recently bitten by something I consider to be a rather nasty
feature in the Java language, and decided to write up about it.

I had to do a couple of simple changes to a relatively shaky and
not-so-testable codebase, so I chose to do the absolute minimum
required: change initialization values for a couple of String
fields
in a class, compile the changed class, and run the code manually
to
observe that I had achieved the desired change.
...

You have my sympathy for your frustration, but I would call this
story "The curse of manual builds".

Back in the 1970's, long before Java was invented, I worked on a
large project that avoided reassembling all the code by having a
team of programmer techs who constructed the list of modules that
needed to be assembled due to each change. The programmer techs did
their best, but it was a *very* error prone procedure.

Since then, I have seen complaints, for several languages, of the
form "I did a manual build, and got into trouble due to [some
aspect
of the programming language or environment]".

I am convinced that the only viable choices are rebuilding
everything
for every change or using language-appropriate automatic dependency
analysis.

And tools like ant make it easy.

Easy to get it wrong, I think you mean. Ant's dependency analysis
amounts to:

Compile A.java if A.class is either absent or older than A.java.

Ant will not recompile A.java on the grounds that it uses contants
from class B and B.java is newer than A.class. When accepting a set
of changes from elsewhere (say, when updating from an SCM system), the
only safe procedure is to do a full, clean rebuild.
 
W

Wesley MacIntosh

Mike said:
Ant will not recompile A.java on the grounds that it uses contants
from class B and B.java is newer than A.class. When accepting a set
of changes from elsewhere (say, when updating from an SCM system), the
only safe procedure is to do a full, clean rebuild.

Shouldn't running the ant build twice suffice?

First time, all the .java files that define constants that have changed
will get recompiled.

Second time, all the .java files that use constants that have changed,
and weren't recompiled in the first pass, will get recompiled because
they now have a dependency whose .class is newer.

Ex:

A.java uses constants in B.java last mod Dec 12 A.class last mod Dec 12
B.java last mod Dec 13 B.class last mod Dec 12
C.java uses constants in B.java last mod Dec 12 C.class last mod Dec 12

Pass 1:

A.class younger than A.java and dependency B.class, leave A.java alone.
B.class older than B.java. Recompile B.java. B.class now modified Dec 13
C.class younger than C.java but older than B.class. Recompile C.java.

A.java uses constants in B.java last mod Dec 12 A.class last mod Dec 12
B.java last mod Dec 13 B.class last mod Dec 13
C.java uses constants in B.java last mod Dec 12 C.class last mod Dec 13

Pass 2:

A.class younger than A.java but older than B.class. Recompile A.java.
B.class younger than B.java. Leave B.java alone.
C.class younger than C.java and dependency B.class. Leave C.java alone.

A.java uses constants in B.java last mod Dec 12 A.class last mod Dec 13
B.java last mod Dec 13 B.class last mod Dec 13
C.java uses constants in B.java last mod Dec 12 C.class last mod Dec 13

A and C should both be up to date after the second pass, although A
wasn't after the first pass.

Or does ant get things wrong in a more complicated way than you
originally stated?
 
M

Mark Thornton

Wesley said:
Shouldn't running the ant build twice suffice?

No because class files that logically depend on a constant defined
elsewhere do not include any reference to that dependency. To do this
dependency analysis reliably you have to analyse the source yourself
with a tool other than javac. You can not do it just from the .class files.

Mark Thornton
 
A

Arne Vajhøj

Mike said:
Arne said:
Patricia said:
Juha Laiho wrote:
I got recently bitten by something I consider to be a rather nasty
feature in the Java language, and decided to write up about it.

I had to do a couple of simple changes to a relatively shaky and
not-so-testable codebase, so I chose to do the absolute minimum
required: change initialization values for a couple of String
fields
in a class, compile the changed class, and run the code manually
to
observe that I had achieved the desired change.
...

You have my sympathy for your frustration, but I would call this
story "The curse of manual builds".

Back in the 1970's, long before Java was invented, I worked on a
large project that avoided reassembling all the code by having a
team of programmer techs who constructed the list of modules that
needed to be assembled due to each change. The programmer techs did
their best, but it was a *very* error prone procedure.

Since then, I have seen complaints, for several languages, of the
form "I did a manual build, and got into trouble due to [some
aspect
of the programming language or environment]".

I am convinced that the only viable choices are rebuilding
everything
for every change or using language-appropriate automatic dependency
analysis.
And tools like ant make it easy.

Easy to get it wrong, I think you mean. Ant's dependency analysis
amounts to:

Compile A.java if A.class is either absent or older than A.java.

Ant will not recompile A.java on the grounds that it uses contants
from class B and B.java is newer than A.class. When accepting a set
of changes from elsewhere (say, when updating from an SCM system), the
only safe procedure is to do a full, clean rebuild.

I meant that ant make rebuilding everything easy.

Clean all dirs, extract from VCS, compile src->build, pack jars.

Arne
 
A

Arne Vajhøj

Wesley said:
Shouldn't running the ant build twice suffice?

First time, all the .java files that define constants that have changed
will get recompiled.

Second time, all the .java files that use constants that have changed,
and weren't recompiled in the first pass, will get recompiled because
they now have a dependency whose .class is newer.

Ex:

A.java uses constants in B.java last mod Dec 12 A.class last mod Dec 12
B.java last mod Dec 13 B.class last mod Dec 12
C.java uses constants in B.java last mod Dec 12 C.class last mod Dec 12

Pass 1:

A.class younger than A.java and dependency B.class, leave A.java alone.
B.class older than B.java. Recompile B.java. B.class now modified Dec 13
C.class younger than C.java but older than B.class. Recompile C.java.

A.java uses constants in B.java last mod Dec 12 A.class last mod Dec 12
B.java last mod Dec 13 B.class last mod Dec 13
C.java uses constants in B.java last mod Dec 12 C.class last mod Dec 13

Pass 2:

A.class younger than A.java but older than B.class. Recompile A.java.
B.class younger than B.java. Leave B.java alone.
C.class younger than C.java and dependency B.class. Leave C.java alone.

A.java uses constants in B.java last mod Dec 12 A.class last mod Dec 13
B.java last mod Dec 13 B.class last mod Dec 13
C.java uses constants in B.java last mod Dec 12 C.class last mod Dec 13

A and C should both be up to date after the second pass, although A
wasn't after the first pass.

Or does ant get things wrong in a more complicated way than you
originally stated?

His statement:

and yours:
> A.class younger than A.java but older than B.class. Recompile A.java.

are different.

Arne
 
M

Mike Schilling

Arne said:
Mike said:
Arne said:
Patricia Shanahan wrote:
Juha Laiho wrote:
I got recently bitten by something I consider to be a rather
nasty
feature in the Java language, and decided to write up about it.

I had to do a couple of simple changes to a relatively shaky and
not-so-testable codebase, so I chose to do the absolute minimum
required: change initialization values for a couple of String
fields
in a class, compile the changed class, and run the code manually
to
observe that I had achieved the desired change.
...

You have my sympathy for your frustration, but I would call this
story "The curse of manual builds".

Back in the 1970's, long before Java was invented, I worked on a
large project that avoided reassembling all the code by having a
team of programmer techs who constructed the list of modules that
needed to be assembled due to each change. The programmer techs
did
their best, but it was a *very* error prone procedure.

Since then, I have seen complaints, for several languages, of the
form "I did a manual build, and got into trouble due to [some
aspect
of the programming language or environment]".

I am convinced that the only viable choices are rebuilding
everything
for every change or using language-appropriate automatic
dependency
analysis.
And tools like ant make it easy.

Easy to get it wrong, I think you mean. Ant's dependency analysis
amounts to:

Compile A.java if A.class is either absent or older than A.java.

Ant will not recompile A.java on the grounds that it uses contants
from class B and B.java is newer than A.class. When accepting a
set
of changes from elsewhere (say, when updating from an SCM system),
the only safe procedure is to do a full, clean rebuild.

I meant that ant make rebuilding everything easy.

Clean all dirs, extract from VCS, compile src->build, pack jars.

That's quite true, though of course it would be true of any scripting
language. Back in C and C++ days, there were fairly simple tools to
optimize builds: makemake scripts that analyzed source code to build
dependency lists and generate make scripts, so you really could do a
build that was both safe and minimal. Java (in particular, the fact
that a .java file is the equivalent of both .c and .h files) makes
this astonishingly difficult.
 
A

Arne Vajhøj

Mike said:
Arne said:
Mike said:
Arne Vajhøj wrote:
Patricia Shanahan wrote:
Juha Laiho wrote:
I got recently bitten by something I consider to be a rather
nasty
feature in the Java language, and decided to write up about it.

I had to do a couple of simple changes to a relatively shaky and
not-so-testable codebase, so I chose to do the absolute minimum
required: change initialization values for a couple of String
fields
in a class, compile the changed class, and run the code manually
to
observe that I had achieved the desired change.
...

You have my sympathy for your frustration, but I would call this
story "The curse of manual builds".

Back in the 1970's, long before Java was invented, I worked on a
large project that avoided reassembling all the code by having a
team of programmer techs who constructed the list of modules that
needed to be assembled due to each change. The programmer techs
did
their best, but it was a *very* error prone procedure.

Since then, I have seen complaints, for several languages, of the
form "I did a manual build, and got into trouble due to [some
aspect
of the programming language or environment]".

I am convinced that the only viable choices are rebuilding
everything
for every change or using language-appropriate automatic
dependency
analysis.
And tools like ant make it easy.
Easy to get it wrong, I think you mean. Ant's dependency analysis
amounts to:

Compile A.java if A.class is either absent or older than A.java.

Ant will not recompile A.java on the grounds that it uses contants
from class B and B.java is newer than A.class. When accepting a
set
of changes from elsewhere (say, when updating from an SCM system),
the only safe procedure is to do a full, clean rebuild.
I meant that ant make rebuilding everything easy.

Clean all dirs, extract from VCS, compile src->build, pack jars.

That's quite true, though of course it would be true of any scripting
language. Back in C and C++ days, there were fairly simple tools to
optimize builds: makemake scripts that analyzed source code to build
dependency lists and generate make scripts, so you really could do a
build that was both safe and minimal. Java (in particular, the fact
that a .java file is the equivalent of both .c and .h files) makes
this astonishingly difficult.

Huge C/C++ build often took a long time, because of the include thing,
optimizing by the compiler and because systems were slower back then.

Java doesn't use include, javac doesn't optimize and the systems are
fast.

I don't see full rebuild as a problem.

Arne
 
M

Martin Gregorie

Mike said:
Arne said:
Mike Schilling wrote:
Arne Vajhøj wrote:
Patricia Shanahan wrote:
Juha Laiho wrote:
I got recently bitten by something I consider to be a rather nasty
feature in the Java language, and decided to write up about it.

I had to do a couple of simple changes to a relatively shaky and
not-so-testable codebase, so I chose to do the absolute minimum
required: change initialization values for a couple of String
fields
in a class, compile the changed class, and run the code manually
to
observe that I had achieved the desired change.
...

You have my sympathy for your frustration, but I would call this
story "The curse of manual builds".

Back in the 1970's, long before Java was invented, I worked on a
large project that avoided reassembling all the code by having a
team of programmer techs who constructed the list of modules that
needed to be assembled due to each change. The programmer techs did
their best, but it was a *very* error prone procedure.

Since then, I have seen complaints, for several languages, of the
form "I did a manual build, and got into trouble due to [some
aspect
of the programming language or environment]".

I am convinced that the only viable choices are rebuilding
everything
for every change or using language-appropriate automatic dependency
analysis.
And tools like ant make it easy.
Easy to get it wrong, I think you mean. Ant's dependency analysis
amounts to:

Compile A.java if A.class is either absent or older than A.java.

Ant will not recompile A.java on the grounds that it uses contants
from class B and B.java is newer than A.class. When accepting a set
of changes from elsewhere (say, when updating from an SCM system),
the only safe procedure is to do a full, clean rebuild.
I meant that ant make rebuilding everything easy.

Clean all dirs, extract from VCS, compile src->build, pack jars.

That's quite true, though of course it would be true of any scripting
language. Back in C and C++ days, there were fairly simple tools to
optimize builds: makemake scripts that analyzed source code to build
dependency lists and generate make scripts, so you really could do a
build that was both safe and minimal. Java (in particular, the fact
that a .java file is the equivalent of both .c and .h files) makes this
astonishingly difficult.

Huge C/C++ build often took a long time, because of the include thing,
optimizing by the compiler and because systems were slower back then.

Java doesn't use include, javac doesn't optimize and the systems are
fast.
'make' doesn't take kindly to circular references, while Java often seems
to end up with something remarkably similar - is this why javac seems to
prefer simultaneous compiles of all source files in a package?

I've tried to use 'make' for Java but it doesn't take kindly to Java. I
ended up leaving dependencies out of the makefile to stop 'make'
complaining about circular references. This in turn meant that the only
way I could guarantee a complete, clean build was to force a full build
by deleting the class files. Once I hit this situation I stopped trying
to use 'make' for Java and switched to ant: in any case even a full ant
build is faster than a 'make' build thanks to ant taking advantage of
javac's parallel compile capability, but I still periodically find I have
to force a complete build.

'make' only works well for C compilation and other tasks where its
possible to define a straight dependency tree with no crosslinks.
I don't see full rebuild as a problem.
Agreed, but its a bit irritating. Do you preempt it by writing the ant
script to delete all class files before it runs javac?
 
A

Arne Vajhøj

Martin said:
'make' doesn't take kindly to circular references, while Java often seems
to end up with something remarkably similar - is this why javac seems to
prefer simultaneous compiles of all source files in a package?

I think so.
I've tried to use 'make' for Java but it doesn't take kindly to Java. I
ended up leaving dependencies out of the makefile to stop 'make'
complaining about circular references. This in turn meant that the only
way I could guarantee a complete, clean build was to force a full build
by deleting the class files. Once I hit this situation I stopped trying
to use 'make' for Java and switched to ant: in any case even a full ant
build is faster than a 'make' build thanks to ant taking advantage of
javac's parallel compile capability, but I still periodically find I have
to force a complete build.

It is usually best to use the build tool matching the technology. Ant or
maven for Java. Make for C/C++. Nant or MSBuild for .NET.
Agreed, but its a bit irritating. Do you preempt it by writing the ant
script to delete all class files before it runs javac?

I believe it is very common to have a clean target.

Arne
 
T

Tom Anderson

Back in C and C++ days, there were fairly simple tools to optimize
builds: makemake scripts that analyzed source code to build dependency
lists and generate make scripts, so you really could do a build that was
both safe and minimal. Java (in particular, the fact that a .java file
is the equivalent of both .c and .h files) makes this astonishingly
difficult.

Really?

I was thinking about this a while ago, and it seemed to me that the
dependencies a class had were moderately simple - every other class it
references directly, and every class which is a superclass of a class it
references. Am i missing something?

Further (and i'm rambling now), in all cases (i *think*), a class only
depends on the interface of the other class, so if you can work out if a
class's interface has changed (by comparing the fresh class file with the
old one, for instance), you can work out if a change is one that means
dependents have to be recompiled. If you also captured information about
whether the dependency was on public, package-access or protected
features, you could possibly also prune your list of recompilations
further. In the limit, you could track dependencies on individual
features, and be very selective about what you have to recompile. Although
there are gotchas around inheritance - eg adding a new protected final
method to a class breaks subclasses which define a method of the same
name.

Eclipse has an incremental compiler, which has been released as a
standalone tool. I wonder if this might be a solution, or the foundation
for a solution, to this problem.

tom
 
M

Martin Gregorie

I believe it is very common to have a clean target.
Yes, and I do have one. Of course.

What I was wondering was if anybody has ever found it beneficial to chain
a step to delete all class files before the compile step. This could be
because the structure of the class hierarchy, due to externally visible
constants or whatever, was complex and/or tangled enough to need a full
compile more often than not.
 
A

Arne Vajhøj

Tom said:
Really?

I was thinking about this a while ago, and it seemed to me that the
dependencies a class had were moderately simple - every other class it
references directly, and every class which is a superclass of a class it
references. Am i missing something?

Further (and i'm rambling now), in all cases (i *think*), a class only
depends on the interface of the other class, so if you can work out if a
class's interface has changed (by comparing the fresh class file with
the old one, for instance), you can work out if a change is one that
means dependents have to be recompiled. If you also captured information
about whether the dependency was on public, package-access or protected
features, you could possibly also prune your list of recompilations
further. In the limit, you could track dependencies on individual
features, and be very selective about what you have to recompile.
Although there are gotchas around inheritance - eg adding a new
protected final method to a class breaks subclasses which define a
method of the same name.

I don't think it is simple.

As far as I can see, then:
* it is not possible only using class files - source code analysis is
needed
* it is not possible only with current code - both current and previous
code is necessary

Classpath for source code is not a well defined concept.

And a database with info for different versions adds work.

Arne
 
A

Arne Vajhøj

Martin said:
Yes, and I do have one. Of course.

What I was wondering was if anybody has ever found it beneficial to chain
a step to delete all class files before the compile step. This could be
because the structure of the class hierarchy, due to externally visible
constants or whatever, was complex and/or tangled enough to need a full
compile more often than not.

I would expect almost everyone to do clean before an official
build.

It is only with developer builds that safe versus quick
is an issue.

I must admit that in development I don't always clean as
part of build.

Arne
 
M

Mike Schilling

Arne said:
I would expect almost everyone to do clean before an official
build.

Me too. We run build/test cycles nightly, and they always starts
with:

Clean the world
Update all the source from the SCM system
Build (etc.)

The clean build eats up only machine resources. Figuring out whether
a failure is a real failure or an anomaly that would be fixed by a
clean build uses far more valuable human time.
It is only with developer builds that safe versus quick
is an issue.

I must admit that in development I don't always clean as
part of build.

Me too, though usually if I don't clean, it's because I made all the
changes myself and know that a non-clean build is safe. (Or think I
know it, anyway).
 
M

Mike Schilling

Tom said:
I was thinking about this a while ago, and it seemed to me that the
dependencies a class had were moderately simple - every other class
it
references directly, and every class which is a superclass of a
class
it references. Am i missing something?

Further (and i'm rambling now), in all cases (i *think*), a class
only
depends on the interface of the other class,

And the values of their constants, which is where we came in.
so if you can work out
if a class's interface has changed (by comparing the fresh class
file
with the old one, for instance), you can work out if a change is one
that means dependents have to be recompiled.

And if there were any changes to that class's ancestor classes, or any
interfaces it implements (directly or indirectly.)
If you also captured
information about whether the dependency was on public,
package-access or protected features, you could possibly also prune
your list of recompilations further. In the limit, you could track
dependencies on individual features, and be very selective about
what
you have to recompile. Although there are gotchas around inheritance
- eg adding a new protected final method to a class breaks
subclasses
which define a method of the same name.

All of this is feasible, but it's complicated. Once you've figured
out in detail all of the possible sorts of changes you'd like to track
(and that's complex in itself), you've now got to make the right
tradeoff between. And you'd like to build this all into the compiler,
rather than having to run successive compilation passes, as each one
invalidates a new set of classes.

1. Keeping very detailed dependencies, which minimizes the number of
recompilations, but explodes the size of your dependency database, and
2. Keeping coarser dependencies, which minimized the size of your
dependency database, but increases the number of recompilations

Doing comparisons between file dates is a lot simpler.
 
M

Mike Schilling

Arne said:
I don't think it is simple.

As far as I can see, then:
* it is not possible only using class files - source code analysis
is
needed

I think everything you need is in the class file. What's missing?
* it is not possible only with current code - both current and
previous code is necessary

Yes, you'd need a database of "stuff I saw last time I compiled
X.java". (And, of course, if that's missing, you'd compile X.java
regardless.)
 
A

Arne Vajhøj

Mike said:
I think everything you need is in the class file. What's missing?

What this thread started with.

If the class file contains 123, then you don't know that it really
was mypack.MyClass.ConstX ...
Yes, you'd need a database of "stuff I saw last time I compiled
X.java". (And, of course, if that's missing, you'd compile X.java
regardless.)

Arne
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,743
Messages
2,569,478
Members
44,898
Latest member
BlairH7607

Latest Threads

Top