dependency scanner

A

Aryeh M. Friedman

I am looking for a class/program that will scan a .java file for imports
(recursively) to discover which ones need to be rebuilt. This is
close to GNU's makedepend or cook's c_incl.
 
A

Arne Vajhøj

Aryeh said:
I am looking for a class/program that will scan a .java file for imports
(recursively) to discover which ones need to be rebuilt. This is close
to GNU's makedepend or cook's c_incl.

You don't use that type of tools with Java.

The javac compiler itself compiles missing parts.

And the ant tool (which you should use for build) checks what to
rebuild based on dates similar to ant.

Just use ant and javac and forget about dependencies.

If you use maven for build it will even be able to download
external packages you need for you.

Arne
 
M

Mike Schilling

Arne said:
You don't use that type of tools with Java.

The javac compiler itself compiles missing parts.

And the ant tool (which you should use for build) checks what to
rebuild based on dates similar to ant.

Just use ant and javac and forget about dependencies.

And when that's not good enough (like when superclass change, require
subclasses to recompile too), curse a bit, delete all the .class
files, and rebuild the whole thing.
 
A

Arne Vajhøj

Mike said:
And when that's not good enough (like when superclass change, require
subclasses to recompile too), curse a bit, delete all the .class
files, and rebuild the whole thing.

Many people make a clean target in build.xml !

Arne
 
J

jolz

The javac compiler itself compiles missing parts.
And the ant tool (which you should use for build) checks what to
rebuild based on dates similar to ant.

Just use ant and javac and forget about dependencies.

Compiling based on dates isn't enough. The most anoying problem (but not
the only one) during compilation is inlining constants. The only way to
be 100% sure of proper compilation is to allways recompile all sources.
Fortunatelly recompilation based on dates works often enought to make
ant/maven quite usefull tools.
 
M

Mike Schilling

Arne said:
Many people make a clean target in build.xml !

I always do, for several reasons; the above is one of them. My
comment was a complaint about the fact that the <javac> task is
*almost* good enough to handle dependencies, but not quite. When I,
for instance, get a batch of newer source code from the SCM system, I
always do a clean build, since there's no way to be sure it isn't
necessary, and I don't want to waste time with spurious problems from
not building clean when it was required.
 
J

jolz

I am looking for a class/program that will scan a .java file for imports
(recursively) to discover which ones need to be rebuilt. This is close
to GNU's makedepend or cook's c_incl.

You didn't consider import x.y.* and classes in the same package. You'll
have to parse the whole source to do this properly (the most popular
tool is javacc) - it may not be much faster that simply recompiling
those sources.
 
A

Arne Vajhøj

Mike said:
I always do, for several reasons; the above is one of them. My
comment was a complaint about the fact that the <javac> task is
*almost* good enough to handle dependencies, but not quite. When I,
for instance, get a batch of newer source code from the SCM system, I
always do a clean build, since there's no way to be sure it isn't
necessary, and I don't want to waste time with spurious problems from
not building clean when it was required.

And usually Java builds are reasonable fast even when building from
scratch.

Arne
 
M

Mike Schilling

Lew said:
Given that 'import' is just syntactic sugar for FQNs, any dependency
analysis tool would only use FQNs for its analysis, I should think.

Certainly. But the OP was suggesting a tool that scanned the .java
file lexically for imports, and that would fail to find

p1.p2.p3.class1 c = null;

Personally, I'd build a dependency tool that scanned the .class file,
not the .java file. (If there is no .class file, you don't need to
know anything about dependencies to realize that the .java file needs
to be recompiled.) The bytecode has everything you need, and there is
lots of open-source code out there to help you parse it.
 
A

Aryeh M. Friedman

Lew said:
Doesn't every existing tool that does dependency scanning work off the
.class files?

It escaped me that the question restricted the scan to source files. I
simply figured any dependency tool would simply have to work off the
.class files, never even dreaming that anyone would try to do it from
the source.

Isn't this a catch-22 though in that if you are in a mixed lang
enviroment and either need the java compiled first (in the case of it
being the backend) or the other lang being done first (JNI) and it is
being compiled by someone who has not developed it then it then follows
there are no .class files. Why did sun get rid of -depend on javac
anyways? I also found JavaDepend
(http://ptolemy.eecs.berkeley.edu/~cxh/java/javadepend/) but it doesn't
work on anything newer then 1.1 and I am doing 1.5 projects.

Also in an other post in the thread someone mentioned if a compile
doesn't work with say javac `find . -name '*.java'` or javac *.java then
it is time to break it apart into standalone jar's. This too raises a
depend issue in that if oyu have several packages which ones need to be
recompiled when X changes.

In short I am *NOT IMPRESSED* with Java's ability to manage large
projects, In a way this is the same issue recursive make has
http://aegis.sourceforge.net/auug97.pdf
 
R

Roedy Green

I am looking for a class/program that will scan a .java file for imports

Imports are not sufficient since you can have fully qualified names in
the source text.

Further imports such as import javax.swing.* brings in much more than
you would actually need.

If you scanned the class files, you would miss new classes referenced
since the last compile.

So that means you have to scan source and parse it. JavaC can
probably do that faster than you can.

I think you are better off to:

1. use ANT so Javac does not need to reloaded for multiple compiles.

2. just feel the whole shebang on the command line and let JavaC
figure out what needs to be recompiled.

3. from time to time do a clean compile (erase all class files), and
certainly before any release.
 
A

Aryeh M. Friedman

Lew said:
Funny, I don't remember anyone claiming that Java is a
project-management system. I thought it was a programming language. Do
you blame C for not having the features of make, I wonder?

C does at least has a trivial way to find depends (#include)
Use Ant, of course. Maven for things Ant can't.

Neither seems to be able to manage non-Java projects and if one wants to
have the project a part of a OS dist (or ports collection) then almost
all of them only accept make compatible build scripts. An other
requirement I have is the ability to stich the source tree together from
disjoint dirs (i.e. some of it is in a repo (the unmodified code) and
some is in a local dir (the modified code)). The only dependency
management tool I have ever found that manages this in anything like a
reasonable manner is cook
(http://miller.emu.id.au/pmiller/software/cook/) probally because it was
written by the same person that wrote my SCM tool (aegis.sf.net). The
only other alternative I know of is something like unionfs which is
horrible. Oh forgot to mention cook also has an builtin Makefile
generator in it so the OS ports/dist is happy.
 
A

Aryeh M. Friedman

Lew said:
Huh? What does #include have to do with managing object file dependencies?


As with any type of build procedure there are implicit and explicit
depends that make/cook/whatever needs to handle... Explicit ones are
the ones you are thinking of when you are thinking that foo.c depends on
ack.c because it uses bar() from ack.c... *BUT* since ANSI (and almost
all compilors accept ANSI by default now) requires a prototype (even if
it returns an int or void) this means I need to define the prototype for
bar() somewhere and more often then not you do that in ack.h... now in
foo.c I have:

#include "ack.h"

Assuming that I use the same name for the .h and .c it goes to I only
need to scan for #include's in foo.c and do a batch processed search and
replace (say with sed in unix) and replace all .h with .c (or actually
..o for techinical reasons)... this now gives me a list of all my
implicit depends (due to the need to prototype about 95% of the total
for a typical project)... All I need to do in the
makefile/cookbook/whatever is add a rule like this (cook has nicer
syntax then make so I will use it):

%0%.o: %0%.c
/*in reality I would also add a call to the dep scanner above*/
{
gcc [command line opts] -c %0%.c -o [target];
}

You should be asking now how to handle explicit depends... thats when
you actually hard/softcode it into the makefile/cookbook/whatever with
something like:

bin/foo: lib/liback.a
{
gcc -Llib foo.c -o [target];
}

lib/liback.a: lib/ack.o lib/fred.o ....
{
ar rv [target] ack.o fred.o ...
}

Almost all these can be soft coded by various tricks with a good depend
scanner for example to get the "ack.o fred.o ..." list I would just say
every .c in lib/ needs to become a .o.


Now cook does something really smart at this point compared to make. I
creates the entire DAG *BEFORE* attempting to build any target thus
allows you to avoid all the issues in RMCF.

Due to this premade DAG approach I can use the same cookbook to build
and package multiple packages (using the java term) *BUT* only when
something in them or a package they depend on changes.

The reason for the orginal question is I do not want to have to use
nothing but explicit depends and if I am writting this for public
release the person who DL's the source code will not have any .class
files yet.
 
M

Mike Schilling

Aryeh said:
Neither seems to be able to manage non-Java projects and if one wants
to have the project a part of a OS dist (or ports collection) then
almost all of them only accept make compatible build scripts.

Ant can run any java program or native exceutable you like. We use it to
build a combined Java and C# system. In fact, the standard Ant distribution
includes tasks to build .NET programs, though we've had to modify them for
our requirements.
 
M

Mike Schilling

Lew said:
Doesn't every existing tool that does dependency scanning work off
the .class files?

It escaped me that the question restricted the scan to source files. I
simply figured any dependency tool would simply have to work off
the .class files, never even dreaming that anyone would try to do it
from the source.

There are more things in Heaven and Earth, Horatio, than are dreamt of in
your philosphy. :)
 
C

Christopher Brooks

I manage the page with javadepend, which has not been updated in
years. I stashed a copy of javadepend there mainly to preserve
history.

We had a similar problem where we needed to find dependencies and
someone use Ivy, which seemed to work reasonably well.

http://java-source.net/open-source/build-systems/ivy

Yes, it is always possible to miss classes because of reflection, but
Ivy gave us a leg up on analyzing dependencies. If you really want to
know what packages are used, do treeshaking by running java -verbose
and exercise the program. We use this to quite effectively for
shipping small jars. However, it is difficult to do treeshaking
automatically and get a complete set of jars unless you effectively
exercise the program.

Also, the Eclipse Metrics package has a dependency visualizer that
might be of interest. I looked at it briefly this week.

About the ant/make thing. After using both and shipping both, I've
come to the conclusion that that Ant and Make to be pretty much
functionally equivalent. One can argue about the details though.

To ship software, what I need is the following:
1. Before compilation, some checks and searches need to happen.
Checks for the right version of the compiler etc., searches for
extensions. Historically, this is done with configure, though ant
can do more of this than it used to.
Unfortunately, portions of my user community does not want to run
configure before starting Eclipse. Nor do they want to run an
ant task from within Eclipse.
2. During compilation, packages that won't compile need to be skipped.
Many years ago, ant could not handle dependencies, now it can.
Also, many years ago, the ant page said it was written by people
who did not understand make. I've seen this issue before when
people complain about make and then start writing their own system.
3. At release time, each package needs to have some sort of manifest
that lists files and subdirectories to be shipped. Currently, the
manifest for Ptolemy is in makefiles, but it could be in separate text
files, which could be read by recursive ant files.
4. At user run time, I'd like to provide modules and provided updates
similar to Eclipse/Equinox and the oSGI work. This requires dependency
analysis based on versions, including supporting modules that
themselves might require different versions of _the_same_ jar file.
This is difficult, and requires classpath support.


The manifest files are what refute some of the arguments put forth in
"recursive make considered harmful."

Ant and Eclipse's "Just compile everything" strategy does not work for
research projects where some packages in a tree will not compile
unless missing dependencies are present.

Thus, I'm interested in Maven, but really looking for something that
handles complex configuration and package dependencies.

Personally, I'm pretty happy with the package system, it does make
modularizing large projects (>300k LOC) possible. However, more
work needs to be done concerning supporting multiple projects.

_Christopher


Aryeh M. Friedman said:
Isn't this a catch-22 though in that if you are in a mixed lang
enviroment and either need the java compiled first (in the case of it
being the backend) or the other lang being done first (JNI) and it is
being compiled by someone who has not developed it then it then
follows there are no .class files. Why did sun get rid of -depend on
javac anyways? I also found JavaDepend
(http://ptolemy.eecs.berkeley.edu/~cxh/java/javadepend/) but it
doesn't work on anything newer then 1.1 and I am doing 1.5 projects.

Also in an other post in the thread someone mentioned if a compile
doesn't work with say javac `find . -name '*.java'` or javac *.java
then it is time to break it apart into standalone jar's. This too
raises a depend issue in that if oyu have several packages which ones
need to be recompiled when X changes.

In short I am *NOT IMPRESSED* with Java's ability to manage large
projects, In a way this is the same issue recursive make has
http://aegis.sourceforge.net/auug97.pdf

--
Christopher Brooks (cxh at eecs berkeley edu) University of California
Chess Executive Director US Mail: 337 Cory Hall #1774
Programmer/Analyst Chess/Ptolemy/Trust Berkeley, CA 94720-1774
ph: 510.643.9841 fax:510.642.2718 (office: 400A Cory)
home: (F-Tu) 707.665.0131 (W-F) 510.655.5480
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top