dependency scanner

Discussion in 'Java' started by Aryeh M. Friedman, May 11, 2008.

  1. I am looking for a class/program that will scan a .java file for imports
    (recursively) to discover which ones need to be rebuilt. This is
    close to GNU's makedepend or cook's c_incl.
     
    Aryeh M. Friedman, May 11, 2008
    #1
    1. Advertising

  2. Aryeh M. Friedman

    Stefan Ram Guest

    Stefan Ram, May 11, 2008
    #2
    1. Advertising

  3. Aryeh M. Friedman

    Arne Vajhøj Guest

    Aryeh M. Friedman wrote:
    > I am looking for a class/program that will scan a .java file for imports
    > (recursively) to discover which ones need to be rebuilt. This is close
    > to GNU's makedepend or cook's c_incl.


    You don't use that type of tools with Java.

    The javac compiler itself compiles missing parts.

    And the ant tool (which you should use for build) checks what to
    rebuild based on dates similar to ant.

    Just use ant and javac and forget about dependencies.

    If you use maven for build it will even be able to download
    external packages you need for you.

    Arne
     
    Arne Vajhøj, May 11, 2008
    #3
  4. Arne Vajhøj wrote:
    > Aryeh M. Friedman wrote:
    >> I am looking for a class/program that will scan a .java file for
    >> imports (recursively) to discover which ones need to be rebuilt.
    >> This is close to GNU's makedepend or cook's c_incl.

    >
    > You don't use that type of tools with Java.
    >
    > The javac compiler itself compiles missing parts.
    >
    > And the ant tool (which you should use for build) checks what to
    > rebuild based on dates similar to ant.
    >
    > Just use ant and javac and forget about dependencies.


    And when that's not good enough (like when superclass change, require
    subclasses to recompile too), curse a bit, delete all the .class
    files, and rebuild the whole thing.
     
    Mike Schilling, May 11, 2008
    #4
  5. Aryeh M. Friedman

    Arne Vajhøj Guest

    Mike Schilling wrote:
    > Arne Vajhøj wrote:
    >> Aryeh M. Friedman wrote:
    >>> I am looking for a class/program that will scan a .java file for
    >>> imports (recursively) to discover which ones need to be rebuilt.
    >>> This is close to GNU's makedepend or cook's c_incl.

    >> You don't use that type of tools with Java.
    >>
    >> The javac compiler itself compiles missing parts.
    >>
    >> And the ant tool (which you should use for build) checks what to
    >> rebuild based on dates similar to ant.
    >>
    >> Just use ant and javac and forget about dependencies.

    >
    > And when that's not good enough (like when superclass change, require
    > subclasses to recompile too), curse a bit, delete all the .class
    > files, and rebuild the whole thing.


    Many people make a clean target in build.xml !

    Arne
     
    Arne Vajhøj, May 11, 2008
    #5
  6. Aryeh M. Friedman

    jolz Guest

    > The javac compiler itself compiles missing parts.
    >
    > And the ant tool (which you should use for build) checks what to
    > rebuild based on dates similar to ant.
    >
    > Just use ant and javac and forget about dependencies.


    Compiling based on dates isn't enough. The most anoying problem (but not
    the only one) during compilation is inlining constants. The only way to
    be 100% sure of proper compilation is to allways recompile all sources.
    Fortunatelly recompilation based on dates works often enought to make
    ant/maven quite usefull tools.
     
    jolz, May 11, 2008
    #6
  7. Arne Vajhøj wrote:
    > Mike Schilling wrote:
    >> Arne Vajhøj wrote:
    >>> Aryeh M. Friedman wrote:
    >>>> I am looking for a class/program that will scan a .java file for
    >>>> imports (recursively) to discover which ones need to be rebuilt.
    >>>> This is close to GNU's makedepend or cook's c_incl.
    >>> You don't use that type of tools with Java.
    >>>
    >>> The javac compiler itself compiles missing parts.
    >>>
    >>> And the ant tool (which you should use for build) checks what to
    >>> rebuild based on dates similar to ant.
    >>>
    >>> Just use ant and javac and forget about dependencies.

    >>
    >> And when that's not good enough (like when superclass change,
    >> require
    >> subclasses to recompile too), curse a bit, delete all the .class
    >> files, and rebuild the whole thing.

    >
    > Many people make a clean target in build.xml !


    I always do, for several reasons; the above is one of them. My
    comment was a complaint about the fact that the <javac> task is
    *almost* good enough to handle dependencies, but not quite. When I,
    for instance, get a batch of newer source code from the SCM system, I
    always do a clean build, since there's no way to be sure it isn't
    necessary, and I don't want to waste time with spurious problems from
    not building clean when it was required.
     
    Mike Schilling, May 11, 2008
    #7
  8. Aryeh M. Friedman

    jolz Guest

    > I am looking for a class/program that will scan a .java file for imports
    > (recursively) to discover which ones need to be rebuilt. This is close
    > to GNU's makedepend or cook's c_incl.


    You didn't consider import x.y.* and classes in the same package. You'll
    have to parse the whole source to do this properly (the most popular
    tool is javacc) - it may not be much faster that simply recompiling
    those sources.
     
    jolz, May 11, 2008
    #8
  9. jolz wrote:
    >> I am looking for a class/program that will scan a .java file for
    >> imports (recursively) to discover which ones need to be rebuilt.
    >> This is close to GNU's makedepend or cook's c_incl.

    >
    > You didn't consider import x.y.* and classes in the same package.


    And the use of fully qualified named with no import.
     
    Mike Schilling, May 11, 2008
    #9
  10. Aryeh M. Friedman

    Arne Vajhøj Guest

    Mike Schilling wrote:
    > Arne Vajhøj wrote:
    >> Mike Schilling wrote:
    >>> Arne Vajhøj wrote:
    >>>> Aryeh M. Friedman wrote:
    >>>>> I am looking for a class/program that will scan a .java file for
    >>>>> imports (recursively) to discover which ones need to be rebuilt.
    >>>>> This is close to GNU's makedepend or cook's c_incl.
    >>>> You don't use that type of tools with Java.
    >>>>
    >>>> The javac compiler itself compiles missing parts.
    >>>>
    >>>> And the ant tool (which you should use for build) checks what to
    >>>> rebuild based on dates similar to ant.
    >>>>
    >>>> Just use ant and javac and forget about dependencies.
    >>> And when that's not good enough (like when superclass change,
    >>> require
    >>> subclasses to recompile too), curse a bit, delete all the .class
    >>> files, and rebuild the whole thing.

    >> Many people make a clean target in build.xml !

    >
    > I always do, for several reasons; the above is one of them. My
    > comment was a complaint about the fact that the <javac> task is
    > *almost* good enough to handle dependencies, but not quite. When I,
    > for instance, get a batch of newer source code from the SCM system, I
    > always do a clean build, since there's no way to be sure it isn't
    > necessary, and I don't want to waste time with spurious problems from
    > not building clean when it was required.


    And usually Java builds are reasonable fast even when building from
    scratch.

    Arne
     
    Arne Vajhøj, May 11, 2008
    #10
  11. Lew wrote:
    > Mike Schilling wrote:
    >> jolz wrote:
    >>>> I am looking for a class/program that will scan a .java file for
    >>>> imports (recursively) to discover which ones need to be rebuilt.
    >>>> This is close to GNU's makedepend or cook's c_incl.
    >>> You didn't consider import x.y.* and classes in the same package.

    >>
    >> And the use of fully qualified named with no import.

    >
    > Given that 'import' is just syntactic sugar for FQNs, any dependency
    > analysis tool would only use FQNs for its analysis, I should think.


    Certainly. But the OP was suggesting a tool that scanned the .java
    file lexically for imports, and that would fail to find

    p1.p2.p3.class1 c = null;

    Personally, I'd build a dependency tool that scanned the .class file,
    not the .java file. (If there is no .class file, you don't need to
    know anything about dependencies to realize that the .java file needs
    to be recompiled.) The bytecode has everything you need, and there is
    lots of open-source code out there to help you parse it.
     
    Mike Schilling, May 12, 2008
    #11
  12. Lew wrote:
    > Mike Schilling wrote:
    >> Lew wrote:
    >>> Mike Schilling wrote:
    >>>> jolz wrote:
    >>>>>> I am looking for a class/program that will scan a .java file for
    >>>>>> imports (recursively) to discover which ones need to be rebuilt.
    >>>>>> This is close to GNU's makedepend or cook's c_incl.
    >>>>> You didn't consider import x.y.* and classes in the same package.
    >>>> And the use of fully qualified named with no import.
    >>> Given that 'import' is just syntactic sugar for FQNs, any dependency
    >>> analysis tool would only use FQNs for its analysis, I should think.

    >>
    >> Certainly. But the OP was suggesting a tool that scanned the .java
    >> file lexically for imports, and that would fail to find
    >>
    >> p1.p2.p3.class1 c = null;
    >>
    >> Personally, I'd build a dependency tool that scanned the .class file,
    >> not the .java file. (If there is no .class file, you don't need to
    >> know anything about dependencies to realize that the .java file needs
    >> to be recompiled.) The bytecode has everything you need, and there is
    >> lots of open-source code out there to help you parse it.

    >
    > Doesn't every existing tool that does dependency scanning work off the
    > .class files?
    >
    > It escaped me that the question restricted the scan to source files. I
    > simply figured any dependency tool would simply have to work off the
    > .class files, never even dreaming that anyone would try to do it from
    > the source.
    >


    Isn't this a catch-22 though in that if you are in a mixed lang
    enviroment and either need the java compiled first (in the case of it
    being the backend) or the other lang being done first (JNI) and it is
    being compiled by someone who has not developed it then it then follows
    there are no .class files. Why did sun get rid of -depend on javac
    anyways? I also found JavaDepend
    (http://ptolemy.eecs.berkeley.edu/~cxh/java/javadepend/) but it doesn't
    work on anything newer then 1.1 and I am doing 1.5 projects.

    Also in an other post in the thread someone mentioned if a compile
    doesn't work with say javac `find . -name '*.java'` or javac *.java then
    it is time to break it apart into standalone jar's. This too raises a
    depend issue in that if oyu have several packages which ones need to be
    recompiled when X changes.

    In short I am *NOT IMPRESSED* with Java's ability to manage large
    projects, In a way this is the same issue recursive make has
    http://aegis.sourceforge.net/auug97.pdf
     
    Aryeh M. Friedman, May 12, 2008
    #12
  13. Aryeh M. Friedman

    Roedy Green Guest

    On Sun, 11 May 2008 11:03:34 -0400, "Aryeh M. Friedman"
    <> wrote, quoted or indirectly quoted someone
    who said :

    >I am looking for a class/program that will scan a .java file for imports


    Imports are not sufficient since you can have fully qualified names in
    the source text.

    Further imports such as import javax.swing.* brings in much more than
    you would actually need.

    If you scanned the class files, you would miss new classes referenced
    since the last compile.

    So that means you have to scan source and parse it. JavaC can
    probably do that faster than you can.

    I think you are better off to:

    1. use ANT so Javac does not need to reloaded for multiple compiles.

    2. just feel the whole shebang on the command line and let JavaC
    figure out what needs to be recompiled.

    3. from time to time do a clean compile (erase all class files), and
    certainly before any release.
    --

    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
     
    Roedy Green, May 12, 2008
    #13
  14. Lew wrote:
    > Aryeh M. Friedman wrote:
    >> In short I am *NOT IMPRESSED* with Java's ability to manage large
    >> projects, In a way this is the same issue recursive make has
    >> http://aegis.sourceforge.net/auug97.pdf

    >
    > Funny, I don't remember anyone claiming that Java is a
    > project-management system. I thought it was a programming language. Do
    > you blame C for not having the features of make, I wonder?


    C does at least has a trivial way to find depends (#include)

    >
    > Use Ant, of course. Maven for things Ant can't.
    >


    Neither seems to be able to manage non-Java projects and if one wants to
    have the project a part of a OS dist (or ports collection) then almost
    all of them only accept make compatible build scripts. An other
    requirement I have is the ability to stich the source tree together from
    disjoint dirs (i.e. some of it is in a repo (the unmodified code) and
    some is in a local dir (the modified code)). The only dependency
    management tool I have ever found that manages this in anything like a
    reasonable manner is cook
    (http://miller.emu.id.au/pmiller/software/cook/) probally because it was
    written by the same person that wrote my SCM tool (aegis.sf.net). The
    only other alternative I know of is something like unionfs which is
    horrible. Oh forgot to mention cook also has an builtin Makefile
    generator in it so the OS ports/dist is happy.
     
    Aryeh M. Friedman, May 12, 2008
    #14
  15. Lew wrote:
    > Aryeh M. Friedman wrote:
    >> Lew wrote:
    >>> Aryeh M. Friedman wrote:
    >>>> In short I am *NOT IMPRESSED* with Java's ability to manage large
    >>>> projects, In a way this is the same issue recursive make has
    >>>> http://aegis.sourceforge.net/auug97.pdf
    >>>
    >>> Funny, I don't remember anyone claiming that Java is a
    >>> project-management system. I thought it was a programming language.
    >>> Do you blame C for not having the features of make, I wonder?

    >>
    >> C does at least has a trivial way to find depends (#include)

    >
    > Huh? What does #include have to do with managing object file dependencies?



    As with any type of build procedure there are implicit and explicit
    depends that make/cook/whatever needs to handle... Explicit ones are
    the ones you are thinking of when you are thinking that foo.c depends on
    ack.c because it uses bar() from ack.c... *BUT* since ANSI (and almost
    all compilors accept ANSI by default now) requires a prototype (even if
    it returns an int or void) this means I need to define the prototype for
    bar() somewhere and more often then not you do that in ack.h... now in
    foo.c I have:

    #include "ack.h"

    Assuming that I use the same name for the .h and .c it goes to I only
    need to scan for #include's in foo.c and do a batch processed search and
    replace (say with sed in unix) and replace all .h with .c (or actually
    ..o for techinical reasons)... this now gives me a list of all my
    implicit depends (due to the need to prototype about 95% of the total
    for a typical project)... All I need to do in the
    makefile/cookbook/whatever is add a rule like this (cook has nicer
    syntax then make so I will use it):

    %0%.o: %0%.c
    /*in reality I would also add a call to the dep scanner above*/
    {
    gcc [command line opts] -c %0%.c -o [target];
    }

    You should be asking now how to handle explicit depends... thats when
    you actually hard/softcode it into the makefile/cookbook/whatever with
    something like:

    bin/foo: lib/liback.a
    {
    gcc -Llib foo.c -o [target];
    }

    lib/liback.a: lib/ack.o lib/fred.o ....
    {
    ar rv [target] ack.o fred.o ...
    }

    Almost all these can be soft coded by various tricks with a good depend
    scanner for example to get the "ack.o fred.o ..." list I would just say
    every .c in lib/ needs to become a .o.


    Now cook does something really smart at this point compared to make. I
    creates the entire DAG *BEFORE* attempting to build any target thus
    allows you to avoid all the issues in RMCF.

    Due to this premade DAG approach I can use the same cookbook to build
    and package multiple packages (using the java term) *BUT* only when
    something in them or a package they depend on changes.

    The reason for the orginal question is I do not want to have to use
    nothing but explicit depends and if I am writting this for public
    release the person who DL's the source code will not have any .class
    files yet.

    >
    > If you recompile foo.c and bar.o depends on foo.o, nothing in C is going
    > to help you figure that one out, via #include or otherwise. You need
    > make for that.
    >
    > #include is a *pre*-processing step; it happens before the compilation
    > even looks at the source code. There is no equivalent in Java. The
    > 'import' directive does not include anything at all; it just helps the
    > compiler with name resolution.
    >
     
    Aryeh M. Friedman, May 12, 2008
    #15
  16. Aryeh M. Friedman wrote:

    >> Use Ant, of course. Maven for things Ant can't.
    >>

    >
    > Neither seems to be able to manage non-Java projects and if one wants
    > to have the project a part of a OS dist (or ports collection) then
    > almost all of them only accept make compatible build scripts.


    Ant can run any java program or native exceutable you like. We use it to
    build a combined Java and C# system. In fact, the standard Ant distribution
    includes tasks to build .NET programs, though we've had to modify them for
    our requirements.
     
    Mike Schilling, May 12, 2008
    #16
  17. Lew wrote:
    >> Certainly. But the OP was suggesting a tool that scanned the .java
    >> file lexically for imports, and that would fail to find
    >>
    >> p1.p2.p3.class1 c = null;
    >>
    >> Personally, I'd build a dependency tool that scanned the .class file,
    >> not the .java file. (If there is no .class file, you don't need to
    >> know anything about dependencies to realize that the .java file needs
    >> to be recompiled.) The bytecode has everything you need, and there
    >> is lots of open-source code out there to help you parse it.

    >
    > Doesn't every existing tool that does dependency scanning work off
    > the .class files?
    >
    > It escaped me that the question restricted the scan to source files. I
    > simply figured any dependency tool would simply have to work off
    > the .class files, never even dreaming that anyone would try to do it
    > from the source.


    There are more things in Heaven and Earth, Horatio, than are dreamt of in
    your philosphy. :)
     
    Mike Schilling, May 12, 2008
    #17
  18. Aryeh M. Friedman wrote:
    > In short I am *NOT IMPRESSED* with Java's ability to manage large
    > projects,


    How many years of experience do you have using ant ?

    Arne
     
    Arne Vajhøj, May 12, 2008
    #18
  19. I manage the page with javadepend, which has not been updated in
    years. I stashed a copy of javadepend there mainly to preserve
    history.

    We had a similar problem where we needed to find dependencies and
    someone use Ivy, which seemed to work reasonably well.

    http://java-source.net/open-source/build-systems/ivy

    Yes, it is always possible to miss classes because of reflection, but
    Ivy gave us a leg up on analyzing dependencies. If you really want to
    know what packages are used, do treeshaking by running java -verbose
    and exercise the program. We use this to quite effectively for
    shipping small jars. However, it is difficult to do treeshaking
    automatically and get a complete set of jars unless you effectively
    exercise the program.

    Also, the Eclipse Metrics package has a dependency visualizer that
    might be of interest. I looked at it briefly this week.

    About the ant/make thing. After using both and shipping both, I've
    come to the conclusion that that Ant and Make to be pretty much
    functionally equivalent. One can argue about the details though.

    To ship software, what I need is the following:
    1. Before compilation, some checks and searches need to happen.
    Checks for the right version of the compiler etc., searches for
    extensions. Historically, this is done with configure, though ant
    can do more of this than it used to.
    Unfortunately, portions of my user community does not want to run
    configure before starting Eclipse. Nor do they want to run an
    ant task from within Eclipse.
    2. During compilation, packages that won't compile need to be skipped.
    Many years ago, ant could not handle dependencies, now it can.
    Also, many years ago, the ant page said it was written by people
    who did not understand make. I've seen this issue before when
    people complain about make and then start writing their own system.
    3. At release time, each package needs to have some sort of manifest
    that lists files and subdirectories to be shipped. Currently, the
    manifest for Ptolemy is in makefiles, but it could be in separate text
    files, which could be read by recursive ant files.
    4. At user run time, I'd like to provide modules and provided updates
    similar to Eclipse/Equinox and the oSGI work. This requires dependency
    analysis based on versions, including supporting modules that
    themselves might require different versions of _the_same_ jar file.
    This is difficult, and requires classpath support.


    The manifest files are what refute some of the arguments put forth in
    "recursive make considered harmful."

    Ant and Eclipse's "Just compile everything" strategy does not work for
    research projects where some packages in a tree will not compile
    unless missing dependencies are present.

    Thus, I'm interested in Maven, but really looking for something that
    handles complex configuration and package dependencies.

    Personally, I'm pretty happy with the package system, it does make
    modularizing large projects (>300k LOC) possible. However, more
    work needs to be done concerning supporting multiple projects.

    _Christopher


    "Aryeh M. Friedman" <> writes:

    > Lew wrote:
    > > Mike Schilling wrote:
    > >> Lew wrote:
    > >>> Mike Schilling wrote:
    > >>>> jolz wrote:
    > >>>>>> I am looking for a class/program that will scan a .java file for
    > >>>>>> imports (recursively) to discover which ones need to be rebuilt.
    > >>>>>> This is close to GNU's makedepend or cook's c_incl.
    > >>>>> You didn't consider import x.y.* and classes in the same package.
    > >>>> And the use of fully qualified named with no import.
    > >>> Given that 'import' is just syntactic sugar for FQNs, any dependency
    > >>> analysis tool would only use FQNs for its analysis, I should think.
    > >>
    > >> Certainly. But the OP was suggesting a tool that scanned the .java
    > >> file lexically for imports, and that would fail to find
    > >>
    > >> p1.p2.p3.class1 c = null;
    > >>
    > >> Personally, I'd build a dependency tool that scanned the .class
    > >> file, not the .java file. (If there is no .class file, you don't
    > >> need to know anything about dependencies to realize that the .java
    > >> file needs to be recompiled.) The bytecode has everything you
    > >> need, and there is lots of open-source code out there to help you
    > >> parse it.

    > > Doesn't every existing tool that does dependency scanning work off
    > > the .class files?
    > > It escaped me that the question restricted the scan to source files.
    > > I simply figured any dependency tool would simply have to work off
    > > the .class files, never even dreaming that anyone would try to do it
    > > from the source.
    > >

    >
    > Isn't this a catch-22 though in that if you are in a mixed lang
    > enviroment and either need the java compiled first (in the case of it
    > being the backend) or the other lang being done first (JNI) and it is
    > being compiled by someone who has not developed it then it then
    > follows there are no .class files. Why did sun get rid of -depend on
    > javac anyways? I also found JavaDepend
    > (http://ptolemy.eecs.berkeley.edu/~cxh/java/javadepend/) but it
    > doesn't work on anything newer then 1.1 and I am doing 1.5 projects.
    >
    > Also in an other post in the thread someone mentioned if a compile
    > doesn't work with say javac `find . -name '*.java'` or javac *.java
    > then it is time to break it apart into standalone jar's. This too
    > raises a depend issue in that if oyu have several packages which ones
    > need to be recompiled when X changes.
    >
    > In short I am *NOT IMPRESSED* with Java's ability to manage large
    > projects, In a way this is the same issue recursive make has
    > http://aegis.sourceforge.net/auug97.pdf


    --
    Christopher Brooks (cxh at eecs berkeley edu) University of California
    Chess Executive Director US Mail: 337 Cory Hall #1774
    Programmer/Analyst Chess/Ptolemy/Trust Berkeley, CA 94720-1774
    ph: 510.643.9841 fax:510.642.2718 (office: 400A Cory)
    home: (F-Tu) 707.665.0131 (W-F) 510.655.5480
     
    Christopher Brooks, May 16, 2008
    #19
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?V29uZw==?=

    Asp.net upload file links to virus scanner

    =?Utf-8?B?V29uZw==?=, Aug 30, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    522
    =?Utf-8?B?V29uZw==?=
    Aug 30, 2004
  2. =?Utf-8?B?V29uZw==?=

    Asp.net Upload File links to Virus Scanner

    =?Utf-8?B?V29uZw==?=, Aug 30, 2004, in forum: ASP .Net
    Replies:
    4
    Views:
    4,039
    bsandhu
    Dec 16, 2005
  3. Replies:
    5
    Views:
    1,685
    tehka
    Sep 5, 2006
  4. cccc

    Scanner image manager

    cccc, Jun 15, 2004, in forum: Java
    Replies:
    1
    Views:
    470
    Dave Neary
    Jun 15, 2004
  5. el goog
    Replies:
    0
    Views:
    7,344
    el goog
    Feb 16, 2005
Loading...

Share This Page