The program that would'nt Quit

R

Roedy Green

I have I think discovered a strange bug in both JDK 1.5 and 1.6.

the program ends like this:

System.out.println( "done" );
// System.exit( 0 );
} // end main

Without that System.exit, the program keeps on running endlessly using
about 94% of my CPU time after printing "done".

The odd thing is this is not a gui and it has but a single thread. I
have no finalizers.

It works fine in Jet.

I don't know where to begin to track this down or how to reproduce it
for Sun. The jar is 271K and the data files it works on are 14 MB.

Any ideas on what could cause such behaviour?
 
R

Roedy Green

the program ends like this:

System.out.println( "done" );
// System.exit( 0 );
} // end main

there in the class containing main. I don't see anything out of the
ordinary about it.

package com.mindprod.htmlmacros;
import java.io.File;
import java.io.FilenameFilter;
import java.util.ArrayList;

import com.mindprod.filter.ExtensionListFilter;

/**
* Feeds the entire Mindprod website to the Replacer macro mechanism
refreshing
* expansions of all macros. Expands HTML boilerplate refresher macros
e.g. <!--
* macro Include dogs.txt --> <!-- generated --> <img
* src="images/animals/dog.jpg" height="410" width="511" alt="dog">
<!--
* /generated by Include --> Allows arbitrary expander macro code.
Futures:
* Error messages give offset/line number, nearby text check for
generated by
* macro not just generated to remove old text. Can embed a " in a
parm by
* encoding it as &quot;
*/
public class ReplaceMindprod extends Replacer {

/**
* True if want debugging output, counts of files gathered from
different
* dirs.
*/
private static final boolean DEBUGGING = false;

/**
* total count of files gathered
*/
private static int gatherCount;

/**
* arrayList where we build list of files to process.
*/
static ArrayList<String> filesToProcess;

/**
* Estimated files will process.
*/
public static final int ESTIMATED_TOTAL_FILES = 4500;

/**
* Collect all files in the given dirName for processing.
*
* @param dirName
* Name of fully qualified directory where to look for the
html
* files.
* @param filter
* filter to use to decide while files in that dir to
process.
*/
static void gather ( String dirName, FilenameFilter filter )
{
String[] filenames = new File( dirName ).list( filter );
if ( filenames == null )
{
throw new IllegalArgumentException(
"com.mindprod.htmlmacros.ReplaceMindprod: No such dir:
"
+ dirName );
}
gatherCount += filenames.length;
if ( DEBUGGING )
{
System.out.println( filenames.length
+ " files gathered from "
+ dirName );
}
for ( String filename : filenames )
{
filesToProcess.add( dirName + "/" + filename );
}
}

/**
* Get fully qualified name of all the files to process
*/
public static void gatherFiles ()
{

// add *.html files in mindprod

filesToProcess = new ArrayList<String>( ESTIMATED_TOTAL_FILES
);
FilenameFilter f = new ExtensionListFilter( new String[] {
"html"
} );

gather( "e:/mindprod", f );
gather( "e:/mindprod/animalrights", f );
gather( "e:/mindprod/applets", f );
gather( "e:/mindprod/applications", f );
gather( "e:/mindprod/bgloss", f );
// bookcovers
gather( "e:/mindprod/contact", f );
gather( "e:/mindprod/deepthoughts", f );
// dvdcovers
gather( "e:/mindprod/environment", f );
gather( "e:/mindprod/esperanto", f );
gather( "e:/mindprod/ethics", f );
gather( "e:/mindprod/feedback", f );
gather( "e:/mindprod/ggloss", f );
gather( "e:/mindprod/humanrights", f );
// icon
// images
// include won't work since macros in wrong dir
gather( "e:/mindprod/jgloss", f );
// jgloss/include
// kjv, rarely changes, do manually when it needs it.
gather( "e:/mindprod/livinglove", f );
gather( "e:/mindprod/livinglove/methods", f );
gather( "e:/mindprod/money", f );
// pads
gather( "e:/mindprod/phone", f );
gather( "e:/mindprod/politics", f );
gather( "e:/mindprod/politics/laser", f );
// precis
gather( "e:/mindprod/projects", f );
gather( "e:/mindprod/quadra", f );

// redir, only makes sense to expand them in place. Don't
disturb copied
// versions
gather( "e:/mindprod/redir", f );
gather( "e:/mindprod/religion", f );
// renney
// replicator
gather( "e:/mindprod/reunion", f );
// screenshots
// snippets
// sound
gather( "e:/mindprod/utilities", f );
gather( "e:/mindprod/webstarts", f );
// zips

// all files are ready to go.

}

/**
* Expands macros written in Java in HTML files files.
*
* @param args
* names of files to process, no wildcards.
*/
public static void main ( String[] args )
{
System.out.println( "\nExpanding macros on entire Mindprod
website" );

Tools.setWebRoot( "e:\\mindprod" );
gatherFiles();

System.out.println( gatherCount
+ " files gathered for mindprod site macro expansion." );

assert gatherCount <= ESTIMATED_TOTAL_FILES;
for ( String fileToProcess : filesToProcess )
{
assert fileToProcess != null;
assert new File( fileToProcess ).exists() : "file to
process missing";
try
{
expandMacrosInFile( new File( fileToProcess ), true /*
quiet */);
}
catch ( Exception e )
{
System.err.println( e.getMessage()
+ " in file "
+ fileToProcess );
System.err.println();
e.printStackTrace();
// we don't exit. We keep on processing files.
}
} // end for each file loop
System.out.println( "done" );
// don't remove the System.exit( 0 ). It bypasses a Sun bug.
System.exit( 0 );
} // end main

} // end ReplaceMindProd
 
S

Stefan Ram

You could use Thread.activeCount() to confirm that there is
only one thread at the end.
Tools.setWebRoot( "e:\\mindprod" ); (...)
gatherFiles(); (...)
for ( String fileToProcess : filesToProcess ) (...)
expandMacrosInFile( new File( fileToProcess )

Comment out (hide) the call of "expandMacrosInFile" and see
whether the behavior still occurs.

If yes, remove "gatherFiles" and see whether ...

If no: The behavior might have been cause by
something in "expandMacrosInFile": So now, edit
this method, commenting out its final half or so.

Recursively repeat this until the cause is found.

(This approach is not always possible or might not
always work.)
 
A

abmccullough

Seems like taking a javacore after the "done" message is printed would
a)
tell you what thread is active and what it is doing and b)
provide the information Sun would need to fix it, if it is truly a bug.

-Andrew
 
T

Thomas Hawtin

Roedy said:
Without that System.exit, the program keeps on running endlessly using
about 94% of my CPU time after printing "done".

The odd thing is this is not a gui and it has but a single thread. I
have no finalizers.

It works fine in Jet.

I don't know where to begin to track this down or how to reproduce it
for Sun. The jar is 271K and the data files it works on are 14 MB.

Any ideas on what could cause such behaviour?

Have you tried using jstack (or similar) to see what is continuing to run?

Tom Hawtin
 
O

opalpa

Good day Roedy,

I empathize with this situation; I share an aversion for System.exit .
Some ideas:

There are situations where this occurs according to definition in
non-gui programs. For example when using a JMF player one needs to
stop(), deallocate(), and close() before graceful exit. Perhaps you
are inadvartanly and indirectly creating some event loop?
---
I know one program that exits gracefully after about ten seconds. Have
you waited this long? The general rule is if there is a thread that
only wakes up every X seconds then you might wait upto X seconds for it
to return from its run().
---
I believe Java spec says that JVM can choose to exit if only demon
threads areleft, but does not have to. I doubt this is the case
because Sun's jvm always seems to take the exit choice within a couple
of seconds.
---
Perhaps you should view running threads using JConsole. You will also
see each thread's state. There is a negative in that you will see
threads related to JConsole listed; this negative is not that large.

Hopefully this email is of some service

All the best,
Opalinski
(e-mail address removed)
http://www.geocities.com/opalpaweb/


P.S. Jconsole takes a total of maybe five minutes to learn. Run your
executable with -Dcom.sun.management.jmxremote and seperately execute
jconsole found within java's bin directory. That's it

Here is some documenation:
http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html
 
R

Roedy Green

There are situations where this occurs according to definition in
non-gui programs. For example when using a JMF player one needs to
stop(), deallocate(), and close() before graceful exit. Perhaps you
are inadvartanly and indirectly creating some event loop?

This is weird. I am doing pure batch text processing. There are no
events or listeners in the whole program. I don't spawn any threads --
any of the usual suspects in refusal to shut down.

Keep going. I hope I have overlooked something.
 
R

Roedy Green

I know one program that exits gracefully after about ten seconds. Have
you waited this long? The general rule is if there is a thread that
only wakes up every X seconds then you might wait upto X seconds for it
to return from its run().

There are no timers, execs or threads. About the only exotic feature
of this program is using Class.forName to load most of its classes.

Perhaps some sort of incomplete i/o? I do some auxiliary disk
processing and perhaps a failure is getting lost, maybe an unclosed
file somewhere. Unfortunately Java has no open method you can grep.
Also I don't know of a way of testing if and what files are still
open.
 
R

Roedy Green

Perhaps you should view running threads using JConsole. You will also
see each thread's state. There is a negative in that you will see
threads related to JConsole listed; this negative is not that large.

I'm running it now It turns out to be ten times easier to set up than
it first looks. I have documented this, as is my wont, at
http://mindprod.com/jgloss/jconsole.html

Surprises for me includes:

1. my "single thread' app had 18 threads.

3. garbage collections are extremely frequent. There over a 1000 of
them in the time it took to connect and start looking. However
JConsole said that only a few seconds were used in GC. I had assumed
they were much less frequent and far more painful.

4. the different aged collections of objects truly do behave
differently. the older collections are rock stable. The eden has a two
frequency periodicity, like a sound with harmonics.

5. My app uses about 15 MB of heap space by default. I had never done
much thinking about ram usage. It is a bit of a shock. In the 1970s I
think I would have pulled off an app like this in 200K including the
run time.

6. There is a quantum mechanical effect. Watching the apps with
JConsole makes it work!!
 
T

Thomas Hawtin

Roedy said:
Jstack it turns out does not run on windows.

jstack, by default, does much the same as ctrl-\ (which I think is
ctrl-break on Windows) only with less fiddling. There are some options,
like -m which causes it to NPE lots.

Tom Hawtin
 
R

Roedy Green

That certainly smells like a threading issue. Every time I've used
tools to inspect the code and the code started to work, it has always
turned out to be a threading problem.

http://catb.org/~esr/jargon/html/H/heisenbug.html

more weirdness. The bug has now gone altogether. What changed? I
added some convenience methods to one of the base classes, but I don't
yet use them anywhere.

This is like the old days of FORTRAN when making tiny changes to code
changed which code got clobbered when subscripts when out of range.

I don't like this. I have been hit twice in a day with programs that
work fine on one compiler but not another. It is a sign Java has
grown past the manageable bounds of complexity for a group of humans.

There is no JNI in my code.
 
O

opalpa

This is weird. I am doing pure batch text processing.

Opening files is preceeded by SecurityManager checks. That may involve
threads. Stefan Ram suggests a gradual isolation which may be
reasonable.
6. There is a quantum mechanical effect. Watching the apps with
JConsole makes it work!!

Maybe, a long shot, when watching you give extra time which you don't
give when not watching and something time limited gets to complete.

Opalinski
(e-mail address removed)
http://www.geocities.com/opalpaweb/
 
O

opalpa

M

Monique Y. Mudama

I don't like this. I have been hit twice in a day with programs that
work fine on one compiler but not another. It is a sign Java has
grown past the manageable bounds of complexity for a group of
humans.

Now, now. I happen to know for a fact that they've had problems like
this all along. In my graphics class in college, we were using Java
1.1.7. I always made sure to test my projects, which I developed on
my windows box, on the lab boxes before submitting them. My professor
laughed at me and told me that Java works everywhere.

Well. I was running late for the final project of the semester, and I
was running into some weird environment problem, so I turned in the
project, a wireframe activated by keypresses, which worked flawlessly
on my Windows box. I got a zero on the project because the keypress
code didn't work the same between Windows and Solaris. I got a C for
the course instead of an A.

I know @!#! well that Java has *never* worked perfectly across all
systems.
 
A

abmccullough

It's not an API, it's a "dump" of the JVM state that you can use to
debug things. It is automatically created in certain error conditions
(like a bad signal from JNI code), but you can request one to debug a
problem. On Windows, type Ctrl-BREAK in the command window in which
the Java.exe is active, on *NIX, issue a kill -3 to the java.exe
process. In windows, it will dump to standard out (the console window
-- usually, you cannot do this in an IDE) a list of all threads, the
currently active thread, held locks and monitors, and a variety of
other highly useful information. On *NIX it will write the same
information to a file named javacoreTIMESTAMP.txt in the default
directory. I use it all the time to determine why things don't stop
and other problems. Some web searching should help you find some
pointers on interpreting the contents, but the thread dump is pretty
straightforward -- it lists all the threads and their current stack.


-Andrew
 
R

Roedy Green

On Windows, type Ctrl-BREAK in the command window in which
the Java.exe is active,

All I have even seen is a abort. does it hide this dump somewhere
without telling you?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,768
Messages
2,569,574
Members
45,048
Latest member
verona

Latest Threads

Top