My app runs too slow (lots of code follows)

[Mikey]

Let me start out by saying that I'm admittedly not a great programmer.
I'm sure that my problems are my own fault and not really Java's. I
have a program that I had originally written in Perl. For various
reasons I decided to rewrite it in Java, but now that I'm nearly done,
I'm not sure if I want to use it because it's very slow and resource
intensive compared to my Perl version.

Quick summary of what the program does: it performs MD5 sums on a
number of files, stores the sums and compares against them on future
runs -- sort of a cheesy tripwire, checks that some processes are
running, checks the contents of a few small files like /etc/passwd,
/etc/shadow, etc. It's strictly a Unix program for me.

My old perl version generally runs the comparisons, checks file
permissions, reads a couple of files, along with reading the last few
minutes of a few logfiles (something I have yet to really implement in
my new Java version, which will make it run even longer, of course) in
less than a second. My java version does it (minus the logfile
reading) in anywhere between 10 and 15 seconds.

Here's my -Xprof output:

Flat profile of 0.74 secs (37 total ticks): process reaper

Thread-local ticks:
100.0% 37 Unknown: no last frame


Flat profile of 0.69 secs (36 total ticks): process reaper

Thread-local ticks:
100.0% 36 Unknown: no last frame


Flat profile of 0.47 secs (24 total ticks): process reaper

Thread-local ticks:
100.0% 24 Unknown: no last frame


Flat profile of 0.79 secs (41 total ticks): process reaper

Thread-local ticks:
100.0% 41 Unknown: no last frame


Flat profile of 10.98 secs (324 total ticks): main

Interpreted + native Method
1.2% 0 + 2 java.io.UnixFileSystem.getBooleanAttributes0
1.2% 0 + 2 java.lang.System.arraycopy
0.6% 1 + 0 java.io.File.getName
0.6% 0 + 1 ssm.FileStat.getOwner
0.6% 0 + 1 java.util.ResourceBundle.<clinit>
0.6% 1 + 0 java.io.BufferedInputStream.fill
0.6% 1 + 0 sun.nio.cs.US_ASCII$Decoder.decodeArrayLoop
0.6% 0 + 1 java.io.FileInputStream.open
5.8% 3 + 7 Total interpreted

Compiled + native Method
34.7% 59 + 1 java.io.ByteArrayOutputStream.write
27.2% 47 + 0 java.io.BufferedInputStream.read
18.5% 32 + 0 sun.security.provider.MD5.transform
1.2% 2 + 0 java.util.Arrays.mergeSort
1.2% 2 + 0 ssm.MD5Sum.loadByteData
0.6% 1 + 0 java.lang.StringBuffer.length
0.6% 1 + 0 ssm.UnixCrypt.D_ENCRYPT
0.6% 1 + 0 java.lang.String.toString
84.4% 145 + 1 Total compiled

Stub + native Method
0.6% 0 + 1 java.lang.System.arraycopy
0.6% 0 + 1 java.io.FileInputStream.readBytes
1.2% 0 + 2 Total stub

Thread-local ticks:
46.6% 151 Blocked (of total)
0.6% 1 Class loader
7.5% 13 Unknown: no last frame
0.6% 1 Unknown: thread_state


Flat profile of 0.00 secs (1 total ticks): Thread-1

Interpreted + native Method
100.0% 1 + 0
java.security.AccessController.getStackAccessControlContext
100.0% 1 + 0 Total interpreted


Global summary of 11.01 seconds:
100.0% 327 Received ticks
0.3% 1 Received GC ticks
0.6% 2 Compilation
0.3% 1 Class loader
46.5% 152 Unknown code


And, because I'm not really sure what to point anyone at, I'll just
attach
the bulk of my code, sorry...

main:=========================================================
import ssm.*;
import java.io.*;
import java.util.*;
import java.net.InetAddress;
import java.text.SimpleDateFormat;

class SystemsSecurityMonitor {
private static HashManager hm = new HashManager();
private static FileListManager flm = new FileListManager();
private static SendMail sm = new SendMail();
private static StringBuffer emailtext = new StringBuffer();
private static ProcessMonitor pm = new ProcessMonitor();
private static SimpleDateFormat df = new SimpleDateFormat();
private static GregorianCalendar cal = new GregorianCalendar();
private static GregorianCalendar tm = new GregorianCalendar();
private static final long MILLIS_PER_MINUTE = 60000;
private static String earliest = null;

public static void printUsage() {
System.out.println("Usage: [-f <file>] [-i <min>] [-h] [-s]
[-u] [-p]" +
"\n\nwhere min is the number of minutes in between\n" +
"executions of this program\n\n" +
"Default values can be found below between the <>\n\n" +
" -f -- specify the file to use as the md5 database
</.md5db>\n" +
" -i -- interval between executions <5>\n" +
" -h -- help (this screen)\n" +
" -p -- print the list of files to look at\n" +
" -s -- display md5 sums of system binaries and exit\n" +
" -u -- update the md5 database and exit\n" +
" -v -- verbose mode\n\n");
}

public static String goBackIntervalMinutes(String dateFormat) {
int min = Integer.parseInt(System.getProperty("interval")) +
1;

df.applyPattern(dateFormat);
tm.setTime(cal.getTime());
tm.setTimeInMillis(cal.getTimeInMillis() - (MILLIS_PER_MINUTE
* min));
return df.format(tm.getTime());
}

// public static boolean isEarlierThanWeWant(String date) {
// try {


public static void printFileList() {
String[] list = flm.getFileList();

System.out.println("Listing files to inspect:\n");
for(int i = 0; i < list.length; i++) {
System.out.println("\to " + list);
}
}

public static void showSums() {
hm.getHashProperties().list(System.out);
}

public static void updateSums(String filename) {
try {
hm.storeHash(filename);
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}
}

public static void compareSums() {
String storedhash, newhash = null;
Properties p1 = null;

try {
p1 = hm.getStoredHash(System.getProperty("md5file"));
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}

for(int i = 0; i < flm.getFileList().length; i++) {
String theFile = flm.getFileList();
storedhash = p1.getProperty(theFile);
try {
newhash = hm.hashFile(theFile);
}
catch (IOException ioe) { /* and ignore */ }

if(! newhash.equals(storedhash)) {
emailtext.append(
"--------------------------------------------------------\n"+
"Incorrect file info for file: " + theFile + "\n" +
"Stored Sum: " + storedhash + "\n" +
"Actual Sum: " + newhash + "\n" +
"--------------------------------------------------------\n");
}
}
}

public static boolean procIsRunning(String procName) {
boolean isRunning = false;
int retval = -2;

try {
retval = pm.isProcessRunning(procName);
if(retval == -1) {
System.err.println("Error during isProcessRunning()");
}
else if(retval == -2) {
System.err.println("Error during procIsRunning()");
}
else if(retval > 0) {
isRunning = true;
}
}
catch(IOException ioe) {
System.err.println(ioe.toString());
}

return isRunning;
}

public static void checkProcesses() {
String[] procs = System.getProperty("processes").split(";");

for(int i = 0; i < procs.length; i++) {
if(! procIsRunning(procs)) {
emailtext.append("Process not found: " + procs);
}
}
}

public static void checkFtpUsers() {
String line;
String ftpusers = System.getProperty("files.ftpusers");
String[] users = System.getProperty("users.noftp").split(" ");

try {
BufferedReader reader = new BufferedReader(
new FileReader(new File(ftpusers)));

while((line = reader.readLine()) != null) {
for(int i = 0; i < users.length; i++) {
if(users.equals(line)) {
users = "";
}
}
}
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}

for(int i = 0; i < users.length; i++) {
if(! users.equals("")) {
emailtext.append(ftpusers + " : user not found: " +
users + "\n");
}
}
}

public static void lookForPlusInRhosts() {
String line;
String rhosts = System.getProperty("files.rhosts");

try {
BufferedReader reader = new BufferedReader(
new FileReader(new File(rhosts)));

while((line = reader.readLine()) != null) {
if(line.matches(".*\\+.*"))
emailtext.append(rhosts + " : found +\n");
}
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}
}

public static void checkPasswordFile() {
boolean sawRoot = false;
String line, user, pass, uid, gid, gecos, home, shell;
String passwd = System.getProperty("files.passwd");
String root = System.getProperty("users.root");
String[] info;
String[] shells = System.getProperty("shells").split(" ");
Arrays.sort(shells);

try {
BufferedReader reader = new BufferedReader(
new FileReader(new File(passwd)));

while((line = reader.readLine()) != null) {
info = line.split(":");

user = info[0];
pass = info[1];
uid = info[2];
gid = info[3];
gecos = info[4];
home = info[5];

if(info.length == 7) {
shell = info[6];
} else {
shell = null;
}

if(user.equals(root) && uid.equals("0")) {
sawRoot = true;
}
else if(uid.equals("0")){
emailtext.append(passwd + " : uid=0: " + user);
}

if(shell != null && (Arrays.binarySearch(shells,
shell) < 0))
emailtext.append(passwd + " : " + user +
" has invalid shell: " + shell + "\n");
}
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}

if(! sawRoot)
emailtext.append("Never saw root user: " + root);
}

public static void checkShadowFile() {
String line, salt, emptypass;
String shadow = System.getProperty("files.shadow");
String[] info;

try {
BufferedReader reader = new BufferedReader(
new FileReader(new File(shadow)));

while((line = reader.readLine()) != null) {
salt = null;
emptypass = null;
info = line.split(":");

if(info[1].equals("")) {
emailtext.append("user has no password: " +
info[0]);
}
else {
salt = info[1].substring(0, 2);
emptypass = UnixCrypt.crypt(salt, "");
if(info[1].equals(emptypass))
emailtext.append("user hit enter for his
password: " +
info[0]);
}
}
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}
}

public static void setupAppProperties() throws IOException {
String propfile = "default_settings.properties";

Properties p = new Properties(System.getProperties());
String sp = System.getProperty("os.name") +
"_settings.properties";
File f = new File("./" + sp);
if(f.exists())
propfile = sp;

FileInputStream props = new FileInputStream(propfile);
p.load(props);
props.close();
System.setProperties(p);

flm.setFileList(System.getProperties());
}

public static void main(String[] args) {
int ch = -1;

boolean opt_s = false;
boolean opt_u = false;
boolean verbose = false;
boolean ckpsflag = false;

String hostname = "unknown";

GetOpt go = new GetOpt(args, "suhlvi:f:");

try {
hostname = InetAddress.getLocalHost().getHostName();
}
catch (java.net.UnknownHostException uhe) { /* and ignore */ }

try {
setupAppProperties();
}
catch (IOException ioe) {
System.err.println("complication with app properties file:
" +
ioe.toString());
System.exit(1);
}

go.optErr = true;
while ((ch = go.getopt()) != go.optEOF) {
if((char)ch == 's') {
opt_s = true;
}
else if((char)ch == 'u') {
opt_u = true;
}
else if((char)ch == 'h') {
printUsage();
System.exit(0);
}
else if((char)ch == 'l') {
printFileList();
System.exit(0);
}
else if((char)ch == 'v') {
verbose = true;
}
else if((char)ch == 'i') {
System.setProperty("interval", go.optArgGet());
}
else if((char)ch == 'f') {
System.setProperty("md5file", go.optArgGet());
}
else {
printUsage();
System.exit(1); // undefined
option
}
}

try {
hm.setHashProperties(flm.getFileList());
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}

if(opt_s) {
showSums();
System.exit(0);
}
else if(opt_u) {
updateSums(System.getProperty("md5file"));
System.exit(0);
}

compareSums();
checkProcesses();
checkFtpUsers();
lookForPlusInRhosts();
checkPasswordFile();
checkShadowFile();

if(emailtext.length() > 0) {
try {
if(verbose)
System.out.println(emailtext);
sm.send("mailhost", "root",
"SECURITY ALERT: " + hostname,
emailtext.toString());
}
catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
}

md5:=================================================================
package ssm;

import java.io.*;
import java.security.*;

public class MD5Sum {
private byte[] loadByteData(String filename) throws IOException {
BufferedInputStream bis = new BufferedInputStream(
new FileInputStream(filename));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int ch;

while((ch = bis.read()) != -1) {
baos.write(ch);
}

return baos.toByteArray();
}

private String hexDigit(byte x) {
StringBuffer sb = new StringBuffer();
char c;

c = (char)((x >> 4) & 0x0f);
if(c > 9) {
c = (char)((c - 10) + 'a');
}
else {
c = (char)(c + '0');
}

sb.append(c);

c = (char)(x & 0xf);
if(c > 9) {
c = (char)((c - 10) + 'a');
}
else {
c = (char)(c + '0');
}

sb.append(c);
return sb.toString();
}

public String hash_file(String filename) throws IOException {
MessageDigest md5 = null;
byte[] content = null;

try {
md5 = MessageDigest.getInstance("MD5");
content = loadByteData(filename);
}
catch (NoSuchAlgorithmException nsae) {
nsae.printStackTrace();
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}

if(content != null) {
md5.update(content);
byte[] digest = md5.digest();
StringBuffer hexString = new StringBuffer();
int digestLength = digest.length;

for(int i=0; i < digest.length; i++) {
hexString.append(hexDigit(digest));
}

return hexString.toString();
}
else {
return null;
}
}
}

hashmanager:========================================================
package ssm;

import java.io.*;
import java.util.*;
import ssm.MD5Sum;
import ssm.FileStat;

public class HashManager {
private Properties hash;
FileStat fs = new FileStat();
MD5Sum md = new MD5Sum();

public void setHashProperties(String[] files) throws IOException {
hash = new Properties();
String prop;

for(int i = 0; i < files.length; i++) {
prop = md.hash_file(files) + "-" + fs.getMode(files)
+
"-" + fs.getOwner(files) + ":" +
fs.getGroup(files);
hash.setProperty(files, prop);
}
}

public Properties getHashProperties() {
return hash;
}

public Properties getStoredHash(String file) throws IOException {
FileInputStream store = new FileInputStream(file);
Properties storedhash = new Properties();
storedhash.load(store);

return storedhash;
}

public void storeHash(String file) throws IOException {
FileOutputStream out = new FileOutputStream(file);
getHashProperties().store(out, null);
out.close();
}

public String hashFile(String file) throws IOException {
return md.hash_file(file) + "-" + fs.getMode(file) +
"-" + fs.getOwner(file) + ":" + fs.getGroup(file);
}
}

filelistmanager:=====================================================
package ssm;

import java.io.*;
import java.util.*;

public class FileListManager {
private String flseparator = ":";
private StringBuffer filelist;

public void setFileList(Properties p) {
filelist = new StringBuffer();
String thisElement, fsep, sep, file;
String[] path = {"/usr/bin", "/usr/sbin", "/sbin"}, bin;

try {
sep = p.getProperty("path.separator");
fsep = p.getProperty("file.separator");
path = p.getProperty("env.path").split(sep);
}
catch (NullPointerException npe) {
sep = ":";
fsep = "/";
System.err.println("Running without default properties?");
}

for(Enumeration e = p.propertyNames(); e.hasMoreElements() {
thisElement = e.nextElement().toString();

if(thisElement.startsWith("files.")) {
file = p.getProperty(thisElement);

File f = new File(file);
if(f.exists()) {
filelist.append(p.getProperty(thisElement) + ":");
}
}
else if(thisElement.equals("binaries")) {
bin = p.getProperty(thisElement).split(" ");

for(int i = 0; i < path.length; i++) {
for(int j = 0; j < bin.length; j++) {
String thisbinary = path + fsep + bin[j];
File b = new File(thisbinary);
if(b.exists()) {
filelist.append(thisbinary + ":");
}
}
}
}
else if(thisElement.equals("shells")) {
bin = p.getProperty(thisElement).split(" ");

for(int i = 0; i < bin.length; i++) {
File f = new File(bin);
if(f.exists()) {
filelist.append(bin + ":");
}
}
}
}
}

public String[] getFileList() {
String[] flist = filelist.toString().split(flseparator);
Arrays.sort(flist);
return flist;
}
}

 

[Michael Borgwardt]

My old perl version generally runs the comparisons, checks file
permissions, reads a couple of files, along with reading the last few
minutes of a few logfiles (something I have yet to really implement in
my new Java version, which will make it run even longer, of course) in
less than a second. My java version does it (minus the logfile
reading) in anywhere between 10 and 15 seconds.

Here's my -Xprof output: []
Flat profile of 10.98 secs (324 total ticks): main []
34.7% 59 + 1 java.io.ByteArrayOutputStream.write
27.2% 47 + 0 java.io.BufferedInputStream.read

Well, it's pretty obvious where the cycles are being burned, isn't it.

Here's the culprit:

while((ch = bis.read()) != -1) {
baos.write(ch);
}

You pump data from the input stream into the byte array one byte at a
time - horrendously inefficient. Furthermore, the byte array only exists
to be passed to the MessageDigest, a copy step which is completely
superfluous.

use the read() method that takes a buffer byte array as an argument,
pay attention to the number of bytes read (the return value) and pass
the filled buffer portion directly to the MessageDigest. Oh, and
do this with the inputStream directly, without wrapping it into a
BufferedInputStream, since you already use a buffer.

This saves you billions of method invocations and two times
copying your entire data. The resulting code may well be faster than
the perl code.

 

[John C. Bollinger]

Let me start out by saying that I'm admittedly not a great programmer.
I'm sure that my problems are my own fault and not really Java's. I
have a program that I had originally written in Perl. For various
reasons I decided to rewrite it in Java, but now that I'm nearly done,
I'm not sure if I want to use it because it's very slow and resource
intensive compared to my Perl version.

Quick summary of what the program does: it performs MD5 sums on a
number of files, stores the sums and compares against them on future
runs -- sort of a cheesy tripwire, checks that some processes are
running, checks the contents of a few small files like /etc/passwd,
/etc/shadow, etc. It's strictly a Unix program for me.

My old perl version generally runs the comparisons, checks file
permissions, reads a couple of files, along with reading the last few
minutes of a few logfiles (something I have yet to really implement in
my new Java version, which will make it run even longer, of course) in
less than a second. My java version does it (minus the logfile
reading) in anywhere between 10 and 15 seconds.

Michael Borgwardt offered a well targetted suggestion for improving your
code's performance. You should also be aware, however, that any Java
program

(a) suffers a one-time performance penalty due to JVM startup. This can
be negligible or a few seconds, depending on many factors.

(b) speeds up as it runs, because of just-in-time compilation.

(c) has a memory footprint that includes the JVM's, typically several Mb.

This means that Java is not an ideal choice for an app that must perform
a brief task, once, as quickly as possible, nor for an app that must
have a minimal memory footprint.

Of course, Perl scripts suffer from Perl's load time and memory
footprint unless they are compiled down to machine code -- but in that
case they aren't really Perl any more.


John Bollinger
(e-mail address removed)

 

[Mikey]

use the read() method that takes a buffer byte array as an argument,
pay attention to the number of bytes read (the return value) and pass
the filled buffer portion directly to the MessageDigest. Oh, and
do this with the inputStream directly, without wrapping it into a
BufferedInputStream, since you already use a buffer.

This saves you billions of method invocations and two times
copying your entire data. The resulting code may well be faster than
the perl code.

Thanks very much for the response and the help. I'm pretty new, so I
just wanted to make sure that the code below is what you had in mind.
Incidentally, this code definitely drops me down to a 6-8 second
execution vs. 10-15. Thanks a bunch! The -Xprof output seems to show
a lot of MD5.transform execution now, but I suppose that's to be
expected, right?


Flat profile of 0.72 secs (36 total ticks): process reaper

Thread-local ticks:
100.0% 36 Unknown: no last frame


Flat profile of 0.49 secs (22 total ticks): process reaper

Thread-local ticks:
100.0% 22 Unknown: no last frame


Flat profile of 0.83 secs (41 total ticks): process reaper

Thread-local ticks:
100.0% 41 Unknown: no last frame


Flat profile of 1.13 secs (56 total ticks): process reaper

Thread-local ticks:
100.0% 56 Unknown: no last frame


Flat profile of 6.69 secs (246 total ticks): main

Interpreted + native Method
5.6% 0 + 4 java.io.UnixFileSystem.getBooleanAttributes0
1.4% 1 + 0 ssm.UnixCrypt.des_set_key
1.4% 0 + 1 ssm.HashManager.setHashProperties
1.4% 1 + 0 java.lang.String.<init>
1.4% 0 + 1
java.text.AttributedCharacterIterator$Attribute.<init>
1.4% 1 + 0 java.nio.ByteBuffer.<init>
1.4% 1 + 0 java.nio.HeapCharBuffer.<init>
1.4% 0 + 1 ssm.FileStat.getGroup
15.3% 4 + 7 Total interpreted

Compiled + native Method
50.0% 36 + 0 sun.security.provider.MD5.transform
2.8% 2 + 0 java.util.regex.Pattern$Single.match
1.4% 0 + 1
java.util.jar.Manifest$FastInputStream.readLine
1.4% 1 + 0 sun.security.provider.MD5.engineUpdate
1.4% 1 + 0 vtable chunks
56.9% 40 + 1 Total compiled

Stub + native Method
1.4% 0 + 1 java.io.FileInputStream.readBytes
1.4% 0 + 1 Total stub

Thread-local ticks:
70.7% 174 Blocked (of total)
8.3% 6 Class loader
4.2% 3 Compilation
13.9% 10 Unknown: no last frame


Global summary of 6.74 seconds:
100.0% 251 Received ticks
0.4% 1 Received GC ticks
0.8% 2 Compilation
0.4% 1 Other VM operations
2.4% 6 Class loader
65.7% 165 Unknown code

md5sum:========================================================
package ssm;

import java.io.*;
import java.security.*;

public class MD5Sum {
MessageDigest md5 = null;
FileInputStream in = null;

private String hexDigit(byte x) {
StringBuffer sb = new StringBuffer();
char c;

c = (char)((x >> 4) & 0x0f);
if(c > 9) {
c = (char)((c - 10) + 'a');
}
else {
c = (char)(c + '0');
}

sb.append(c);

c = (char)(x & 0xf);
if(c > 9) {
c = (char)((c - 10) + 'a');
}
else {
c = (char)(c + '0');
}

sb.append(c);
return sb.toString();
}

public String hash_file(String filename) throws IOException {
byte[] buf = new byte[4096];
int len = 0;

try {
in = new FileInputStream(new File(filename));
md5 = MessageDigest.getInstance("MD5");
while((len = in.read(buf)) >= 0) {
md5.update(buf, 0, len);
}
in.close();
}
catch (NoSuchAlgorithmException nsae) {
nsae.printStackTrace();
}
catch (IOException ioe) {
System.err.println(ioe.toString());
}

if(len == -1) { // we must've read something
byte[] digest = md5.digest();
StringBuffer hexString = new StringBuffer();
int digestLength = digest.length;

for(int i=0; i < digest.length; i++) {
hexString.append(hexDigit(digest));
}

return hexString.toString();
}
else {
return null;
}
}
}

 

[Michael Borgwardt]

Thanks very much for the response and the help. I'm pretty new, so I
just wanted to make sure that the code below is what you had in mind.

Yes, that's it.

Incidentally, this code definitely drops me down to a 6-8 second
execution vs. 10-15. Thanks a bunch! The -Xprof output seems to show
a lot of MD5.transform execution now, but I suppose that's to be
expected, right?

Yes. It may be that sun's MD5 implementation is not the fastest and
the factors John mentioned definitely also play their part.

 

[Michael Borgwardt]

Thread-local ticks:
70.7% 174 Blocked (of total)

BTW, the above seems to indicate that the app spends a lot of time blocked on IO.
This might improve if the buffer size below is increased:

byte[] buf = new byte[4096];

 

[Mikey]

Thread-local ticks:
70.7% 174 Blocked (of total)

BTW, the above seems to indicate that the app spends a lot of time blocked on IO.
This might improve if the buffer size below is increased:

byte[] buf = new byte[4096];

I tried it at a few different sizes (8192, 16384), but I really didn't
see much affect. In fact, while not exactly scientific, it looked
like my execution times may have gone up in some cases (although it's
possible that something else on my system could've slowed thigs down).
A few samples of -Xprof output seemed to show that the blocked I/O
remained around 70%. Could another class be causing that?

On a side note, thanks so much for all of your help. You've been
great.

 

[Mikey]

Thread-local ticks:
70.7% 174 Blocked (of total)

BTW, the above seems to indicate that the app spends a lot of time blocked on IO.
This might improve if the buffer size below is increased:

byte[] buf = new byte[4096];

scratch my last post. It does seem to be down around 5 seconds
consistently now. Sweet! Still though, I have the 70% blocked. How
can I tell where that's coming from?

# timex /usr/local/scripts/ssm/runssm

Flat profile of 5.11 secs (153 total ticks): main

Interpreted + native Method
2.2% 0 + 1 java.io.FileInputStream.readBytes
2.2% 0 + 1 SystemsSecurityMonitor.<clinit>
2.2% 1 + 0 java.lang.Integer.toString
2.2% 0 + 1 sun.text.resources.LocaleElements.getContents
2.2% 1 + 0 java.util.Properties.load
2.2% 0 + 1 java.io.UnixFileSystem.getBooleanAttributes0
13.3% 2 + 4 Total interpreted

Compiled + native Method
53.3% 24 + 0 sun.security.provider.MD5.transform
2.2% 1 + 0 java.util.regex.Matcher.find
2.2% 1 + 0 sun.security.provider.MD5.engineUpdate
57.8% 26 + 0 Total compiled

Thread-local ticks:
70.6% 108 Blocked (of total)
2.2% 1 Class loader
6.7% 3 Compilation
20.0% 9 Unknown: no last frame


Global summary of 5.12 seconds:
100.0% 156 Received ticks
1.3% 2 Compilation
0.6% 1 Other VM operations
0.6% 1 Class loader
5.8% 9 Unknown code

real 5.87
user 2.53
sys 0.85