My app runs too slow (lots of code follows)

Discussion in 'Java' started by Mikey, Dec 11, 2003.

  1. Mikey

    Mikey Guest

    Let me start out by saying that I'm admittedly not a great programmer.
    I'm sure that my problems are my own fault and not really Java's. I
    have a program that I had originally written in Perl. For various
    reasons I decided to rewrite it in Java, but now that I'm nearly done,
    I'm not sure if I want to use it because it's very slow and resource
    intensive compared to my Perl version.

    Quick summary of what the program does: it performs MD5 sums on a
    number of files, stores the sums and compares against them on future
    runs -- sort of a cheesy tripwire, checks that some processes are
    running, checks the contents of a few small files like /etc/passwd,
    /etc/shadow, etc. It's strictly a Unix program for me.

    My old perl version generally runs the comparisons, checks file
    permissions, reads a couple of files, along with reading the last few
    minutes of a few logfiles (something I have yet to really implement in
    my new Java version, which will make it run even longer, of course) in
    less than a second. My java version does it (minus the logfile
    reading) in anywhere between 10 and 15 seconds.

    Here's my -Xprof output:

    Flat profile of 0.74 secs (37 total ticks): process reaper

    Thread-local ticks:
    100.0% 37 Unknown: no last frame


    Flat profile of 0.69 secs (36 total ticks): process reaper

    Thread-local ticks:
    100.0% 36 Unknown: no last frame


    Flat profile of 0.47 secs (24 total ticks): process reaper

    Thread-local ticks:
    100.0% 24 Unknown: no last frame


    Flat profile of 0.79 secs (41 total ticks): process reaper

    Thread-local ticks:
    100.0% 41 Unknown: no last frame


    Flat profile of 10.98 secs (324 total ticks): main

    Interpreted + native Method
    1.2% 0 + 2 java.io.UnixFileSystem.getBooleanAttributes0
    1.2% 0 + 2 java.lang.System.arraycopy
    0.6% 1 + 0 java.io.File.getName
    0.6% 0 + 1 ssm.FileStat.getOwner
    0.6% 0 + 1 java.util.ResourceBundle.<clinit>
    0.6% 1 + 0 java.io.BufferedInputStream.fill
    0.6% 1 + 0 sun.nio.cs.US_ASCII$Decoder.decodeArrayLoop
    0.6% 0 + 1 java.io.FileInputStream.open
    5.8% 3 + 7 Total interpreted

    Compiled + native Method
    34.7% 59 + 1 java.io.ByteArrayOutputStream.write
    27.2% 47 + 0 java.io.BufferedInputStream.read
    18.5% 32 + 0 sun.security.provider.MD5.transform
    1.2% 2 + 0 java.util.Arrays.mergeSort
    1.2% 2 + 0 ssm.MD5Sum.loadByteData
    0.6% 1 + 0 java.lang.StringBuffer.length
    0.6% 1 + 0 ssm.UnixCrypt.D_ENCRYPT
    0.6% 1 + 0 java.lang.String.toString
    84.4% 145 + 1 Total compiled

    Stub + native Method
    0.6% 0 + 1 java.lang.System.arraycopy
    0.6% 0 + 1 java.io.FileInputStream.readBytes
    1.2% 0 + 2 Total stub

    Thread-local ticks:
    46.6% 151 Blocked (of total)
    0.6% 1 Class loader
    7.5% 13 Unknown: no last frame
    0.6% 1 Unknown: thread_state


    Flat profile of 0.00 secs (1 total ticks): Thread-1

    Interpreted + native Method
    100.0% 1 + 0
    java.security.AccessController.getStackAccessControlContext
    100.0% 1 + 0 Total interpreted


    Global summary of 11.01 seconds:
    100.0% 327 Received ticks
    0.3% 1 Received GC ticks
    0.6% 2 Compilation
    0.3% 1 Class loader
    46.5% 152 Unknown code


    And, because I'm not really sure what to point anyone at, I'll just
    attach
    the bulk of my code, sorry...

    main:=========================================================
    import ssm.*;
    import java.io.*;
    import java.util.*;
    import java.net.InetAddress;
    import java.text.SimpleDateFormat;

    class SystemsSecurityMonitor {
    private static HashManager hm = new HashManager();
    private static FileListManager flm = new FileListManager();
    private static SendMail sm = new SendMail();
    private static StringBuffer emailtext = new StringBuffer();
    private static ProcessMonitor pm = new ProcessMonitor();
    private static SimpleDateFormat df = new SimpleDateFormat();
    private static GregorianCalendar cal = new GregorianCalendar();
    private static GregorianCalendar tm = new GregorianCalendar();
    private static final long MILLIS_PER_MINUTE = 60000;
    private static String earliest = null;

    public static void printUsage() {
    System.out.println("Usage: [-f <file>] [-i <min>] [-h] [-s]
    [-u] [-p]" +
    "\n\nwhere min is the number of minutes in between\n" +
    "executions of this program\n\n" +
    "Default values can be found below between the <>\n\n" +
    " -f -- specify the file to use as the md5 database
    </.md5db>\n" +
    " -i -- interval between executions <5>\n" +
    " -h -- help (this screen)\n" +
    " -p -- print the list of files to look at\n" +
    " -s -- display md5 sums of system binaries and exit\n" +
    " -u -- update the md5 database and exit\n" +
    " -v -- verbose mode\n\n");
    }

    public static String goBackIntervalMinutes(String dateFormat) {
    int min = Integer.parseInt(System.getProperty("interval")) +
    1;

    df.applyPattern(dateFormat);
    tm.setTime(cal.getTime());
    tm.setTimeInMillis(cal.getTimeInMillis() - (MILLIS_PER_MINUTE
    * min));
    return df.format(tm.getTime());
    }

    // public static boolean isEarlierThanWeWant(String date) {
    // try {


    public static void printFileList() {
    String[] list = flm.getFileList();

    System.out.println("Listing files to inspect:\n");
    for(int i = 0; i < list.length; i++) {
    System.out.println("\to " + list);
    }
    }

    public static void showSums() {
    hm.getHashProperties().list(System.out);
    }

    public static void updateSums(String filename) {
    try {
    hm.storeHash(filename);
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }
    }

    public static void compareSums() {
    String storedhash, newhash = null;
    Properties p1 = null;

    try {
    p1 = hm.getStoredHash(System.getProperty("md5file"));
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }

    for(int i = 0; i < flm.getFileList().length; i++) {
    String theFile = flm.getFileList();
    storedhash = p1.getProperty(theFile);
    try {
    newhash = hm.hashFile(theFile);
    }
    catch (IOException ioe) { /* and ignore */ }

    if(! newhash.equals(storedhash)) {
    emailtext.append(
    "--------------------------------------------------------\n"+
    "Incorrect file info for file: " + theFile + "\n" +
    "Stored Sum: " + storedhash + "\n" +
    "Actual Sum: " + newhash + "\n" +
    "--------------------------------------------------------\n");
    }
    }
    }

    public static boolean procIsRunning(String procName) {
    boolean isRunning = false;
    int retval = -2;

    try {
    retval = pm.isProcessRunning(procName);
    if(retval == -1) {
    System.err.println("Error during isProcessRunning()");
    }
    else if(retval == -2) {
    System.err.println("Error during procIsRunning()");
    }
    else if(retval > 0) {
    isRunning = true;
    }
    }
    catch(IOException ioe) {
    System.err.println(ioe.toString());
    }

    return isRunning;
    }

    public static void checkProcesses() {
    String[] procs = System.getProperty("processes").split(";");

    for(int i = 0; i < procs.length; i++) {
    if(! procIsRunning(procs)) {
    emailtext.append("Process not found: " + procs);
    }
    }
    }

    public static void checkFtpUsers() {
    String line;
    String ftpusers = System.getProperty("files.ftpusers");
    String[] users = System.getProperty("users.noftp").split(" ");

    try {
    BufferedReader reader = new BufferedReader(
    new FileReader(new File(ftpusers)));

    while((line = reader.readLine()) != null) {
    for(int i = 0; i < users.length; i++) {
    if(users.equals(line)) {
    users = "";
    }
    }
    }
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }

    for(int i = 0; i < users.length; i++) {
    if(! users.equals("")) {
    emailtext.append(ftpusers + " : user not found: " +
    users + "\n");
    }
    }
    }

    public static void lookForPlusInRhosts() {
    String line;
    String rhosts = System.getProperty("files.rhosts");

    try {
    BufferedReader reader = new BufferedReader(
    new FileReader(new File(rhosts)));

    while((line = reader.readLine()) != null) {
    if(line.matches(".*\\+.*"))
    emailtext.append(rhosts + " : found +\n");
    }
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }
    }

    public static void checkPasswordFile() {
    boolean sawRoot = false;
    String line, user, pass, uid, gid, gecos, home, shell;
    String passwd = System.getProperty("files.passwd");
    String root = System.getProperty("users.root");
    String[] info;
    String[] shells = System.getProperty("shells").split(" ");
    Arrays.sort(shells);

    try {
    BufferedReader reader = new BufferedReader(
    new FileReader(new File(passwd)));

    while((line = reader.readLine()) != null) {
    info = line.split(":");

    user = info[0];
    pass = info[1];
    uid = info[2];
    gid = info[3];
    gecos = info[4];
    home = info[5];

    if(info.length == 7) {
    shell = info[6];
    } else {
    shell = null;
    }

    if(user.equals(root) && uid.equals("0")) {
    sawRoot = true;
    }
    else if(uid.equals("0")){
    emailtext.append(passwd + " : uid=0: " + user);
    }

    if(shell != null && (Arrays.binarySearch(shells,
    shell) < 0))
    emailtext.append(passwd + " : " + user +
    " has invalid shell: " + shell + "\n");
    }
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }

    if(! sawRoot)
    emailtext.append("Never saw root user: " + root);
    }

    public static void checkShadowFile() {
    String line, salt, emptypass;
    String shadow = System.getProperty("files.shadow");
    String[] info;

    try {
    BufferedReader reader = new BufferedReader(
    new FileReader(new File(shadow)));

    while((line = reader.readLine()) != null) {
    salt = null;
    emptypass = null;
    info = line.split(":");

    if(info[1].equals("")) {
    emailtext.append("user has no password: " +
    info[0]);
    }
    else {
    salt = info[1].substring(0, 2);
    emptypass = UnixCrypt.crypt(salt, "");
    if(info[1].equals(emptypass))
    emailtext.append("user hit enter for his
    password: " +
    info[0]);
    }
    }
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }
    }

    public static void setupAppProperties() throws IOException {
    String propfile = "default_settings.properties";

    Properties p = new Properties(System.getProperties());
    String sp = System.getProperty("os.name") +
    "_settings.properties";
    File f = new File("./" + sp);
    if(f.exists())
    propfile = sp;

    FileInputStream props = new FileInputStream(propfile);
    p.load(props);
    props.close();
    System.setProperties(p);

    flm.setFileList(System.getProperties());
    }

    public static void main(String[] args) {
    int ch = -1;

    boolean opt_s = false;
    boolean opt_u = false;
    boolean verbose = false;
    boolean ckpsflag = false;

    String hostname = "unknown";

    GetOpt go = new GetOpt(args, "suhlvi:f:");

    try {
    hostname = InetAddress.getLocalHost().getHostName();
    }
    catch (java.net.UnknownHostException uhe) { /* and ignore */ }

    try {
    setupAppProperties();
    }
    catch (IOException ioe) {
    System.err.println("complication with app properties file:
    " +
    ioe.toString());
    System.exit(1);
    }

    go.optErr = true;
    while ((ch = go.getopt()) != go.optEOF) {
    if((char)ch == 's') {
    opt_s = true;
    }
    else if((char)ch == 'u') {
    opt_u = true;
    }
    else if((char)ch == 'h') {
    printUsage();
    System.exit(0);
    }
    else if((char)ch == 'l') {
    printFileList();
    System.exit(0);
    }
    else if((char)ch == 'v') {
    verbose = true;
    }
    else if((char)ch == 'i') {
    System.setProperty("interval", go.optArgGet());
    }
    else if((char)ch == 'f') {
    System.setProperty("md5file", go.optArgGet());
    }
    else {
    printUsage();
    System.exit(1); // undefined
    option
    }
    }

    try {
    hm.setHashProperties(flm.getFileList());
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }

    if(opt_s) {
    showSums();
    System.exit(0);
    }
    else if(opt_u) {
    updateSums(System.getProperty("md5file"));
    System.exit(0);
    }

    compareSums();
    checkProcesses();
    checkFtpUsers();
    lookForPlusInRhosts();
    checkPasswordFile();
    checkShadowFile();

    if(emailtext.length() > 0) {
    try {
    if(verbose)
    System.out.println(emailtext);
    sm.send("mailhost", "root",
    "SECURITY ALERT: " + hostname,
    emailtext.toString());
    }
    catch (IOException ioe) {
    ioe.printStackTrace();
    }
    }
    }
    }

    md5:=================================================================
    package ssm;

    import java.io.*;
    import java.security.*;

    public class MD5Sum {
    private byte[] loadByteData(String filename) throws IOException {
    BufferedInputStream bis = new BufferedInputStream(
    new FileInputStream(filename));
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    int ch;

    while((ch = bis.read()) != -1) {
    baos.write(ch);
    }

    return baos.toByteArray();
    }

    private String hexDigit(byte x) {
    StringBuffer sb = new StringBuffer();
    char c;

    c = (char)((x >> 4) & 0x0f);
    if(c > 9) {
    c = (char)((c - 10) + 'a');
    }
    else {
    c = (char)(c + '0');
    }

    sb.append(c);

    c = (char)(x & 0xf);
    if(c > 9) {
    c = (char)((c - 10) + 'a');
    }
    else {
    c = (char)(c + '0');
    }

    sb.append(c);
    return sb.toString();
    }

    public String hash_file(String filename) throws IOException {
    MessageDigest md5 = null;
    byte[] content = null;

    try {
    md5 = MessageDigest.getInstance("MD5");
    content = loadByteData(filename);
    }
    catch (NoSuchAlgorithmException nsae) {
    nsae.printStackTrace();
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }

    if(content != null) {
    md5.update(content);
    byte[] digest = md5.digest();
    StringBuffer hexString = new StringBuffer();
    int digestLength = digest.length;

    for(int i=0; i < digest.length; i++) {
    hexString.append(hexDigit(digest));
    }

    return hexString.toString();
    }
    else {
    return null;
    }
    }
    }

    hashmanager:========================================================
    package ssm;

    import java.io.*;
    import java.util.*;
    import ssm.MD5Sum;
    import ssm.FileStat;

    public class HashManager {
    private Properties hash;
    FileStat fs = new FileStat();
    MD5Sum md = new MD5Sum();

    public void setHashProperties(String[] files) throws IOException {
    hash = new Properties();
    String prop;

    for(int i = 0; i < files.length; i++) {
    prop = md.hash_file(files) + "-" + fs.getMode(files)
    +
    "-" + fs.getOwner(files) + ":" +
    fs.getGroup(files);
    hash.setProperty(files, prop);
    }
    }

    public Properties getHashProperties() {
    return hash;
    }

    public Properties getStoredHash(String file) throws IOException {
    FileInputStream store = new FileInputStream(file);
    Properties storedhash = new Properties();
    storedhash.load(store);

    return storedhash;
    }

    public void storeHash(String file) throws IOException {
    FileOutputStream out = new FileOutputStream(file);
    getHashProperties().store(out, null);
    out.close();
    }

    public String hashFile(String file) throws IOException {
    return md.hash_file(file) + "-" + fs.getMode(file) +
    "-" + fs.getOwner(file) + ":" + fs.getGroup(file);
    }
    }

    filelistmanager:=====================================================
    package ssm;

    import java.io.*;
    import java.util.*;

    public class FileListManager {
    private String flseparator = ":";
    private StringBuffer filelist;

    public void setFileList(Properties p) {
    filelist = new StringBuffer();
    String thisElement, fsep, sep, file;
    String[] path = {"/usr/bin", "/usr/sbin", "/sbin"}, bin;

    try {
    sep = p.getProperty("path.separator");
    fsep = p.getProperty("file.separator");
    path = p.getProperty("env.path").split(sep);
    }
    catch (NullPointerException npe) {
    sep = ":";
    fsep = "/";
    System.err.println("Running without default properties?");
    }

    for(Enumeration e = p.propertyNames(); e.hasMoreElements();) {
    thisElement = e.nextElement().toString();

    if(thisElement.startsWith("files.")) {
    file = p.getProperty(thisElement);

    File f = new File(file);
    if(f.exists()) {
    filelist.append(p.getProperty(thisElement) + ":");
    }
    }
    else if(thisElement.equals("binaries")) {
    bin = p.getProperty(thisElement).split(" ");

    for(int i = 0; i < path.length; i++) {
    for(int j = 0; j < bin.length; j++) {
    String thisbinary = path + fsep + bin[j];
    File b = new File(thisbinary);
    if(b.exists()) {
    filelist.append(thisbinary + ":");
    }
    }
    }
    }
    else if(thisElement.equals("shells")) {
    bin = p.getProperty(thisElement).split(" ");

    for(int i = 0; i < bin.length; i++) {
    File f = new File(bin);
    if(f.exists()) {
    filelist.append(bin + ":");
    }
    }
    }
    }
    }

    public String[] getFileList() {
    String[] flist = filelist.toString().split(flseparator);
    Arrays.sort(flist);
    return flist;
    }
    }
     
    Mikey, Dec 11, 2003
    #1
    1. Advertising

  2. Mikey wrote:
    > My old perl version generally runs the comparisons, checks file
    > permissions, reads a couple of files, along with reading the last few
    > minutes of a few logfiles (something I have yet to really implement in
    > my new Java version, which will make it run even longer, of course) in
    > less than a second. My java version does it (minus the logfile
    > reading) in anywhere between 10 and 15 seconds.
    >
    > Here's my -Xprof output:

    []
    > Flat profile of 10.98 secs (324 total ticks): main

    []
    > 34.7% 59 + 1 java.io.ByteArrayOutputStream.write
    > 27.2% 47 + 0 java.io.BufferedInputStream.read


    Well, it's pretty obvious where the cycles are being burned, isn't it.

    Here's the culprit:

    > while((ch = bis.read()) != -1) {
    > baos.write(ch);
    > }


    You pump data from the input stream into the byte array one byte at a
    time - horrendously inefficient. Furthermore, the byte array only exists
    to be passed to the MessageDigest, a copy step which is completely
    superfluous.

    use the read() method that takes a buffer byte array as an argument,
    pay attention to the number of bytes read (the return value) and pass
    the filled buffer portion directly to the MessageDigest. Oh, and
    do this with the inputStream directly, without wrapping it into a
    BufferedInputStream, since you already use a buffer.

    This saves you billions of method invocations and two times
    copying your entire data. The resulting code may well be faster than
    the perl code.
     
    Michael Borgwardt, Dec 11, 2003
    #2
    1. Advertising

  3. Mikey wrote:

    > Let me start out by saying that I'm admittedly not a great programmer.
    > I'm sure that my problems are my own fault and not really Java's. I
    > have a program that I had originally written in Perl. For various
    > reasons I decided to rewrite it in Java, but now that I'm nearly done,
    > I'm not sure if I want to use it because it's very slow and resource
    > intensive compared to my Perl version.
    >
    > Quick summary of what the program does: it performs MD5 sums on a
    > number of files, stores the sums and compares against them on future
    > runs -- sort of a cheesy tripwire, checks that some processes are
    > running, checks the contents of a few small files like /etc/passwd,
    > /etc/shadow, etc. It's strictly a Unix program for me.
    >
    > My old perl version generally runs the comparisons, checks file
    > permissions, reads a couple of files, along with reading the last few
    > minutes of a few logfiles (something I have yet to really implement in
    > my new Java version, which will make it run even longer, of course) in
    > less than a second. My java version does it (minus the logfile
    > reading) in anywhere between 10 and 15 seconds.


    Michael Borgwardt offered a well targetted suggestion for improving your
    code's performance. You should also be aware, however, that any Java
    program

    (a) suffers a one-time performance penalty due to JVM startup. This can
    be negligible or a few seconds, depending on many factors.

    (b) speeds up as it runs, because of just-in-time compilation.

    (c) has a memory footprint that includes the JVM's, typically several Mb.

    This means that Java is not an ideal choice for an app that must perform
    a brief task, once, as quickly as possible, nor for an app that must
    have a minimal memory footprint.

    Of course, Perl scripts suffer from Perl's load time and memory
    footprint unless they are compiled down to machine code -- but in that
    case they aren't really Perl any more.


    John Bollinger
     
    John C. Bollinger, Dec 11, 2003
    #3
  4. Mikey

    Mikey Guest

    Michael Borgwardt <> wrote in message news:<bra2u1$118g1$-berlin.de>...
    >
    > use the read() method that takes a buffer byte array as an argument,
    > pay attention to the number of bytes read (the return value) and pass
    > the filled buffer portion directly to the MessageDigest. Oh, and
    > do this with the inputStream directly, without wrapping it into a
    > BufferedInputStream, since you already use a buffer.
    >
    > This saves you billions of method invocations and two times
    > copying your entire data. The resulting code may well be faster than
    > the perl code.



    Thanks very much for the response and the help. I'm pretty new, so I
    just wanted to make sure that the code below is what you had in mind.
    Incidentally, this code definitely drops me down to a 6-8 second
    execution vs. 10-15. Thanks a bunch! The -Xprof output seems to show
    a lot of MD5.transform execution now, but I suppose that's to be
    expected, right?


    Flat profile of 0.72 secs (36 total ticks): process reaper

    Thread-local ticks:
    100.0% 36 Unknown: no last frame


    Flat profile of 0.49 secs (22 total ticks): process reaper

    Thread-local ticks:
    100.0% 22 Unknown: no last frame


    Flat profile of 0.83 secs (41 total ticks): process reaper

    Thread-local ticks:
    100.0% 41 Unknown: no last frame


    Flat profile of 1.13 secs (56 total ticks): process reaper

    Thread-local ticks:
    100.0% 56 Unknown: no last frame


    Flat profile of 6.69 secs (246 total ticks): main

    Interpreted + native Method
    5.6% 0 + 4 java.io.UnixFileSystem.getBooleanAttributes0
    1.4% 1 + 0 ssm.UnixCrypt.des_set_key
    1.4% 0 + 1 ssm.HashManager.setHashProperties
    1.4% 1 + 0 java.lang.String.<init>
    1.4% 0 + 1
    java.text.AttributedCharacterIterator$Attribute.<init>
    1.4% 1 + 0 java.nio.ByteBuffer.<init>
    1.4% 1 + 0 java.nio.HeapCharBuffer.<init>
    1.4% 0 + 1 ssm.FileStat.getGroup
    15.3% 4 + 7 Total interpreted

    Compiled + native Method
    50.0% 36 + 0 sun.security.provider.MD5.transform
    2.8% 2 + 0 java.util.regex.Pattern$Single.match
    1.4% 0 + 1
    java.util.jar.Manifest$FastInputStream.readLine
    1.4% 1 + 0 sun.security.provider.MD5.engineUpdate
    1.4% 1 + 0 vtable chunks
    56.9% 40 + 1 Total compiled

    Stub + native Method
    1.4% 0 + 1 java.io.FileInputStream.readBytes
    1.4% 0 + 1 Total stub

    Thread-local ticks:
    70.7% 174 Blocked (of total)
    8.3% 6 Class loader
    4.2% 3 Compilation
    13.9% 10 Unknown: no last frame


    Global summary of 6.74 seconds:
    100.0% 251 Received ticks
    0.4% 1 Received GC ticks
    0.8% 2 Compilation
    0.4% 1 Other VM operations
    2.4% 6 Class loader
    65.7% 165 Unknown code

    md5sum:========================================================
    package ssm;

    import java.io.*;
    import java.security.*;

    public class MD5Sum {
    MessageDigest md5 = null;
    FileInputStream in = null;

    private String hexDigit(byte x) {
    StringBuffer sb = new StringBuffer();
    char c;

    c = (char)((x >> 4) & 0x0f);
    if(c > 9) {
    c = (char)((c - 10) + 'a');
    }
    else {
    c = (char)(c + '0');
    }

    sb.append(c);

    c = (char)(x & 0xf);
    if(c > 9) {
    c = (char)((c - 10) + 'a');
    }
    else {
    c = (char)(c + '0');
    }

    sb.append(c);
    return sb.toString();
    }

    public String hash_file(String filename) throws IOException {
    byte[] buf = new byte[4096];
    int len = 0;

    try {
    in = new FileInputStream(new File(filename));
    md5 = MessageDigest.getInstance("MD5");
    while((len = in.read(buf)) >= 0) {
    md5.update(buf, 0, len);
    }
    in.close();
    }
    catch (NoSuchAlgorithmException nsae) {
    nsae.printStackTrace();
    }
    catch (IOException ioe) {
    System.err.println(ioe.toString());
    }

    if(len == -1) { // we must've read something
    byte[] digest = md5.digest();
    StringBuffer hexString = new StringBuffer();
    int digestLength = digest.length;

    for(int i=0; i < digest.length; i++) {
    hexString.append(hexDigit(digest));
    }

    return hexString.toString();
    }
    else {
    return null;
    }
    }
    }
     
    Mikey, Dec 11, 2003
    #4
  5. Mikey wrote:
    > Thanks very much for the response and the help. I'm pretty new, so I
    > just wanted to make sure that the code below is what you had in mind.


    Yes, that's it.

    > Incidentally, this code definitely drops me down to a 6-8 second
    > execution vs. 10-15. Thanks a bunch! The -Xprof output seems to show
    > a lot of MD5.transform execution now, but I suppose that's to be
    > expected, right?


    Yes. It may be that sun's MD5 implementation is not the fastest and
    the factors John mentioned definitely also play their part.
     
    Michael Borgwardt, Dec 11, 2003
    #5
  6. Mikey wrote:

    > Thread-local ticks:
    > 70.7% 174 Blocked (of total)


    BTW, the above seems to indicate that the app spends a lot of time blocked on IO.
    This might improve if the buffer size below is increased:

    > byte[] buf = new byte[4096];
     
    Michael Borgwardt, Dec 12, 2003
    #6
  7. Mikey

    Mikey Guest

    Michael Borgwardt <> wrote in message news:<brbukq$1hh19$-berlin.de>...
    > Mikey wrote:
    >
    > > Thread-local ticks:
    > > 70.7% 174 Blocked (of total)

    >
    > BTW, the above seems to indicate that the app spends a lot of time blocked on IO.
    > This might improve if the buffer size below is increased:
    >
    > > byte[] buf = new byte[4096];


    I tried it at a few different sizes (8192, 16384), but I really didn't
    see much affect. In fact, while not exactly scientific, it looked
    like my execution times may have gone up in some cases (although it's
    possible that something else on my system could've slowed thigs down).
    A few samples of -Xprof output seemed to show that the blocked I/O
    remained around 70%. Could another class be causing that?

    On a side note, thanks so much for all of your help. You've been
    great.
     
    Mikey, Dec 12, 2003
    #7
  8. Mikey

    Mikey Guest

    Michael Borgwardt <> wrote in message news:<brbukq$1hh19$-berlin.de>...
    > Mikey wrote:
    >
    > > Thread-local ticks:
    > > 70.7% 174 Blocked (of total)

    >
    > BTW, the above seems to indicate that the app spends a lot of time blocked on IO.
    > This might improve if the buffer size below is increased:
    >
    > > byte[] buf = new byte[4096];


    scratch my last post. It does seem to be down around 5 seconds
    consistently now. Sweet! Still though, I have the 70% blocked. How
    can I tell where that's coming from?

    # timex /usr/local/scripts/ssm/runssm

    Flat profile of 5.11 secs (153 total ticks): main

    Interpreted + native Method
    2.2% 0 + 1 java.io.FileInputStream.readBytes
    2.2% 0 + 1 SystemsSecurityMonitor.<clinit>
    2.2% 1 + 0 java.lang.Integer.toString
    2.2% 0 + 1 sun.text.resources.LocaleElements.getContents
    2.2% 1 + 0 java.util.Properties.load
    2.2% 0 + 1 java.io.UnixFileSystem.getBooleanAttributes0
    13.3% 2 + 4 Total interpreted

    Compiled + native Method
    53.3% 24 + 0 sun.security.provider.MD5.transform
    2.2% 1 + 0 java.util.regex.Matcher.find
    2.2% 1 + 0 sun.security.provider.MD5.engineUpdate
    57.8% 26 + 0 Total compiled

    Thread-local ticks:
    70.6% 108 Blocked (of total)
    2.2% 1 Class loader
    6.7% 3 Compilation
    20.0% 9 Unknown: no last frame


    Global summary of 5.12 seconds:
    100.0% 156 Received ticks
    1.3% 2 Compilation
    0.6% 1 Other VM operations
    0.6% 1 Class loader
    5.8% 9 Unknown code

    real 5.87
    user 2.53
    sys 0.85
     
    Mikey, Dec 12, 2003
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. SB
    Replies:
    0
    Views:
    397
  2. Replies:
    1
    Views:
    264
    Steve Holden
    Aug 15, 2006
  3. Joe Stateson
    Replies:
    2
    Views:
    478
    Joe Stateson
    Sep 10, 2007
  4. brad
    Replies:
    9
    Views:
    396
    Bruno Desthuilliers
    Jun 19, 2008
  5. coolneo
    Replies:
    9
    Views:
    222
    coolneo
    Jan 30, 2007
Loading...

Share This Page