greatly differing processing time between java and Linux while calculating hashes?

Discussion in 'Java' started by qwertmonkey@syberianoutpost.ru, Sep 9, 2012.

  1. Guest

    ~
    I have been noticing great differences (> 25%) in the processing time of
    hashes between java and the implementations in the OS, which cannot attributed
    to block size (I played with it as well). What I consistently got was the java
    is much faster for sha512 and sha384, but then for sha256, sha1 and md5 Linux
    becomes then much faster ...
    ~
    So I grabbed two relatively large media files from youtube:
    ~
    $ ls -l DQfUaXLk_sw.mp4
    -rw-r--r-- 1 knoppix knoppix 628588285 Apr 1 18:09 DQfUaXLk_sw.mp4

    $ ls -l 0buBJlPo9us.flv
    -rw-r--r-- 1 knoppix knoppix 475965918 Aug 31 2010 0buBJlPo9us.flv
    ~
    Quickly coded some test:
    ~
    import java.util.Iterator;
    import java.util.Set;
    import java.util.HashSet;

    import java.io.File;
    import java.io.FileInputStream;
    import java.io.FileNotFoundException;
    import java.io.IOException;

    import java.security.MessageDigest;
    import java.security.Security;
    import java.security.Provider;
    import java.security.NoSuchAlgorithmException;

    // __
    public class CheckSum00Test{
    // __
    private static final String aLnSep = System.getProperty("line.separator");
    // __
    private static final int _BLCK_SZ = 512;

    // __
    private static final String[] aSecKeys = new String[]{
    "MessageDigest.", "Alg.Alias.MessageDigest."
    };
    // __
    private static final String[] aSecPrvdrs = getProviders();
    // __
    private static final String[] getProviders() throws SecurityException{
    String[] aArSec = null;
    // __
    Set<String> SSPrvdrs = new HashSet<String>();
    boolean IsPrvdr;
    Iterator Itr;
    int iPrvdrs = Integer.MIN_VALUE, iPrvdrPrfxL = aSecKeys.length, iSSz;
    String aSecKey;
    Provider[] SecPrvdr = Security.getProviders();
    boolean IsSystemPrvdrs = ((SecPrvdr != null) &&
    ((iPrvdrs = SecPrvdr.length) > 0));
    if(IsSystemPrvdrs){
    for(int i = 0; (i < iPrvdrs); ++i){
    Itr = SecPrvdr.keySet().iterator();
    while(Itr.hasNext()){
    String aKey = (String)Itr.next();
    IsPrvdr = false;
    for(int j = 0; (j < iPrvdrPrfxL) && !IsPrvdr; ++j){
    IsPrvdr = aKey.startsWith(aSecKeys[j]);
    // __
    if(IsPrvdr){
    aSecKey = aKey.split(" ")[0];
    if(aSecKey != null){
    aSecKey = aSecKey.trim();
    aSecKey = aSecKey.substring(aSecKeys[j].length());
    if(aSecKey.length() > 0){ SSPrvdrs.add(aSecKey); }
    }// (aSecKey != null)
    }// (IsPrvdr)
    }// j [0, iPrvdrPrfxL)
    }// (Itr.hasNext()){
    }// i [0, iPrvdrs)
    // __
    IsSystemPrvdrs = ((iSSz = SSPrvdrs.size()) > 0);
    if(IsSystemPrvdrs){
    aArSec = new String[iSSz];
    SSPrvdrs.toArray(aArSec);
    }
    }
    // __
    if(!IsSystemPrvdrs){
    String aX = "FATAL ERROR: java implementation does NOT seem to
    include Security Providers!" + aLnSep;
    aX += "// __ java.version: " + System.getProperty("java.version") + aLnSep;
    // ...
    aX += "~";
    throw new SecurityException(aLnSep + "// __ " + aX + aLnSep);
    }
    // __
    return(aArSec);
    }

    // __
    private static String getErrSecAlgos(){
    StringBuilder aBldr = new StringBuilder();
    aBldr.append(aLnSep + "// __ The " + aSecPrvdrs.length + " checksum
    algorithms that could be used are: " + aLnSep + aLnSep);
    for(int i = 0; (i < aSecPrvdrs.length); ++i){ aBldr.append((i + 1) + ": "
    + aSecPrvdrs + aLnSep); }
    return(aBldr.toString());
    }

    // __
    public static void main(String[] args){
    String aKNm = "CheckSum00Test";
    if((args != null) && (args.length == 2)){
    try{
    long lTm00 = System.currentTimeMillis();
    // __
    MessageDigest md = MessageDigest.getInstance(args[0]);
    // __
    FileInputStream fis = new FileInputStream(args[1]);
    byte[] dataBytes = new byte[_BLCK_SZ];
    int nread = 0;
    while ((nread = fis.read(dataBytes)) != -1) { md.update(dataBytes,
    0, nread); };
    fis.close();
    byte[] mdbytes = md.digest();
    long lTm02 = System.currentTimeMillis();
    // __ byte2hex
    StringBuilder aBldr = new StringBuilder();
    for (int i = 0; i < mdbytes.length; i++) {
    aBldr.append(Integer.toString((mdbytes & 0xff) + 0x100, 16)
    ..substring(1));
    }
    System.err.println("// __ " + args[0] + " encrypting \"" + args[1]
    + "\":\"" + aBldr.toString() + "\" took: " +(lTm02 - lTm00) + " (ms)");
    }catch(FileNotFoundException FlNtFndX){ FlNtFndX.
    printStackTrace(System.err); }
    catch(IOException IOX){ IOX.printStackTrace(System.err); }
    catch(NoSuchAlgorithmException NSekAlgoX){
    // System.err.println(getErrSecAlgos());
    NSekAlgoX.printStackTrace(System.err);
    }
    }// ((args != null) && (args.length == 2))
    else{
    System.err.println(aLnSep + "// __ usage: java " + aKNm + "
    <sum algorithm> <input file>" + aLnSep);
    System.err.println(getErrSecAlgos());
    }
    }
    }
    ~
    then I went:
    ~
    date; time java CheckSum00Test SHA-512 <input_file>; date;
    date; time sha512sum -b <input_file>; date;

    date; time java CheckSum00Test SHA-384 <input_file>; date;
    date; time sha384sum -b <input_file>; date;

    date; time java CheckSum00Test SHA-256 <input_file>; date;
    date; time sha256sum -b <input_file>; date;

    date; time java CheckSum00Test SHA-1 <input_file>; date;
    date; time sha1sum -b <input_file>; date;

    date; time java CheckSum00Test MD5 <input_file>; date;
    date; time md5sum -b <input_file>; date;
    ~
    What do you think is going on here?
    ~
    thanks
    lbrtchx
    , Sep 9, 2012
    #1
    1. Advertising

  2. Roedy Green Guest

    On Sun, 9 Sep 2012 02:43:43 +0000 (UTC),
    wrote, quoted or indirectly quoted
    someone who said :

    I think you will find the function in handled by native methods. So it
    has little to do with the difference between Java and C. Factors:
    1. how optimised the code is for your CPU
    2. how smart the C compiler is.
    3. how big your blocks are. You have to traverse the "blood-brain"
    barrier for each call.

    --
    Roedy Green Canadian Mind Products http://mindprod.com
    A new scientific truth does not triumph by convincing its opponents and making them see the light,
    but rather because its opponents eventually die, and a new generation grows up that is familiar with it.
    ~ Max Planck 1858-04-23 1947-10-04
    Roedy Green, Sep 9, 2012
    #2
    1. Advertising

  3. markspace Guest

    Re: greatly differing processing time between java and Linux whilecalculating hashes?

    On 9/8/2012 7:43 PM, wrote:

    > $ ls -l DQfUaXLk_sw.mp4
    > -rw-r--r-- 1 knoppix knoppix 628588285 Apr 1 18:09 DQfUaXLk_sw.mp4
    >
    > $ ls -l 0buBJlPo9us.flv
    > -rw-r--r-- 1 knoppix knoppix 475965918 Aug 31 2010 0buBJlPo9us.flv


    I think we could use some URLs here. I doubt I'd be able find those
    files by name.


    > date; time java CheckSum00Test MD5 <input_file>; date;
    > date; time md5sum -b <input_file>; date;
    > ~
    > What do you think is going on here?
    > ~



    Well for starters I think your constant use of ~ as a line separator is
    pretty annoying. And secondly you posted your homework file with
    <input_file> instead of the actual file name, and no results either, so
    it's pretty hard to say what the result you obtained was, let alone
    guess at a cause.
    markspace, Sep 9, 2012
    #3
  4. Lew Guest

    Re: greatly differing processing time between java and Linux whilecalculating hashes?

    markspace wrote:
    > qwertmonkey wrote:
    > > ~
    > > What do you think is going on here?
    > > ~

    >
    > Well for starters I think your constant use of ~ as a line separator is
    > pretty annoying.


    +1

    Why do you do that, OP?

    --
    Lew
    Lew, Sep 9, 2012
    #4
  5. markspace Guest

    Re: greatly differing processing time between java and Linux whilecalculating hashes?

    On 9/9/2012 12:17 PM, Patricia Shanahan wrote:

    > I am generally not fussy about posting style....
    > However, I've stayed out of this discussion because of two problems,
    > fragmented threads and articles with unreadable formatting.



    I feel the same way. I don't like harping on trivialities or oddities
    of posting style or phrasing, but in this case it really is making it
    harder for me to read. I just noticed the fragmented posts too.

    OP: You may not be very concerned about this, but your posts seem to be
    making it harder than needed for folks to give you additional input.
    I'm just making you aware of the issue, just in case you were not aware
    before. It would help us if your posts were not fragmented in our
    newsreaders (Google groups is a poor UI and known to be broken), and the
    ~ thing is really aggravating.
    markspace, Sep 9, 2012
    #5
  6. Re: greatly differing processing time between java and Linux whilecalculating hashes?

    On 09.09.2012 04:43, wrote:

    > What do you think is going on here?


    As far as I could extract from that code (which I find pretty badly
    readable btw.) you are measuring digest calculation and IO. What
    measures did you take to ensure there are no effects from OS buffering?
    Also, why are you comparing apples (Java) and oranges (Linux) - at
    least in the subject? Are you aware that the JVM has some startup time
    which can be significant when measuring run once applications?

    And, why the heck, are there still people around who write a return like
    a method call - with brackets?

    ....

    robert



    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
    Robert Klemme, Sep 9, 2012
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page