greatly differing processing time between java and Linux while calculating hashes?

Discussion in 'Java' started by qwertmonkey@syberianoutpost.ru, Sep 9, 2012.

  1. Guest

    Roedy, I did play with the size of the input block used for the input stream
    and the files lenghts themselves and the results where the same, so 3 doesn't
    seem to be the case.
    ~
    There must be something causing it, of course! but to me a difference
    of 25% (and the diff of more than 50% of the flip!) is way too much. Especially
    if it is an algorithm that MUST have been optimized (many libraries use them),
    besides those checksums algos, all they do is crunching data and they were
    developed and maintained by the same coders
    ~
    $ md5sum --version
    md5sum (GNU coreutils) 8.13
    Copyright (C) 2011 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.

    Written by Ulrich Drepper, Scott Miller, and David Madore.

    $ sha512sum --version
    sha512sum (GNU coreutils) 8.13
    Copyright (C) 2011 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.

    Written by Ulrich Drepper, Scott Miller, and David Madore.
    ~
    > I think we could use some URLs here ...

    ~
    as I said:
    ~
    youtube.com/watch?v=DQfUaXLk_sw
    ~
    youtube.com/watch?v=0buBJlPo9us
    ~
    you can go:
    ~
    youtube-dl --no-overwrites --continue --no-progress --console-title
    --audio-format best --max-quality mp4 "http://www.youtube.com/watch?v=...&hd=1"
    ~
    and here you have your cake ;-)
    ~
    #!/bin/bash

    echo "// __ uname -a:"
    uname -a

    echo "// __ java -version:"
    java -version

    array=( DQfUaXLk_sw.mp4 0buBJlPo9us.flv )
    for i in "${array[@]}"
    do
    ls -l $i

    echo "// __ checksumming using SHA-512"
    date; time java CheckSum00Test SHA-512 $i; date;
    date; time sha512sum -b $i; date;
    echo "~"

    echo "// __ checksumming using SHA-384"
    date; time java CheckSum00Test SHA-384 $i; date;
    date; time sha384sum -b $i; date;
    echo "~"

    echo "// __ checksumming using SHA-256"
    date; time java CheckSum00Test SHA-256 $i; date;
    date; time sha256sum -b $i; date;
    echo "~"

    echo "// __ checksumming using SHA-1"
    date; time java CheckSum00Test SHA-1 $i; date;
    date; time sha1sum -b $i; date;
    echo "~"

    echo "// __ checksumming using MD5"
    date; time java CheckSum00Test MD5 $i; date;
    date; time md5sum -b $i; date;
    echo "~"

    done

    ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~


    $ sh test_bash_java_dif00.sh
    // __ uname -a:
    Linux Microknoppix 3.3.7 #38 SMP PREEMPT Tue May 22 06:21:01 CEST 2012 i686
    GNU/Linux
    // __ java -version:
    java version "1.7.0_07"
    Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
    Java HotSpot(TM) Client VM (build 23.3-b01, mixed mode)
    -rw-r--r-- 1 knoppix knoppix 628588285 Apr 1 18:09 DQfUaXLk_sw.mp4
    // __ checksumming using SHA-512
    Sun Sep 9 00:20:10 UTC 2012
    // __ SHA-512 encrypting "DQfUaXLk_sw.mp4":
    "f6f2f2b88faf3cef5a5de0cb0bc8f1ed15969490ef785673d3f7e325ab947c8df8b30edf663665
    7e945b27eb5202300244b8acef874a11e630b2053f956982d8" took: 45386 (ms)

    real 0m45.659s
    user 0m43.291s
    sys 0m2.083s
    Sun Sep 9 00:20:55 UTC 2012
    Sun Sep 9 00:20:55 UTC 2012
    f6f2f2b88faf3cef5a5de0cb0bc8f1ed15969490ef785673d3f7e325ab947c8df8b30edf6636657
    e945b27eb5202300244b8acef874a11e630b2053f956982d8 *DQfUaXLk_sw.mp4

    real 1m2.410s
    user 1m1.336s
    sys 0m0.800s
    Sun Sep 9 00:21:58 UTC 2012
    ~
    // __ checksumming using SHA-384
    Sun Sep 9 00:21:58 UTC 2012
    // __ SHA-384 encrypting "DQfUaXLk_sw.mp4":"41d9cc52c2908f2b70f002fcf64a9365b4
    3905173e37b2867e90d9ebb6f95033cd5f0a3126bd6021fa914ae7ba0d4f41"
    took: 45131 (ms)

    real 0m45.469s
    user 0m43.227s
    sys 0m1.920s
    Sun Sep 9 00:22:43 UTC 2012
    Sun Sep 9 00:22:43 UTC 2012
    41d9cc52c2908f2b70f002fcf64a9365b43905173e37b2867e90d9ebb6f95033cd5f0a3126bd602
    1fa914ae7ba0d4f41 *DQfUaXLk_sw.mp4

    real 1m2.677s
    user 1m1.476s
    sys 0m0.740s
    Sun Sep 9 00:23:46 UTC 2012
    ~
    // __ checksumming using SHA-256
    Sun Sep 9 00:23:46 UTC 2012
    // __ SHA-256 encrypting "DQfUaXLk_sw.mp4":"495656683bbf7d5cac6e5a48853bf18b6f
    6e73a782bf0c65dc03bce0b5e814a5" took: 18655 (ms)

    real 0m18.978s
    user 0m17.409s
    sys 0m1.527s
    Sun Sep 9 00:24:05 UTC 2012
    Sun Sep 9 00:24:05 UTC 2012
    495656683bbf7d5cac6e5a48853bf18b6f6e73a782bf0c65dc03bce0b5e814a5
    *DQfUaXLk_sw.mp4

    real 0m8.509s
    user 0m8.093s
    sys 0m0.363s
    Sun Sep 9 00:24:13 UTC 2012
    ~
    // __ checksumming using SHA-1
    Sun Sep 9 00:24:13 UTC 2012
    // __ SHA-1 encrypting "DQfUaXLk_sw.mp4":"afc351c4ea13ac59c0181b6fd66ccb21c60b
    0c48" took: 13338 (ms)

    real 0m13.606s
    user 0m12.113s
    sys 0m1.523s
    Sun Sep 9 00:24:27 UTC 2012
    Sun Sep 9 00:24:27 UTC 2012
    afc351c4ea13ac59c0181b6fd66ccb21c60b0c48 *DQfUaXLk_sw.mp4

    real 0m4.287s
    user 0m3.990s
    sys 0m0.270s
    Sun Sep 9 00:24:31 UTC 2012
    ~
    // __ checksumming using MD5
    Sun Sep 9 00:24:31 UTC 2012
    // __ MD5 encrypting "DQfUaXLk_sw.mp4":"ecbcbe3d0a2fda4ebfddc41836cee528"
    took: 5450 (ms)

    real 0m5.729s
    user 0m4.123s
    sys 0m1.637s
    Sun Sep 9 00:24:37 UTC 2012
    Sun Sep 9 00:24:37 UTC 2012
    ecbcbe3d0a2fda4ebfddc41836cee528 *DQfUaXLk_sw.mp4

    real 0m1.878s
    user 0m1.467s
    sys 0m0.367s
    Sun Sep 9 00:24:39 UTC 2012
    ~
    -rw-r--r-- 1 knoppix knoppix 475965918 Aug 31 2010 0buBJlPo9us.flv
    // __ checksumming using SHA-512
    Sun Sep 9 00:24:39 UTC 2012
    // __ SHA-512 encrypting "0buBJlPo9us.flv":"c77b1694fe82f14235a81f32201790a703
    3a6d1600446017d66ba1f595ceb54960b3b2322546ee6676eb4f878dd8cd83b974738c560614d85
    212bf83c4aee9f3" took: 34330 (ms)

    real 0m34.611s
    user 0m32.995s
    sys 0m1.480s
    Sun Sep 9 00:25:14 UTC 2012
    Sun Sep 9 00:25:14 UTC 2012
    c77b1694fe82f14235a81f32201790a7033a6d1600446017d66ba1f595ceb54960b3b2322546ee6
    676eb4f878dd8cd83b974738c560614d85212bf83c4aee9f3 *0buBJlPo9us.flv

    real 0m47.409s
    user 0m46.517s
    sys 0m0.620s
    Sun Sep 9 00:26:01 UTC 2012
    ~
    // __ checksumming using SHA-384
    Sun Sep 9 00:26:01 UTC 2012
    // __ SHA-384 encrypting "0buBJlPo9us.flv":"cbf8770296c529f19786abac324b7a857e
    2949ca5248d0fc45d767c27b323686279915da10d6710dfd9156f0fff0f30b"
    took: 34892 (ms)

    real 0m35.625s
    user 0m33.291s
    sys 0m1.767s
    Sun Sep 9 00:26:37 UTC 2012
    Sun Sep 9 00:26:37 UTC 2012
    cbf8770296c529f19786abac324b7a857e2949ca5248d0fc45d767c27b323686279915da10d6710
    dfd9156f0fff0f30b *0buBJlPo9us.flv

    real 0m46.871s
    user 0m46.360s
    sys 0m0.350s
    Sun Sep 9 00:27:24 UTC 2012
    ~
    // __ checksumming using SHA-256
    Sun Sep 9 00:27:24 UTC 2012
    // __ SHA-256 encrypting "0buBJlPo9us.flv":"a49a51c7a3e1a5fcbe5bca5a7814697842
    0fdbb2c70467873512fd215b31a644" took: 14187 (ms)

    real 0m14.790s
    user 0m13.106s
    sys 0m1.403s
    Sun Sep 9 00:27:39 UTC 2012
    Sun Sep 9 00:27:39 UTC 2012
    a49a51c7a3e1a5fcbe5bca5a78146978420fdbb2c70467873512fd215b31a644
    *0buBJlPo9us.flv

    real 0m6.432s
    user 0m6.160s
    sys 0m0.250s
    Sun Sep 9 00:27:45 UTC 2012
    ~
    // __ checksumming using SHA-1
    Sun Sep 9 00:27:45 UTC 2012
    // __ SHA-1 encrypting "0buBJlPo9us.flv":"895c3a1c5979ab7d5bacd9eaafcadea40393
    9da5" took: 10197 (ms)

    real 0m10.466s
    user 0m9.216s
    sys 0m1.287s
    Sun Sep 9 00:27:55 UTC 2012
    Sun Sep 9 00:27:55 UTC 2012
    895c3a1c5979ab7d5bacd9eaafcadea403939da5 *0buBJlPo9us.flv

    real 0m3.270s
    user 0m2.970s
    sys 0m0.280s
    Sun Sep 9 00:27:59 UTC 2012
    ~
    // __ checksumming using MD5
    Sun Sep 9 00:27:59 UTC 2012
    // __ MD5 encrypting "0buBJlPo9us.flv":"e96442f896fad41903016882b352fdd9"
    took: 4238 (ms)

    real 0m4.523s
    user 0m3.303s
    sys 0m1.223s
    Sun Sep 9 00:28:03 UTC 2012
    Sun Sep 9 00:28:03 UTC 2012
    e96442f896fad41903016882b352fdd9 *0buBJlPo9us.flv

    real 0m1.421s
    user 0m1.127s
    sys 0m0.267s
    Sun Sep 9 00:28:05 UTC 2012
    ~
    , Sep 9, 2012
    #1
    1. Advertising

  2. Roedy Green Guest

    On Sun, 9 Sep 2012 04:40:28 +0000 (UTC),
    wrote, quoted or indirectly quoted
    someone who said :

    > Roedy, I did play with the size of the input block used for the input stream
    >and the files lenghts themselves and the results where the same, so 3 doesn't
    >seem to be the case.


    you want to make sure the size of the byte / char array given to the
    digest calculator is big so that you have the overhead of the call as
    infrequently as possible. You might want to do a trace to be sure you
    are changing that, not just some other buffering.
    --
    Roedy Green Canadian Mind Products http://mindprod.com
    A new scientific truth does not triumph by convincing its opponents and making them see the light,
    but rather because its opponents eventually die, and a new generation grows up that is familiar with it.
    ~ Max Planck 1858-04-23 1947-10-04
    Roedy Green, Sep 9, 2012
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page