greatly differing processing time between java and Linux while calculating hashes?

[qwertmonkey]

Roedy, I did play with the size of the input block used for the input stream
and the files lenghts themselves and the results where the same, so 3 doesn't
seem to be the case.
~
There must be something causing it, of course! but to me a difference
of 25% (and the diff of more than 50% of the flip!) is way too much. Especially
if it is an algorithm that MUST have been optimized (many libraries use them),
besides those checksums algos, all they do is crunching data and they were
developed and maintained by the same coders
~
$ md5sum --version
md5sum (GNU coreutils) 8.13
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Ulrich Drepper, Scott Miller, and David Madore.

$ sha512sum --version
sha512sum (GNU coreutils) 8.13
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Ulrich Drepper, Scott Miller, and David Madore.
~

I think we could use some URLs here ...

~
as I said:
~
youtube.com/watch?v=DQfUaXLk_sw
~
youtube.com/watch?v=0buBJlPo9us
~
you can go:
~
youtube-dl --no-overwrites --continue --no-progress --console-title
--audio-format best --max-quality mp4 ""
~
and here you have your cake ;-)
~
#!/bin/bash

echo "// __ uname -a:"
uname -a

echo "// __ java -version:"
java -version

array=( DQfUaXLk_sw.mp4 0buBJlPo9us.flv )
for i in "${array[@]}"
do
ls -l $i

echo "// __ checksumming using SHA-512"
date; time java CheckSum00Test SHA-512 $i; date;
date; time sha512sum -b $i; date;
echo "~"

echo "// __ checksumming using SHA-384"
date; time java CheckSum00Test SHA-384 $i; date;
date; time sha384sum -b $i; date;
echo "~"

echo "// __ checksumming using SHA-256"
date; time java CheckSum00Test SHA-256 $i; date;
date; time sha256sum -b $i; date;
echo "~"

echo "// __ checksumming using SHA-1"
date; time java CheckSum00Test SHA-1 $i; date;
date; time sha1sum -b $i; date;
echo "~"

echo "// __ checksumming using MD5"
date; time java CheckSum00Test MD5 $i; date;
date; time md5sum -b $i; date;
echo "~"

done

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~


$ sh test_bash_java_dif00.sh
// __ uname -a:
Linux Microknoppix 3.3.7 #38 SMP PREEMPT Tue May 22 06:21:01 CEST 2012 i686
GNU/Linux
// __ java -version:
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) Client VM (build 23.3-b01, mixed mode)
-rw-r--r-- 1 knoppix knoppix 628588285 Apr 1 18:09 DQfUaXLk_sw.mp4
// __ checksumming using SHA-512
Sun Sep 9 00:20:10 UTC 2012
// __ SHA-512 encrypting "DQfUaXLk_sw.mp4":
"f6f2f2b88faf3cef5a5de0cb0bc8f1ed15969490ef785673d3f7e325ab947c8df8b30edf663665
7e945b27eb5202300244b8acef874a11e630b2053f956982d8" took: 45386 (ms)

real 0m45.659s
user 0m43.291s
sys 0m2.083s
Sun Sep 9 00:20:55 UTC 2012
Sun Sep 9 00:20:55 UTC 2012
f6f2f2b88faf3cef5a5de0cb0bc8f1ed15969490ef785673d3f7e325ab947c8df8b30edf6636657
e945b27eb5202300244b8acef874a11e630b2053f956982d8 *DQfUaXLk_sw.mp4

real 1m2.410s
user 1m1.336s
sys 0m0.800s
Sun Sep 9 00:21:58 UTC 2012
~
// __ checksumming using SHA-384
Sun Sep 9 00:21:58 UTC 2012
// __ SHA-384 encrypting "DQfUaXLk_sw.mp4":"41d9cc52c2908f2b70f002fcf64a9365b4
3905173e37b2867e90d9ebb6f95033cd5f0a3126bd6021fa914ae7ba0d4f41"
took: 45131 (ms)

real 0m45.469s
user 0m43.227s
sys 0m1.920s
Sun Sep 9 00:22:43 UTC 2012
Sun Sep 9 00:22:43 UTC 2012
41d9cc52c2908f2b70f002fcf64a9365b43905173e37b2867e90d9ebb6f95033cd5f0a3126bd602
1fa914ae7ba0d4f41 *DQfUaXLk_sw.mp4

real 1m2.677s
user 1m1.476s
sys 0m0.740s
Sun Sep 9 00:23:46 UTC 2012
~
// __ checksumming using SHA-256
Sun Sep 9 00:23:46 UTC 2012
// __ SHA-256 encrypting "DQfUaXLk_sw.mp4":"495656683bbf7d5cac6e5a48853bf18b6f
6e73a782bf0c65dc03bce0b5e814a5" took: 18655 (ms)

real 0m18.978s
user 0m17.409s
sys 0m1.527s
Sun Sep 9 00:24:05 UTC 2012
Sun Sep 9 00:24:05 UTC 2012
495656683bbf7d5cac6e5a48853bf18b6f6e73a782bf0c65dc03bce0b5e814a5
*DQfUaXLk_sw.mp4

real 0m8.509s
user 0m8.093s
sys 0m0.363s
Sun Sep 9 00:24:13 UTC 2012
~
// __ checksumming using SHA-1
Sun Sep 9 00:24:13 UTC 2012
// __ SHA-1 encrypting "DQfUaXLk_sw.mp4":"afc351c4ea13ac59c0181b6fd66ccb21c60b
0c48" took: 13338 (ms)

real 0m13.606s
user 0m12.113s
sys 0m1.523s
Sun Sep 9 00:24:27 UTC 2012
Sun Sep 9 00:24:27 UTC 2012
afc351c4ea13ac59c0181b6fd66ccb21c60b0c48 *DQfUaXLk_sw.mp4

real 0m4.287s
user 0m3.990s
sys 0m0.270s
Sun Sep 9 00:24:31 UTC 2012
~
// __ checksumming using MD5
Sun Sep 9 00:24:31 UTC 2012
// __ MD5 encrypting "DQfUaXLk_sw.mp4":"ecbcbe3d0a2fda4ebfddc41836cee528"
took: 5450 (ms)

real 0m5.729s
user 0m4.123s
sys 0m1.637s
Sun Sep 9 00:24:37 UTC 2012
Sun Sep 9 00:24:37 UTC 2012
ecbcbe3d0a2fda4ebfddc41836cee528 *DQfUaXLk_sw.mp4

real 0m1.878s
user 0m1.467s
sys 0m0.367s
Sun Sep 9 00:24:39 UTC 2012
~
-rw-r--r-- 1 knoppix knoppix 475965918 Aug 31 2010 0buBJlPo9us.flv
// __ checksumming using SHA-512
Sun Sep 9 00:24:39 UTC 2012
// __ SHA-512 encrypting "0buBJlPo9us.flv":"c77b1694fe82f14235a81f32201790a703
3a6d1600446017d66ba1f595ceb54960b3b2322546ee6676eb4f878dd8cd83b974738c560614d85
212bf83c4aee9f3" took: 34330 (ms)

real 0m34.611s
user 0m32.995s
sys 0m1.480s
Sun Sep 9 00:25:14 UTC 2012
Sun Sep 9 00:25:14 UTC 2012
c77b1694fe82f14235a81f32201790a7033a6d1600446017d66ba1f595ceb54960b3b2322546ee6
676eb4f878dd8cd83b974738c560614d85212bf83c4aee9f3 *0buBJlPo9us.flv

real 0m47.409s
user 0m46.517s
sys 0m0.620s
Sun Sep 9 00:26:01 UTC 2012
~
// __ checksumming using SHA-384
Sun Sep 9 00:26:01 UTC 2012
// __ SHA-384 encrypting "0buBJlPo9us.flv":"cbf8770296c529f19786abac324b7a857e
2949ca5248d0fc45d767c27b323686279915da10d6710dfd9156f0fff0f30b"
took: 34892 (ms)

real 0m35.625s
user 0m33.291s
sys 0m1.767s
Sun Sep 9 00:26:37 UTC 2012
Sun Sep 9 00:26:37 UTC 2012
cbf8770296c529f19786abac324b7a857e2949ca5248d0fc45d767c27b323686279915da10d6710
dfd9156f0fff0f30b *0buBJlPo9us.flv

real 0m46.871s
user 0m46.360s
sys 0m0.350s
Sun Sep 9 00:27:24 UTC 2012
~
// __ checksumming using SHA-256
Sun Sep 9 00:27:24 UTC 2012
// __ SHA-256 encrypting "0buBJlPo9us.flv":"a49a51c7a3e1a5fcbe5bca5a7814697842
0fdbb2c70467873512fd215b31a644" took: 14187 (ms)

real 0m14.790s
user 0m13.106s
sys 0m1.403s
Sun Sep 9 00:27:39 UTC 2012
Sun Sep 9 00:27:39 UTC 2012
a49a51c7a3e1a5fcbe5bca5a78146978420fdbb2c70467873512fd215b31a644
*0buBJlPo9us.flv

real 0m6.432s
user 0m6.160s
sys 0m0.250s
Sun Sep 9 00:27:45 UTC 2012
~
// __ checksumming using SHA-1
Sun Sep 9 00:27:45 UTC 2012
// __ SHA-1 encrypting "0buBJlPo9us.flv":"895c3a1c5979ab7d5bacd9eaafcadea40393
9da5" took: 10197 (ms)

real 0m10.466s
user 0m9.216s
sys 0m1.287s
Sun Sep 9 00:27:55 UTC 2012
Sun Sep 9 00:27:55 UTC 2012
895c3a1c5979ab7d5bacd9eaafcadea403939da5 *0buBJlPo9us.flv

real 0m3.270s
user 0m2.970s
sys 0m0.280s
Sun Sep 9 00:27:59 UTC 2012
~
// __ checksumming using MD5
Sun Sep 9 00:27:59 UTC 2012
// __ MD5 encrypting "0buBJlPo9us.flv":"e96442f896fad41903016882b352fdd9"
took: 4238 (ms)

real 0m4.523s
user 0m3.303s
sys 0m1.223s
Sun Sep 9 00:28:03 UTC 2012
Sun Sep 9 00:28:03 UTC 2012
e96442f896fad41903016882b352fdd9 *0buBJlPo9us.flv

real 0m1.421s
user 0m1.127s
sys 0m0.267s
Sun Sep 9 00:28:05 UTC 2012
~

 

[Roedy Green]

Roedy, I did play with the size of the input block used for the input stream
and the files lenghts themselves and the results where the same, so 3 doesn't
seem to be the case.

you want to make sure the size of the byte / char array given to the
digest calculator is big so that you have the overhead of the call as
infrequently as possible. You might want to do a trace to be sure you
are changing that, not just some other buffering.