Why is JAR so slow?

[Wibble]

done.

See http://mindprod.com/jgloss/compressionutilities.html

In compressing, 7zip shrinks to 42% of original size. Winzip 58%, Jar
68%.

In speed, jar is 68% the speed of Winzip.

All this is under Win2k.

Thanks R

 

[Chris Uppal]

The compression methods in java.util.zip.Deflater are native. How they're
implemented, I can't say.

I believe that it uses the same code as is used in Cygwin (and presumably
Solaris) zip.
http://www.gzip.org/zlib/
http://www.info-zip.org/pub/infozip/Zip.html

I feel the difference Wibble's seeing must due to different file handling
and/or bufferig strategies. Unfortunately, I haven't (yet?) been able to find
the source to the jar program in the platform/JVM source.

I suppose another possibility is that Wibble's Ant tasks are kicking off an
execution of the jar tool for each file that's added (seems unlikely, but you
never know), in which case a large part of the difference would be down to the
difference in startup times between a Cygwin exe (bad), and a JVM-based exe
(much worse).

-- chris

 

[Martin Gregorie]

done.

Useful stuff. Thanks.

In view of the preceding discussion, please consider adding similar
decompression figures. If nothing else, it would be very interesting
indeed to compare the compression/decompression times each utility.

I don't think I'm too surprised that Windows takes longer to copy the
file than zip does to compress it. This is true of any OS on a system
with a fairly quick CPU and a fast compression algorithm. Straight copy
does two disk operations per sector while writing a compressed archive
only does 1.3 to 1.5 disk operations for each sector read.

This also means that, if JAR unpack is optimized, class loading from JAR
files could be faster than from .class files.

I've not seen recent benchmarks, but historically Windows i/o has never
had the world's quickest i/o, primarily because it never used to overlap
peripheral i/o with anything else, i.e. there is no overlap between
devices or with moving the data between the i/o system and the
application. Any OS using this approach will have significantly faster
i/o times with compressed files simply because disk transfers are minimized.

 

[Chris Uppal]

I don't think I'm too surprised that Windows takes longer to copy the
file than zip does to compress it. This is true of any OS on a system
with a fairly quick CPU and a fast compression algorithm. Straight copy
does two disk operations per sector while writing a compressed archive
only does 1.3 to 1.5 disk operations for each sector read.

And even just for input, reducing the amount to data to stream off-disk can be
a win. I've used Windows "compressed folders" before to speed up (by a factor
of nearly two) a program that had to suck in a /lot/ of data.

BTW, another thing that can slow down copying a ZIP file is if you have a virus
checker active. On one system I worked on (where I was using ~ 1GB ZIP files
/compressed/), renaming a ZIP could take tens of minutes while the background
VC scanned through all the entries....

-- chris

 

[Wibble]

Martin Gregorie wrote:




And even just for input, reducing the amount to data to stream off-disk can be
a win. I've used Windows "compressed folders" before to speed up (by a factor
of nearly two) a program that had to suck in a /lot/ of data.

BTW, another thing that can slow down copying a ZIP file is if you have a virus
checker active. On one system I worked on (where I was using ~ 1GB ZIP files
/compressed/), renaming a ZIP could take tens of minutes while the background
VC scanned through all the entries....

-- chris

The problem comes about when jar'ing up lots of small
files, which is typical of our application. The problem
is observed on both Cygwin, and Solaris. Monday I'll
post actual numbers.

Ant does not spawn jar tasks, but uses the same vm
for ant and its subordinate java tasks like jar or javac.
I would expect jar to have a headstart since it
doesn't have to spawn a process. Its no better
without ant.