Why is JAR so slow?

[Wibble]

I've given up on it and started using
gzip. Its about a bazillion times
faster, going from minutes to seconds
for some our large code generated jars.

Anyone else find this?

 

[Ravi]

How will you execute the application getting from gzip?

 

[Roedy Green]

I've given up on it and started using
gzip. Its about a bazillion times
faster, going from minutes to seconds
for some our large code generated jars.

They do quite different things. Gzip compresses an entire file. Jar
prepares many small member entries each separately compressed.

Are you referring to some alternate Jar utility?

Also check out pack200 which compresses an entire uncompressed archive
to get super compression.

 

[Wibble0]

They do quite different things. Gzip compresses an entire file. Jar
prepares many small member entries each separately compressed.

Are you referring to some alternate Jar utility?

Also check out pack200 which compresses an entire uncompressed archive
to get super compression.

I'm talking about the plain old jar utility. I'm creating jar files to
use in a
classpath. I'm not interested that they're executable.

If you add a manifest, gzip makes jar files which java happily accepts.
The jar file creation time is what I find crazy with the jar util that
ships java.

I'm pretty sure that the reason jikes is faster is that it doesn't use
java's jar support but embeds native zip.

 

[IchBin]

I'm talking about the plain old jar utility. I'm creating jar files to
use in a
classpath. I'm not interested that they're executable.

If you add a manifest, gzip makes jar files which java happily accepts.
The jar file creation time is what I find crazy with the jar util that
ships java.

I'm pretty sure that the reason jikes is faster is that it doesn't use
java's jar support but embeds native zip.

Just and aside question? I thought the jikes was not being developed at
IBM any more or they are phasing it out..

--

Thanks in Advance...
IchBin, Pocono Lake, Pa, USA
http://weconsultants.servebeer.com/JHackerAppManager
__________________________________________________________________________

'If there is one, Knowledge is the "Fountain of Youth"'
-William E. Taylor, Regular Guy (1952-)

 

[Wibble0]

Just and aside question? I thought the jikes was not being developed at
IBM any more or they are phasing it out..

--

Thanks in Advance...
IchBin, Pocono Lake, Pa, USA
http://weconsultants.servebeer.com/JHackerAppManager
__________________________________________________________________________

'If there is one, Knowledge is the "Fountain of Youth"'
-William E. Taylor, Regular Guy (1952-)

Jikes is on sourceforge now, no longer ibm.

http://jikes.sourceforge.net/

 

[Wibble0]

They do quite different things. Gzip compresses an entire file. Jar
prepares many small member entries each separately compressed.

Are you referring to some alternate Jar utility?

Also check out pack200 which compresses an entire uncompressed archive
to get super compression.

I meant zip, not gzip. Sorry.

 

[Roedy Green]

If you add a manifest, gzip makes jar files which java happily accepts.
The jar file creation time is what I find crazy with the jar util that
ships java.

To me, GZip means GZIPOutputStream. What does it mean to you?

 

[Roedy Green]

I meant zip, not gzip. Sorry.

You mean winzip, PkZip, 7Zip, WinRar?

Zip utilitiies don't build manifests. I gather you are doing that
manually?

Zip utilities are highly tuned native assembler code. So it is no
surprise that the generic code in jar.exe is much slower.

 

[Wibble0]

You mean winzip, PkZip, 7Zip, WinRar?

Zip utilitiies don't build manifests. I gather you are doing that
manually?

Zip utilities are highly tuned native assembler code. So it is no
surprise that the generic code in jar.exe is much slower.

I'm using solaris & cygwin zip.
I'm creating manifests manually.

I'm exec'ing the zip from ant, as opposed to using ant's jar task.

<macrodef name="fastJar"><!-- jar is slow, so zip -->
<attribute name="jarFile"/>
<attribute name="basedir"/>
<attribute name="javaVersion" default="1.4.2"/>
<attribute name="zipdirs" default="com"/>
<sequential>
<dirname file="@{jarFile}" property="dir_@{jarFile}"/>
<echo message="dbg:fastJar @{jarFile} @{baseDir} @{zipdirs}"/>
<mkdir dir="@{basedir}/META-INF"/>
<property name="metaInf.@{jarFile}" value=" META-INF"/>
<echo file="@{basedir}/META-INF/MANIFEST.MF"
message="Manifest-Version: 1.0${line.separator}Created-By:
@{javaVersion} (Sun Microsystems Inc.)${line.separator}"/>
<echo message="exec: zip -q -r @{jarFile}${metaInf.@{jarFile}}
@{zipdirs}"/>
<dirname property="dir.@{jarFile}" file="@{jarFile}"/>
<mkdir dir="${dir.@{jarFile}}"/>
<tempfile property="tmpJar.@{jarFile}"
destDir="${dir.@{jarFile}}" prefix="tmpJar" suffix=".zip"/>
<exec executable="zip" failonerror="true" dir="@{basedir}">
<arg line="-q -r ${tmpJar.@{jarFile}}${metaInf.@{jarFile}}
@{zipdirs}"/>
</exec>
<move file="${tmpJar.@{jarFile}}" tofile="@{jarFile}"/>
</sequential>
</macrodef>

 

[IchBin]

IchBin wrote:

Jikes is on sourceforge now, no longer ibm.

http://jikes.sourceforge.net/

Thanks for the info..

IchBin, Pocono Lake, Pa, USA
http://weconsultants.servebeer.com/JHackerAppManager
__________________________________________________________________________

'If there is one, Knowledge is the "Fountain of Youth"'
-William E. Taylor, Regular Guy (1952-)

 

[Mike Schilling]

You mean winzip, PkZip, 7Zip, WinRar?

Zip utilitiies don't build manifests. I gather you are doing that
manually?

Zip utilities are highly tuned native assembler code. So it is no
surprise that the generic code in jar.exe is much slower.

By generic you mean "written in C"? All of the file manipulation in
java.util.{jar,zip} is done in native methods.

 

[Wibble]

By generic you mean "written in C"? All of the file manipulation in
java.util.{jar,zip} is done in native methods.

So if its native, why is it soooooo slow?

 

[Stefan Ram]

So if its native, why is it soooooo slow?

The Just-in-time optimizer of the JVM does not apply to
native code.

 

[Roedy Green]

By generic you mean "written in C"? All of the file manipulation in
java.util.{jar,zip} is done in native methods.

the time consuming thing is the compression algorithm. Making this
fast is the key reason to pick one compression utility over another.
See http://mindprod.com/jgloss/compressionutilities.html

Is the compression in jar.exe done in native? If so, it likely is not
hand-tuned assembler though.

 

[Roedy Green]

So if its native, why is it soooooo slow?

It is likely not why it is so slow but why the competition is so fast.
Those assembler algorithms have been fine tuned in a proprietary ways
since the DOS days.

The other thing is how much work did Sun do tuning the compression
code to particular CPUs/platforms. Likely not much. Jar.exe for most
people is adequate. If they were to invest time, they would optimise
the other end, unpacking jars, which is time critical.

I suspect they did native code simply to use an existing C library.
Today's optimisers could probably beat the C code.

 

[Mike Schilling]

the time consuming thing is the compression algorithm. Making this
fast is the key reason to pick one compression utility over another.
See http://mindprod.com/jgloss/compressionutilities.html

Is the compression in jar.exe done in native? If so, it likely is not
hand-tuned assembler though.

The compression methods in java.util.zip.Deflater are native. How they're
implemented, I can't say.

 

[Wibble]

the time consuming thing is the compression algorithm. Making this
fast is the key reason to pick one compression utility over another.
See http://mindprod.com/jgloss/compressionutilities.html

Is the compression in jar.exe done in native? If so, it likely is not
hand-tuned assembler though.

Roedy, can you add jar to you're compression numbers.
I'd be interested in seeing how it stacks up.

 

[Wibble]

It is likely not why it is so slow but why the competition is so fast.
Those assembler algorithms have been fine tuned in a proprietary ways
since the DOS days.

The other thing is how much work did Sun do tuning the compression
code to particular CPUs/platforms. Likely not much. Jar.exe for most
people is adequate. If they were to invest time, they would optimise
the other end, unpacking jars, which is time critical.

I suspect they did native code simply to use an existing C library.
Today's optimisers could probably beat the C code.

We compile to and run out of jars. It was a big time sink
waiting for things to jar. Opening jars is a runtime cost,
but creating them is a development cost, and we all know
development time is valuable.

Unless you're creating lots of jars, a jit optimizer might
not kick in in time unless you were creating alot of jars.

I've never really know what the boundaries and triggers are for the jit
compiler. Will it compile a method while its executing? When
it sees alot of calls to a function?

Sun certainly has access to the solaris zip source. It should have
been easier to snarf that instead of writing a slow version.

 

[Roedy Green]

Roedy, can you add jar to you're compression numbers.
I'd be interested in seeing how it stacks up.

done.

See http://mindprod.com/jgloss/compressionutilities.html

In compressing, 7zip shrinks to 42% of original size. Winzip 58%, Jar
68%.

In speed, jar is 68% the speed of Winzip.

All this is under Win2k.