How to include an external (binary) file into a java program at compile (!) time ???

[Tobias Merler]

I have a java class and a second file which contains some binary data (assume a zip archive or a gif picture).

At the moment when I compile the class I would like to include this file into a variable of this class.
I think of a variable declaration like:

byte[] mybinaryfile = <include dummy.zip>

How can I implement this idea?
Remember again: I don't want to load this file at runtime but this file should be
internal part of the one and only java class.

Tobias

 

[Dave Glasser]

(e-mail address removed) (Tobias Merler) wrote on Fri, 1 Oct 2004 00:14:54 +0200
in comp.lang.java.programmer:

I have a java class and a second file which contains some binary data (assume a zip archive or a gif picture).

At the moment when I compile the class I would like to include this file into a variable of this class.
I think of a variable declaration like:

byte[] mybinaryfile = <include dummy.zip>

How can I implement this idea?
Remember again: I don't want to load this file at runtime but this file should be
internal part of the one and only java class.

The only way I can think of is to declare the file contents as an
array of bytes right in your Java class file, like this:

byte[] mybinary file = {0x65, 0x66, ... };

You could easily write a Java program (or Perl script or whatever)
that would read the binary file and generate the snippet of code for
that declaration.


--
Check out QueryForm, a free, open source, Java/Swing-based
front end for relational databases.

http://qform.sourceforge.net

If you're a musician, check out RPitch Relative Pitch
Ear Training Software.

http://rpitch.sourceforge.net

 

[Jacob]

I have a java class and a second file which contains some binary data (assume a zip archive or a gif picture).

At the moment when I compile the class I would like to include this file into a variable of this class.
I think of a variable declaration like:

byte[] mybinaryfile = <include dummy.zip>

How can I implement this idea?
Remember again: I don't want to load this file at runtime but this file should be
internal part of the one and only java class.

There is really no difference in what you are requesting (textually
import the content of the file into the .java file) and actually
load the file at runtime, other than the latter being more elegant
and better supported by the tools you have available.

If you insist on the former approach you will need to do a
preprocessing step (perl, m4, etc.) as Dave G. suggests.

Otherwise, put your binary file in the classpath (i.e. along
with your class files) and refer to it from you java source like this:

String fileName = "myfile.zip";
String packageName = getClass().getPackage().getName();
String packageLocation = packageName.replace ('.', '/');
String filePath = "/" + packageLocation + "/" + fileName;

InputStream stream = getClass().getResourceAsStream (filePath);

At this point you have a handle to your (yet unloaded) file.
You can then load it into a byte[] using the stream.read() methods.

Remember: By this approach the file is *not* accessed through the
client file system (which I guess is your concern), but from within
the application deployment unit.

 

[Tor Iver Wilhelmsen]

byte[] mybinary file = {0x65, 0x66, ... };

This is immensely inefficient sompared to reading it as a resource
from the classpath. It's the same as

byte[] mybinaryFile = new byte[...];
mybinaryFile[0] = 0x65;
mybinaryFile[1] = 0x66;
....

 

[Michael Borgwardt]

byte[] mybinary file = {0x65, 0x66, ... };

This is immensely inefficient sompared to reading it as a resource
from the classpath.

Most importantly, it will only work for pretty small files because of the 64k
bytecode limitation on method size.

 

[Jean-Francois Briere]

You could download Jatha, which is a preprocessor that generates plain
JAVA sources from JAVA sources with macros.
You can find it in here: http://www.cybercom.net/~kimbly/jatha/

The idea is to transform a JAVA file like the follwing (note the
..jatha extension) into a plain JAVA file without any macro.

File IncludeTest.java.jatha:
----------------------------------------------------------------------
public class IncludeTest {
private static int[] mybinaryfile = @INCLUDE_STREAM(8, dummy.zip);

public static void main(String[] args) {
for (int i = 0; i < mybinaryfile.length; i++) {
System.out.print(mybinaryfile + " ");
if ((i % 20) == 0)
System.out.println();
}
}
}
----------------------------------------------------------------------

In this file there is a macro I created: INCLUDE_STREAM, which is a
custom JAVA class
based on the Jatha library, and which does all the work on reading the
provided binary file
to write its content appropriately in the generated JAVA source:

File macros/INCLUDE_STREAM.java:
----------------------------------------------------------------------
package macros;

import java.io.*;
import jatha.Macro;
import jatha.Expander;

public class INCLUDE_STREAM extends Macro {
static final String LS = System.getProperty("line.separator");

public void expand(String[] args, Writer out, Expander expander) {
if (args.length != 2)
expander.error("INCLUDE_STREAM macro takes two arguments:
startColumn and filename.");

String startColumn = args[0];
String fileName = args[1];
BufferedInputStream bis = null;

try {
int col = Integer.parseInt(startColumn);
byte[] buffer = new byte[20];
int len;

bis = new BufferedInputStream(new
FileInputStream(fileName));
out.write("{" + LS);

while ((len = bis.read(buffer)) != -1) {
tab(out, col);

for (int i = 0; i < len; i++)
out.write(format(buffer) + ", ");

out.write(LS);
}

tab(out, col);
out.write("}");
}
catch (NumberFormatException e) {
expander.error("The startColumn element (" + startColumn +
") is not an int.");
}
catch (IOException e) {
expander.error("IO error while reading file '" + fileName
+ "': " + e.getMessage());
}
finally {
try { bis.close(); }
catch (Exception e) {}
}
}

void tab(Writer out, int col) throws IOException {
for (int i = 0; i < col; i++)
out.write(" ");
}

String format(int n) {
String nStr = Integer.toHexString(n & 0xff);

if (nStr.length() == 1)
nStr = "0" + nStr;

return "0x" + nStr;
}
}
----------------------------------------------------------------------

You simply build this macro JAVA class,
then execute Jatha from the folder where the .jatha file is located:

java jatha.Main

And this will convert IncludeTest.java.jatha into IncludeTest.java:

File IncludeTest.java:
----------------------------------------------------------------------
public class IncludeTest {
private static int[] mybinaryfile = {
0x50, 0x4b, 0x03, 0x04, 0x14, 0x00, 0x00, 0x00, 0x08, 0x00,
0x78, 0x1d, 0x41, 0x31, 0xb0, 0x9a, 0xc9, 0xbb, 0x27, 0x00,
0x00, 0x00, 0x00, 0x00, 0x64, 0x75, 0x6d, 0x6d, 0x79, 0x2e,
0x74, 0x78, 0x74, 0x50, 0x4b, 0x05, 0x06, 0x00, 0x00, 0x00,
0x00, 0x01, 0x00, 0x01, 0x00, 0x37, 0x00, 0x00, 0x00, 0x4e,
0x00, 0x00, 0x00, 0x00, 0x00,
};

public static void main(String[] args) {
for (int i = 0; i < mybinaryfile.length; i++) {
System.out.print(mybinaryfile + " ");
if ((i % 20) == 0)
System.out.println();
}
}
}

 

[Jacob]

You could download Jatha, which is a preprocessor that generates plain
JAVA sources from JAVA sources with macros.

But why?

Compared to accessing this runtime from the deployment unit
(.jar say) this approach seems unnecessary complex.
This goes for the OP problem in particular (as it contains
binary data), but also in general.

 

[Dave Glasser]

byte[] mybinary file = {0x65, 0x66, ... };

This is immensely inefficient sompared to reading it as a resource
from the classpath. It's the same as

byte[] mybinaryFile = new byte[...];
mybinaryFile[0] = 0x65;
mybinaryFile[1] = 0x66;

A couple of points: First, I'm well aware of how to use ClassLoader's
getResource() or getResourceAsStream() methods, but I decided not to
question the OP's reasons when he stated:

Remember again: I don't want to load this file at
runtime but this file should be internal part of the
one and only java class.

and instead give him an answer that, AFAICT, satisfied his
requirement.

Second, although I wouldn't bet money on it, I don't think you're
correct when you say:

It's the same as

byte[] mybinaryFile = new byte[...];
mybinaryFile[0] = 0x65;
mybinaryFile[1] = 0x66;

Since my code snippet (which should have said "mybinaryfile" instead
of "mybinary file") declared the array and initialized it using an
array literal, I believe the array would be created and initialized
entirely at compile time. It would do exactly what the OP wanted, it
just wouldn't do it in the syntactically clean and simple way he was
looking for.

Third, even if you're correct that my code would be equivalent to:

byte[] mybinaryFile = new byte[...];
mybinaryFile[0] = 0x65;
mybinaryFile[1] = 0x66;

it wouldn't be all that expensive compared to reading the same number
of bytes into the same array from a file; in fact it might be less
expensive. Even though populating it from a file may not assign each
array element one at a time (although it may), you wouldn't have all
the buffering and the loading of extra java.io classes and the disk
I/O going on under the covers, using up resources and CPU cycles.


--
Check out QueryForm, a free, open source, Java/Swing-based
front end for relational databases.

http://qform.sourceforge.net

If you're a musician, check out RPitch Relative Pitch
Ear Training Software.

http://rpitch.sourceforge.net

 

[Dave Glasser]

byte[] mybinary file = {0x65, 0x66, ... };

This is immensely inefficient sompared to reading it as a resource
from the classpath.

Most importantly, it will only work for pretty small files because of the 64k
bytecode limitation on method size.

That's only if the OP decided to declare it as a local variable,
rather than an instance or static variable.


--
Check out QueryForm, a free, open source, Java/Swing-based
front end for relational databases.

http://qform.sourceforge.net

If you're a musician, check out RPitch Relative Pitch
Ear Training Software.

http://rpitch.sourceforge.net

 

[Carl Howells]

That's only if the OP decided to declare it as a local variable,
rather than an instance or static variable.

Nope. And that's because you're wrong about the other point. Decompile
a .class file sometime using javap -c, and look at how array constants
are initialized. It's with a series of assignments. Each of those
assignments will be a few bytes of code, too... So you're actually
looking at significantly under 64k as the maximum size for a byte array
declared anywhere in java code.

 

[Carl Howells]

Nope. And that's because you're wrong about the other point. Decompile
a .class file sometime using javap -c, and look at how array constants
are initialized. It's with a series of assignments. Each of those
assignments will be a few bytes of code, too... So you're actually
looking at significantly under 64k as the maximum size for a byte array
declared anywhere in java code.

Ugh. I hate speaking unclearly. ... the maximum size for a byte array
declared *using an array literal* anywhere in java code.

 

[Dave Glasser]

Nope. And that's because you're wrong about the other point. Decompile
a .class file sometime using javap -c, and look at how array constants
are initialized. It's with a series of assignments.

I did, and you're correct. Live and learn.


--
Check out QueryForm, a free, open source, Java/Swing-based
front end for relational databases.

http://qform.sourceforge.net

If you're a musician, check out RPitch Relative Pitch
Ear Training Software.

http://rpitch.sourceforge.net

 

[Tor Iver Wilhelmsen]

I did, and you're correct. Live and learn.

You probably made the assumption that class files have a data segment
like binary code formats (COFF/PE), but they don't - except for string
literals, which end up in a "string pool".

 

[Jean-Francois Briere]

By the way guys, those will eventually produce an error:

byte[] mybinaryfile = {0x65, 0x66, ... };

-or-

byte[] mybinaryFile = new byte[...];
mybinaryFile[0] = 0x65;
mybinaryFile[1] = 0x66;
....

Because in any binary file, you will have some bytes from 0x80 to 0xFF.

The correct code is:

int[] mybinaryfile = {0x65, 0x66, ... };

-or-

int[] mybinaryFile = new int[...];
mybinaryFile[0] = 0x65;
mybinaryFile[1] = 0x66;
....

 

[Yakov]

How can I implement this idea?

Remember again: I don't want to load this file at runtime but this file should be
internal part of the one and only java class.

Tobias

Since it was not explained why the file must be a part of the class,
it's hard to give a proper advice, but here's one of the solutions:

Write a program that reads this file into a variable and then
serializes the class into a file using writeObject(this). Now the
serialized version of this class includes the file. Now you any
program can recreate the instance of this class (that includes the
file) using readObject().

Regards,
Yakov Fain

 

[Ann]

but OP wants to not have any code at runtime to do it

 

[Yakov]

but OP wants to not have any code at runtime to do it

The method that reads this file will be called only once during the
very first run only and never again

Regards,
Yakov Fain