GZIP output and input strange behaviour

[Nikk & Jak Anderson]

Hi,

I have an applicaiton that stores data sent to it. Storing as ASCII is
very large, so I decided to write the data to a gzip output stream.
THis is fine - using gunzip on Solaris, or zip on windows shows all
the data in the file. However, when I try and read the file in a Java
app, I only get the first few lines.

When creting the zip file, I am appending to an existing file - here
is how I make the input stream (this is compiled using jdk1.4):

BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new
GZIPOutputStream(new FileOutputStream(filename, true))));

bw.write("MyData........etc......\r\n");
bw.close();


This works fine, and I get a gzip file that grows as more input is
added - exactly what I need.

Now, for reading the file I use the following readers:

BufferedReader br = new BufferedReader(new InputStreamReader(new
GZIPInputStream(new FileInputStream(FileObject))));

I read the data with the following:

while( (line = br.readLine()) != null){
System.out.println(""+line);
}

 

[Rogan Dawes]

Hi,

I have an applicaiton that stores data sent to it. Storing as ASCII is
very large, so I decided to write the data to a gzip output stream.
THis is fine - using gunzip on Solaris, or zip on windows shows all
the data in the file. However, when I try and read the file in a Java
app, I only get the first few lines.

When creting the zip file, I am appending to an existing file - here
is how I make the input stream (this is compiled using jdk1.4):

BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new
GZIPOutputStream(new FileOutputStream(filename, true))));

bw.write("MyData........etc......\r\n");
bw.close();


This works fine, and I get a gzip file that grows as more input is
added - exactly what I need.

Now, for reading the file I use the following readers:

BufferedReader br = new BufferedReader(new InputStreamReader(new
GZIPInputStream(new FileInputStream(FileObject))));

I read the data with the following:

while( (line = br.readLine()) != null){
System.out.println(""+line);
}

I'm inclined to believe that the Sun GZipInputStream does not support
concatenation of individual GZipOutputStreams

effectively what you have in your file is
GZIP("line 1","line 2", "line 3")GZIP("line 4", "line 5", "line 6")

When you try to read from it, you take an inputstream, wrap a
gzipInputStream around it, and read. When the GZipInputStream gets to
the end of the third line, it considers the stream to be closed, because
at one point in your code, it WAS explicitly closed, and wrote an "end
of zip" trailer to say so.

The reason that you can see all your data if you use the command line
utilities such as zcat, or unzip, is that they will continue reading if
there is more data available, and attempt to unzip the remaining data.

An experiment that may illustrate this for you:

OutputStream os = new FileOutputStream("myfile");
OutputStreamWriter osw = new OutputStreamWriter(new GZipOutputStream(os));
osw.println("line 1");
osw.close();
OutputStreamWriter osw2 = new OutputStreamWriter(new GZipOutputStream(os));
osw2.println("line 1");
osw2.close();
os.close();

InputStream is = new FileInputStream("myfile");
InputStreamReader isr = new InputStreamReader(new GZipInputStream(is));
String line = null;
while ((line = isr.readLine()) != null) {
System.out.println("Zip1: " + line);
}
InputStreamReader isr2 = new InputStreamReader(new GZipInputStream(is));
while ((line = isr.readLine()) != null) {
System.out.println("Zip2: " + line);
}

Regards,

Rogan

 

[Roedy Green]

This works fine, and I get a gzip file that grows as more input is
added - exactly what I need.

don't close the file until you have finished all your writes. flush if
you must. I don't think append works with gzip. gzip is sort of like
one element of a zip file without the headers. It has a complex
structure that must be considered all of a piece. It is not like a
stream of characters. It is closer to a serialised object.