GZIP output and input strange behaviour

Discussion in 'Java' started by Nikk & Jak Anderson, Apr 26, 2004.

  1. Hi,

    I have an applicaiton that stores data sent to it. Storing as ASCII is
    very large, so I decided to write the data to a gzip output stream.
    THis is fine - using gunzip on Solaris, or zip on windows shows all
    the data in the file. However, when I try and read the file in a Java
    app, I only get the first few lines.

    When creting the zip file, I am appending to an existing file - here
    is how I make the input stream (this is compiled using jdk1.4):

    BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new
    GZIPOutputStream(new FileOutputStream(filename, true))));

    bw.write("MyData........etc......\r\n");
    bw.close();


    This works fine, and I get a gzip file that grows as more input is
    added - exactly what I need.

    Now, for reading the file I use the following readers:

    BufferedReader br = new BufferedReader(new InputStreamReader(new
    GZIPInputStream(new FileInputStream(FileObject))));

    I read the data with the following:

    while( (line = br.readLine()) != null){
    System.out.println(""+line);
    }
     
    Nikk & Jak Anderson, Apr 26, 2004
    #1
    1. Advertising

  2. Nikk & Jak Anderson

    Rogan Dawes Guest

    Nikk & Jak Anderson wrote:

    > Hi,
    >
    > I have an applicaiton that stores data sent to it. Storing as ASCII is
    > very large, so I decided to write the data to a gzip output stream.
    > THis is fine - using gunzip on Solaris, or zip on windows shows all
    > the data in the file. However, when I try and read the file in a Java
    > app, I only get the first few lines.
    >
    > When creting the zip file, I am appending to an existing file - here
    > is how I make the input stream (this is compiled using jdk1.4):
    >
    > BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new
    > GZIPOutputStream(new FileOutputStream(filename, true))));
    >
    > bw.write("MyData........etc......\r\n");
    > bw.close();
    >
    >
    > This works fine, and I get a gzip file that grows as more input is
    > added - exactly what I need.
    >
    > Now, for reading the file I use the following readers:
    >
    > BufferedReader br = new BufferedReader(new InputStreamReader(new
    > GZIPInputStream(new FileInputStream(FileObject))));
    >
    > I read the data with the following:
    >
    > while( (line = br.readLine()) != null){
    > System.out.println(""+line);
    > }


    I'm inclined to believe that the Sun GZipInputStream does not support
    concatenation of individual GZipOutputStreams

    effectively what you have in your file is
    GZIP("line 1","line 2", "line 3")GZIP("line 4", "line 5", "line 6")

    When you try to read from it, you take an inputstream, wrap a
    gzipInputStream around it, and read. When the GZipInputStream gets to
    the end of the third line, it considers the stream to be closed, because
    at one point in your code, it WAS explicitly closed, and wrote an "end
    of zip" trailer to say so.

    The reason that you can see all your data if you use the command line
    utilities such as zcat, or unzip, is that they will continue reading if
    there is more data available, and attempt to unzip the remaining data.

    An experiment that may illustrate this for you:

    OutputStream os = new FileOutputStream("myfile");
    OutputStreamWriter osw = new OutputStreamWriter(new GZipOutputStream(os));
    osw.println("line 1");
    osw.close();
    OutputStreamWriter osw2 = new OutputStreamWriter(new GZipOutputStream(os));
    osw2.println("line 1");
    osw2.close();
    os.close();

    InputStream is = new FileInputStream("myfile");
    InputStreamReader isr = new InputStreamReader(new GZipInputStream(is));
    String line = null;
    while ((line = isr.readLine()) != null) {
    System.out.println("Zip1: " + line);
    }
    InputStreamReader isr2 = new InputStreamReader(new GZipInputStream(is));
    while ((line = isr.readLine()) != null) {
    System.out.println("Zip2: " + line);
    }

    Regards,

    Rogan
    --
    Rogan Dawes
    nntp_AT_dawes*DOT*za-DOT-net
     
    Rogan Dawes, Apr 26, 2004
    #2
    1. Advertising

  3. Nikk & Jak Anderson

    Roedy Green Guest

    On 26 Apr 2004 03:08:04 -0700, (Nikk & Jak
    Anderson) wrote or quoted :

    >This works fine, and I get a gzip file that grows as more input is
    >added - exactly what I need.


    don't close the file until you have finished all your writes. flush if
    you must. I don't think append works with gzip. gzip is sort of like
    one element of a zip file without the headers. It has a complex
    structure that must be considered all of a piece. It is not like a
    stream of characters. It is closer to a serialised object.



    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, Apr 26, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    2
    Views:
    445
    Larry I Smith
    May 29, 2005
  2. Paul
    Replies:
    1
    Views:
    431
  3. Mike P2

    GZip output compression with AJAX

    Mike P2, Jun 4, 2007, in forum: ASP .Net
    Replies:
    4
    Views:
    747
    =?Utf-8?B?YmlrZW1pa2U=?=
    Jun 12, 2007
  4. sunrrrise
    Replies:
    2
    Views:
    414
    Chris Angelico
    May 23, 2011
  5. Szymon Rozga

    Trouble with gzip output compression

    Szymon Rozga, May 25, 2006, in forum: Ruby
    Replies:
    1
    Views:
    127
    Robert Klemme
    May 26, 2006
Loading...

Share This Page