Zlib::GzipReader doesn't work as expected

[Thomas Wolf]

Hi,
given 2 files:
cat 5lines.txt
5 lines
5 lines
5 lines
5 lines
5 lines

cat more5lines.txt
More 5 lines
More 5 lines
More 5 lines
More 5 lines
More 5 lines

These files are "gzip"ed as follows:
gzip < 5lines.txt > foo.gz
gzip < more5lines.txt >> foo.gz

zcat foo.gz:
5 lines
5 lines
5 lines
5 lines
5 lines
More 5 lines
More 5 lines
More 5 lines
More 5 lines
More 5 lines

This ruby code only reads the first 5 lines:
#!/usr/bin/ruby
require "zlib"
filename = ARGV[0]

Zlib::GzipReader.open(filename) {|gz|
print gz.read
}

../test.rb foo.gz
5 lines
5 lines
5 lines
5 lines
5 lines

How do I force Zlib::GzipReader do read the whole file?

ruby versions: 1.8.7 and 1.9.0

Thanks and regards,
Thomas Wolf

 

[Robert Klemme]

Hi,
given 2 files:
cat 5lines.txt
5 lines
5 lines
5 lines
5 lines
5 lines

cat more5lines.txt
More 5 lines
More 5 lines
More 5 lines
More 5 lines
More 5 lines

These files are "gzip"ed as follows:
gzip < 5lines.txt > foo.gz
gzip < more5lines.txt >> foo.gz

zcat foo.gz:
5 lines
5 lines
5 lines
5 lines
5 lines
More 5 lines
More 5 lines
More 5 lines
More 5 lines
More 5 lines

This ruby code only reads the first 5 lines:
#!/usr/bin/ruby
require "zlib"
filename = ARGV[0]

Zlib::GzipReader.open(filename) {|gz|
print gz.read
}

./test.rb foo.gz
5 lines
5 lines
5 lines
5 lines
5 lines

How do I force Zlib::GzipReader do read the whole file?

That's a fairly common limitation of GZip libs (Java's standard lib also
has this limitation, or at least hat last time I checked).

You might get away with wrapping the GzipReader around an open IO object
and wrapping another GzipReader when the first finishes.

Kind regards

robert

 

[Simon Krahnke]

These files are "gzip"ed as follows:
gzip < 5lines.txt > foo.gz
gzip < more5lines.txt >> foo.gz

So you have two streams of gzipped data in foo.gz.

And the ruby library reads only the first one.

How do I force Zlib::GzipReader do read the whole file?

I don't know, read the source.

mfg, simon .... l

 

[Simon Krahnke]

You might get away with wrapping the GzipReader around an open IO object
and wrapping another GzipReader when the first finishes.

Like this:

,----[ gz.rb ]
| #!/usr/bin/env ruby
|
| require 'zlib'
| require 'pp'
|
| filename = *ARGV
|
| File.open filename do | f |
| gz1 = Zlib::GzipReader.new(f)
| pp gz1.read
| pp Zlib::GzipReader.new(f).read
| end
`----

Doesn't work.

mfg, simon .... l

 

[Thomas Wolf]

Am 25.04.2012 21:03, schrieb Robert Klemme:

That's a fairly common limitation of GZip libs (Java's standard lib also
has this limitation, or at least hat last time I checked).

You might get away with wrapping the GzipReader around an open IO object
and wrapping another GzipReader when the first finishes.

Thank you.

I found the following thread:
http://www.velocityreviews.com/foru...iple-compressed-blobs-in-a-single-stream.html

and that code works with ruby 1.9.3p0:

require 'stringio'
require 'zlib'

def inflate(filename)
File.open(filename) do |file|
zio = file
loop do
io = Zlib::GzipReader.new zio
puts io.read
unused = io.unused
io.finish
break if unused.nil?
zio.pos -= unused.length
end
end
end

inflate "foo.gz"

Regards,
Thomas

 

[Simon Krahnke]

require 'stringio'

This is unneeded.

require 'zlib'

def inflate(filename)
File.open(filename) do |file|
zio = file

You could just use | zio | instead of |file| and get rid of the
assignment.

loop do
io = Zlib::GzipReader.new zio
puts io.read

puts here will put another "\n" at the end of the output, use print
instead.

unused = io.unused
io.finish
break if unused.nil?
zio.pos -= unused.length
end
end
end

inflate "foo.gz"

Note that as said in the thread this works only for files and other
seekable sources.

So "(seq 1 5 | gzip; seq 6 10 | gzip) | yourscript.rb" won't work.

mfg, simon .... hth