zlib large files problem

[greg]

I am trying to decompress a 80MB file (decompresses to 300MB), but I
keep getting a buffer error. I thought I had a workaround using
chunks, but it fails with 20M left to go. Any help or suggestions are
greatly appreciated.

in `read': buffer error (Zlib::BufError)


class Zlib::GzipReader
def each_chunk(chunk_size=1024)
yield read(chunk_size) until eof
end
end

gz = Zlib::GzipReader.open(zip_file)

File.open( non_zip_file, 'wb' ) do |f|
gz.each_chunk {|chunk| f.write(chunk)}
end

 

[mully]

I am trying to decompress a 80MB file (decompresses to 300MB), but I
keep getting a buffer error. I thought I had a workaround using
chunks, but it fails with 20M left to go. Any help or suggestions are
greatly appreciated.

in `read': buffer error (Zlib::BufError)

class Zlib::GzipReader
def each_chunk(chunk_size=1024)
yield read(chunk_size) until eof
end
end

gz = Zlib::GzipReader.open(zip_file)

File.open( non_zip_file, 'wb' ) do |f|
gz.each_chunk {|chunk| f.write(chunk)}
end

I get good results with the rubyzip gem, extracting 300+mb files from
Zip archives...

require 'zip/zip'
zip = Zip::ZipInputStream:pen(zipfile)
zip.get_next_entry
File.open(output_file, 'wb') do |f|
f.write(zip.read)
end
zip.close

Hope that helps.

David

 

[eden li]

The default decompress method works for me...

$ irb -r zlib
irb(main):001:0> File.stat('threehundred.gz').size / (1024**2)
=> 81
irb(main):002:0> Zlib::GzipReader.open('threehundred.gz') { |r|
File.open('threehundred', 'wb') { |f| f.write r.read } }
=> 314835846
irb(main):003:0> File.stat('threehundred').size / (1024**2)
=> 300

What version of ruby are you using?

 

[greg]

ruby -v
ruby 1.8.4 (2006-04-14) [i386-mswin32]

I should also say that the same code I am using works fine on a file
of 20MB that expands to 60MB

does the zip gem (or any other gem) handle .gz files?

Thanks,
Greg

 

[eden li]

strange... all I can think of is that the ruby installer you used has
a foobar'ed zlib1.dll. Have you tried upgrading to 1.8.5? Also, what
version of zlib does your ruby claim to have?

ruby -r zlib -e 'puts Zlib::ZLIB_VERSION'

1.2.3

ruby -v
ruby 1.8.4 (2006-04-14) [i386-mswin32]

I should also say that the same code I am using works fine on a file
of 20MB that expands to 60MB

does the zip gem (or any other gem) handle .gz files?

Thanks,
Greg

The default decompress method works for me...

$ irb -r zlib
irb(main):001:0> File.stat('threehundred.gz').size / (1024**2)
=> 81
irb(main):002:0> Zlib::GzipReader.open('threehundred.gz') { |r|
File.open('threehundred', 'wb') { |f| f.write r.read } }
=> 314835846
irb(main):003:0> File.stat('threehundred').size / (1024**2)
=> 300

What version of ruby are you using?

 

[greg]

ruby -r zlib -e 'puts Zlib::ZLIB_VERSION'
1.2.3

strange... all I can think of is that the ruby installer you used has
a foobar'ed zlib1.dll. Have you tried upgrading to 1.8.5? Also, what
version ofzlibdoes your ruby claim to have?

ruby -rzlib-e 'putsZlib::ZLIB_VERSION'

1.2.3

ruby -v
ruby 1.8.4 (2006-04-14) [i386-mswin32]

I should also say that the same code I am using works fine on a file
of 20MB that expands to 60MB

does the zip gem (or any other gem) handle .gz files?

 

[Clifford Heath]

ruby -r zlib -e 'puts Zlib::ZLIB_VERSION'
1.2.3

I recall having had that problem a year or two ago.
I think it was fixed by a version update, but I don't
know the version numbers any more. It was definitely
a bug in zlib though, and I definitely got around it
without having to patch the source.

Sorry I can't help more right now.

Clifford Heath.

 

[greg]

I am using the rubyzip library now- it works fine.