Rubyzip - `dup': can't dup NilClass (TypeError)

[Luka Stolyarov]

Hello. I've trying to figure out rubyzip. Here's the code I had:

require 'rubygems'
require 'zip/zip'

zf = Zip::ZipFile.open('616910.zip')

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup': can't dup NilClass (TypeError)
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`map'
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup'
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1399:in
`initialize'
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`new'
from
/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open'
from zip.rb:5

I'm pretty sure the zip file is there and is not empty. What might be
causing it? I googled the problem, but couldn't find a definitive
answer.

Thank you,
Luka

 

[Brian Candler]

require 'rubygems'
require 'zip/zip'

zf = Zip::ZipFile.open('616910.zip')

However, when I run it, it throws an error

/Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
`dup': can't dup NilClass (TypeError)

Works fine for me with rubyzip-0.9.1 + "ruby 1.8.6 (2007-09-24
patchlevel 111) [i486-linux]", and I just upgraded to rubyzip-0.9.4 with
the same results.

What platform are you on?

The offending code is here:

# deep clone
def dup
newZipEntrySet = ZipEntrySet.new(@entrySet.values.map { |e| e.dup
})
end

which suggests to me that the zipfile is corrupt or an unsupported
format, since @entrySet must contain a {value=>nil} pair.

Try modifying this code (line 657):

def ZipEntry.read_c_dir_entry(io) #:nodoc:all
entry = new(io.path)
entry.read_c_dir_entry(io)
return entry
rescue ZipError
return nil
end

For example, comment out the rescue ZipError // return nil pair.

It seems that this error handling is bad. Either an exception should be
raised here, or the bad entry should be skipped (not saved as a nil
value in @entrySet which causes the dup error you saw)

Regards,

Brian.

 

[Luka Stolyarov]

I'll give it a try, thank you!

 

[Brian Candler]

I'll give it a try, thank you!

Just to be clear: I'd expect the program to crash still, but this time
at an earlier point which will give a much more useful error about what
went wrong when parsing the zip directory entry.

 

[Scott Bronson]

The 1.57 zipfile on this page fails for me:
http://www.vim.org/scripts/script.php?script_id=2441

The Unix unzip command unzips it just fine.

Here's the direct link to the failing zip:
http://www.vim.org/scripts/download_script.php?src_id=11978

To test it in irb:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src_id=11978',
'rb') { |f| Zip::ZipFile.open(f) }

this dies with:

TypeError: can't dup NilClass
from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `dup'
from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `block in dup'
from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `map'
...etc


the zipfiles before 1.57 are fine. For instance:

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src_id=11524',
'rb') { |f| Zip::ZipFile.open(f) }

works just fine.


The nil entry is actually being added by read_central_directory_entries
around line 1250 (I added the raise):

@entrySet = ZipEntrySet.new
@size.times {
@entrySet << ZipEntry.read_c_dir_entry(io) || raise("nil
entry!")
}

For some reason, read_c_dir_entry is returning nil. I haven't tried to
figure out why since I'm not familiar with the internals of a zipfile.

 

[Brian Candler]

require 'open-uri'
require 'zip/zipfilesystem'
open('http://www.vim.org/scripts/download_script.php?src_id=11978',
'rb') { |f| Zip::ZipFile.open(f) }

For me (with ruby 1.8.7 and rubyzip-0.9.4) that dies with "cannot
convert Tempfile to String", but if I download the zip locally and then
do Zip::ZipFile.open("ert.zip") then I get the same error as you.

For some reason, read_c_dir_entry is returning nil. I haven't tried to
figure out why since I'm not familiar with the internals of a zipfile.

I suggest you apply the following patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:34:19.673351372 +0100
@@ -658,8 +658,8 @@
entry = new(io.path)
entry.read_c_dir_entry(io)
return entry
- rescue ZipError
- return nil
+ #rescue ZipError
+ # return nil
end

def file_stat(path) # :nodoc:

Then you get a more helpful error:

/var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:645:in
`read_c_dir_entry': unknown file type 00 (Zip::ZipInternalError)
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:659:in
`read_c_dir_entry'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1255:in
`read_central_directory_entries'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
`times'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
`read_central_directory_entries'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1261:in
`read_from_stream'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`initialize'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`open'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
`initialize'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in `new'
from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
`open'

To be honest, I have no idea why rubyzip is catching these errors and
returning nil, instead of letting them propagate upwards. All it does is
make a more obscure error later on ("cannot dup nil")

With another patch:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:38:07.475854345 +0100
@@ -642,7 +642,7 @@
when 012
@ftype = :symlink
else
- raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)}"
+ raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)} for entry #{@name.inspect}"
end
else
if name_is_directory?

you can see that the affected entry is .DS_Store. That unpacks as a
regular file from unix unzip.

I don't know why this entry happens to have a type of 0, but you can
allow it like this:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
case (@externalFileAttributes >> 28)
when 04
@ftype = :directory
- when 010
+ when 010, 00
@ftype = :file
when 012
@ftype = :symlink

Anyway, since you have a good test case for this, I suggest you post it
to the rubyzip mailing list or tracker, if there is one.

Regards,

Brian.

 

[Thomas Sondergaard]

I don't know why this entry happens to have a type of 0, but you can
allow it like this:

--- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
+0100
+++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
@@ -637,12 +637,12 @@
case (@externalFileAttributes >> 28)
when 04
@ftype = :directory
- when 010
+ when 010, 00
@ftype = :file
when 012
@ftype = :symlink

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running "zipinfo -v" on the zip-file
and notice that it prints a '?' as the file type in the external file
attributes.

rubyzip should not throw if the value is unknown - it should simply
translate it to :unknown.

My familiarity with both ruby and rubyzip are rusting, but perhaps
something like this:

diff -u -r1.47 zip.rb
--- lib/zip/zip.rb 14 May 2010 20:19:08 -0000 1.47
+++ lib/zip/zip.rb 10 Sep 2010 19:51:32 -0000
@@ -643,7 +643,7 @@
when 012
@ftype = :symlink
else
- raise ZipInternalError, "unknown file type #{'0%o' %
(@externalFileAttributes >> 28)}"
+ @ftype = :unknown
end
else
if name_is_directory?
@@ -709,11 +709,11 @@
when :symlink
ft = 012
@unix_perms ||= 0755
- else
- raise ZipInternalError, "unknown file type #{self.inspect}"
end

- @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) <<
16
+ if (!ft.nil?)
+ @externalFileAttributes = (ft << 12 | (@unix_perms & 07777))
<< 16
+ end
end

io <<

The problem should also be reported to the author of the tool generating
the zip archives.

Regards,

Thomas

 

[Scott Bronson]

The encoding of the external file attributes appears to be illegal. To
verify this with another tool try running "zipinfo -v" on the zip-file
and notice that it prints a '?' as the file type in the external file
attributes.

Very strange. My zipinfo -v doesn't show anything amiss. I'm using:

ZipInfo 3.00 of 20 April 2009, by Greg Roelofs and the Info-ZIP
group.

The problem should also be reported to the author of the tool generating
the zip archives.

Whatever tool is generating them, it's already in widespread use.
For instance, most zipfiles in the following packages are valid, but
rubyzip can't handle them.

http://www.vim.org/scripts/script.php?script_id=3114
http://www.vim.org/scripts/script.php?script_id=3123
http://www.vim.org/scripts/script.php?script_id=3148
http://www.vim.org/scripts/script.php?script_id=3150
http://www.vim.org/scripts/script.php?script_id=3169

Thanks Thomas. For now I've worked around it by shelling out to the
unzip command. If I have time I'll try your patch next week.

- Scott

 

[Scott Bronson]

Whatever tool is generating them, it's already in widespread use.

That might not actually be true... Most failing zipfiles were upped by
the same author. I'll ask Peter Odding what he used to zip them.

- Scott

 

[Peter Odding]

Hi Scott,

That might not actually be true... Most failing zipfiles were upped by
the same author. I'll ask Peter Odding what he used to zip them.

I publish my Vim plug-ins with a Python script that uses the zipfile [1]
module to generate ZIP files. All of the archives uploaded in the last
few months were generated on Ubuntu 9.10 (Karmic) with Python 2.6,
although I just yesterday upgraded my laptop to Ubuntu 10.4 (Lucid) so I
can't check the exact version. The ZIP file generation comes down to the
following code:

from zipfile import ZipFile, ZIP_DEFLATED
archive = ZipFile('easytags.zip', 'w', ZIP_DEFLATED)
for filename in ['plugin/easytags.vim', 'autoload/easytags.vim', ...]:
# This is where "git cat-file" is used, see below.
with open(filename) as handle:
archive.writestr(filename, handle.read())
archive.close()

One peculiarity about the Python code is that it uses ZipFile.writestr()
[2] instead of ZipFile.write() [3] so that it will only package
committed versions of my scripts (using "git cat-file -p
HEAD:some/file").

Let me know if you need any more information. If it helps I might be
able to resurrect my old environment in a virtual machine, but I haven't
tried yet.

- Peter Odding

[1] http://docs.python.org/library/zipfile.html
[2] http://docs.python.org/library/zipfile.html#zipfile.ZipFile.writestr
[3] http://docs.python.org/library/zipfile.html#zipfile.ZipFile.write

 

[Thomas Sondergaard]

I've filed the following issue against the python ZipFile module:

http://bugs.python.org/issue9835