Rubyzip - `dup': can't dup NilClass (TypeError)

Discussion in 'Ruby' started by Luka Stolyarov, Jun 8, 2010.

  1. Hello. I've trying to figure out rubyzip. Here's the code I had:

    require 'rubygems'
    require 'zip/zip'

    zf = Zip::ZipFile.open('616910.zip')

    However, when I run it, it throws an error

    /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
    `dup': can't dup NilClass (TypeError)
    from
    /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
    `dup'
    from
    /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
    `map'
    from
    /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
    `dup'
    from
    /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1399:in
    `initialize'
    from
    /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
    `new'
    from
    /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
    `open'
    from zip.rb:5

    I'm pretty sure the zip file is there and is not empty. What might be
    causing it? I googled the problem, but couldn't find a definitive
    answer.

    Thank you,
    Luka
    --
    Posted via http://www.ruby-forum.com/.
    Luka Stolyarov, Jun 8, 2010
    #1
    1. Advertising

  2. Luka Stolyarov wrote:
    > require 'rubygems'
    > require 'zip/zip'
    >
    > zf = Zip::ZipFile.open('616910.zip')
    >
    > However, when I run it, it throws an error
    >
    > /Users/lukastolyarov/.gem/ruby/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1163:in
    > `dup': can't dup NilClass (TypeError)


    Works fine for me with rubyzip-0.9.1 + "ruby 1.8.6 (2007-09-24
    patchlevel 111) [i486-linux]", and I just upgraded to rubyzip-0.9.4 with
    the same results.

    What platform are you on?

    The offending code is here:

    # deep clone
    def dup
    newZipEntrySet = ZipEntrySet.new(@entrySet.values.map { |e| e.dup
    })
    end

    which suggests to me that the zipfile is corrupt or an unsupported
    format, since @entrySet must contain a {value=>nil} pair.

    Try modifying this code (line 657):

    def ZipEntry.read_c_dir_entry(io) #:nodoc:all
    entry = new(io.path)
    entry.read_c_dir_entry(io)
    return entry
    rescue ZipError
    return nil
    end

    For example, comment out the rescue ZipError // return nil pair.

    It seems that this error handling is bad. Either an exception should be
    raised here, or the bad entry should be skipped (not saved as a nil
    value in @entrySet which causes the dup error you saw)

    Regards,

    Brian.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Jun 8, 2010
    #2
    1. Advertising

  3. Luka Stolyarov, Jun 8, 2010
    #3
  4. Luka Stolyarov wrote:
    > I'll give it a try, thank you!


    Just to be clear: I'd expect the program to crash still, but this time
    at an earlier point which will give a much more useful error about what
    went wrong when parsing the zip directory entry.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Jun 8, 2010
    #4
  5. The 1.57 zipfile on this page fails for me:
    http://www.vim.org/scripts/script.php?script_id=2441

    The Unix unzip command unzips it just fine.

    Here's the direct link to the failing zip:
    http://www.vim.org/scripts/download_script.php?src_id=11978

    To test it in irb:

    require 'open-uri'
    require 'zip/zipfilesystem'
    open('http://www.vim.org/scripts/download_script.php?src_id=11978',
    'rb') { |f| Zip::ZipFile.open(f) }

    this dies with:

    TypeError: can't dup NilClass
    from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `dup'
    from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `block in dup'
    from rubyzip-0.9.4/lib/zip/zip.rb:1163:in `map'
    ...etc


    the zipfiles before 1.57 are fine. For instance:

    require 'open-uri'
    require 'zip/zipfilesystem'
    open('http://www.vim.org/scripts/download_script.php?src_id=11524',
    'rb') { |f| Zip::ZipFile.open(f) }

    works just fine.


    The nil entry is actually being added by read_central_directory_entries
    around line 1250 (I added the raise):

    @entrySet = ZipEntrySet.new
    @size.times {
    @entrySet << ZipEntry.read_c_dir_entry(io) || raise("nil
    entry!")
    }

    For some reason, read_c_dir_entry is returning nil. I haven't tried to
    figure out why since I'm not familiar with the internals of a zipfile.
    --
    Posted via http://www.ruby-forum.com/.
    Scott Bronson, Aug 27, 2010
    #5
  6. Scott Bronson wrote:
    > require 'open-uri'
    > require 'zip/zipfilesystem'
    > open('http://www.vim.org/scripts/download_script.php?src_id=11978',
    > 'rb') { |f| Zip::ZipFile.open(f) }


    For me (with ruby 1.8.7 and rubyzip-0.9.4) that dies with "cannot
    convert Tempfile to String", but if I download the zip locally and then
    do Zip::ZipFile.open("ert.zip") then I get the same error as you.

    > For some reason, read_c_dir_entry is returning nil. I haven't tried to
    > figure out why since I'm not familiar with the internals of a zipfile.


    I suggest you apply the following patch:

    --- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
    +0100
    +++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:34:19.673351372 +0100
    @@ -658,8 +658,8 @@
    entry = new(io.path)
    entry.read_c_dir_entry(io)
    return entry
    - rescue ZipError
    - return nil
    + #rescue ZipError
    + # return nil
    end

    def file_stat(path) # :nodoc:

    Then you get a more helpful error:

    /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:645:in
    `read_c_dir_entry': unknown file type 00 (Zip::ZipInternalError)
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:659:in
    `read_c_dir_entry'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1255:in
    `read_central_directory_entries'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
    `times'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1254:in
    `read_central_directory_entries'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1261:in
    `read_from_stream'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
    `initialize'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
    `open'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1392:in
    `initialize'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in `new'
    from /var/lib/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1410:in
    `open'

    To be honest, I have no idea why rubyzip is catching these errors and
    returning nil, instead of letting them propagate upwards. All it does is
    make a more obscure error later on ("cannot dup nil")

    With another patch:

    --- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
    +0100
    +++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:38:07.475854345 +0100
    @@ -642,7 +642,7 @@
    when 012
    @ftype = :symlink
    else
    - raise ZipInternalError, "unknown file type #{'0%o' %
    (@externalFileAttributes >> 28)}"
    + raise ZipInternalError, "unknown file type #{'0%o' %
    (@externalFileAttributes >> 28)} for entry #{@name.inspect}"
    end
    else
    if name_is_directory?

    you can see that the affected entry is .DS_Store. That unpacks as a
    regular file from unix unzip.

    I don't know why this entry happens to have a type of 0, but you can
    allow it like this:

    --- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
    +0100
    +++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
    @@ -637,12 +637,12 @@
    case (@externalFileAttributes >> 28)
    when 04
    @ftype = :directory
    - when 010
    + when 010, 00
    @ftype = :file
    when 012
    @ftype = :symlink

    Anyway, since you have a good test case for this, I suggest you post it
    to the rubyzip mailing list or tracker, if there is one.

    Regards,

    Brian.
    --
    Posted via http://www.ruby-forum.com/.
    Brian Candler, Aug 27, 2010
    #6
  7. Brian Candler wrote:
    > I don't know why this entry happens to have a type of 0, but you can
    > allow it like this:
    >
    > --- rubyzip-0.9.4/lib/zip/zip.rb.orig 2010-06-16 21:38:16.755077969
    > +0100
    > +++ rubyzip-0.9.4/lib/zip/zip.rb 2010-08-27 09:41:14.853352187 +0100
    > @@ -637,12 +637,12 @@
    > case (@externalFileAttributes >> 28)
    > when 04
    > @ftype = :directory
    > - when 010
    > + when 010, 00
    > @ftype = :file
    > when 012
    > @ftype = :symlink
    >


    The encoding of the external file attributes appears to be illegal. To
    verify this with another tool try running "zipinfo -v" on the zip-file
    and notice that it prints a '?' as the file type in the external file
    attributes.

    rubyzip should not throw if the value is unknown - it should simply
    translate it to :unknown.

    My familiarity with both ruby and rubyzip are rusting, but perhaps
    something like this:

    diff -u -r1.47 zip.rb
    --- lib/zip/zip.rb 14 May 2010 20:19:08 -0000 1.47
    +++ lib/zip/zip.rb 10 Sep 2010 19:51:32 -0000
    @@ -643,7 +643,7 @@
    when 012
    @ftype = :symlink
    else
    - raise ZipInternalError, "unknown file type #{'0%o' %
    (@externalFileAttributes >> 28)}"
    + @ftype = :unknown
    end
    else
    if name_is_directory?
    @@ -709,11 +709,11 @@
    when :symlink
    ft = 012
    @unix_perms ||= 0755
    - else
    - raise ZipInternalError, "unknown file type #{self.inspect}"
    end

    - @externalFileAttributes = (ft << 12 | (@unix_perms & 07777)) <<
    16
    + if (!ft.nil?)
    + @externalFileAttributes = (ft << 12 | (@unix_perms & 07777))
    << 16
    + end
    end

    io <<

    The problem should also be reported to the author of the tool generating
    the zip archives.

    Regards,

    Thomas
    --
    Posted via http://www.ruby-forum.com/.
    Thomas Sondergaard, Sep 10, 2010
    #7
  8. Thomas Sondergaard wrote:
    > The encoding of the external file attributes appears to be illegal. To
    > verify this with another tool try running "zipinfo -v" on the zip-file
    > and notice that it prints a '?' as the file type in the external file
    > attributes.


    Very strange. My zipinfo -v doesn't show anything amiss. I'm using:

    ZipInfo 3.00 of 20 April 2009, by Greg Roelofs and the Info-ZIP
    group.


    > The problem should also be reported to the author of the tool generating
    > the zip archives.


    Whatever tool is generating them, it's already in widespread use. :(
    For instance, most zipfiles in the following packages are valid, but
    rubyzip can't handle them.

    http://www.vim.org/scripts/script.php?script_id=3114
    http://www.vim.org/scripts/script.php?script_id=3123
    http://www.vim.org/scripts/script.php?script_id=3148
    http://www.vim.org/scripts/script.php?script_id=3150
    http://www.vim.org/scripts/script.php?script_id=3169

    Thanks Thomas. For now I've worked around it by shelling out to the
    unzip command. If I have time I'll try your patch next week.

    - Scott
    --
    Posted via http://www.ruby-forum.com/.
    Scott Bronson, Sep 10, 2010
    #8
  9. Scott Bronson wrote:
    > Whatever tool is generating them, it's already in widespread use. :(


    That might not actually be true... Most failing zipfiles were upped by
    the same author. I'll ask Peter Odding what he used to zip them.

    - Scott
    --
    Posted via http://www.ruby-forum.com/.
    Scott Bronson, Sep 10, 2010
    #9
  10. Luka Stolyarov

    Peter Odding Guest

    Hi Scott,

    >> Whatever tool is generating them, it's already in widespread use. :(

    >
    > That might not actually be true... Most failing zipfiles were upped by
    > the same author. I'll ask Peter Odding what he used to zip them.


    I publish my Vim plug-ins with a Python script that uses the zipfile [1]
    module to generate ZIP files. All of the archives uploaded in the last
    few months were generated on Ubuntu 9.10 (Karmic) with Python 2.6,
    although I just yesterday upgraded my laptop to Ubuntu 10.4 (Lucid) so I
    can't check the exact version. The ZIP file generation comes down to the
    following code:

    from zipfile import ZipFile, ZIP_DEFLATED
    archive = ZipFile('easytags.zip', 'w', ZIP_DEFLATED)
    for filename in ['plugin/easytags.vim', 'autoload/easytags.vim', ...]:
    # This is where "git cat-file" is used, see below.
    with open(filename) as handle:
    archive.writestr(filename, handle.read())
    archive.close()

    One peculiarity about the Python code is that it uses ZipFile.writestr()
    [2] instead of ZipFile.write() [3] so that it will only package
    committed versions of my scripts (using "git cat-file -p
    HEAD:some/file").

    Let me know if you need any more information. If it helps I might be
    able to resurrect my old environment in a virtual machine, but I haven't
    tried yet.

    - Peter Odding

    [1] http://docs.python.org/library/zipfile.html
    [2] http://docs.python.org/library/zipfile.html#zipfile.ZipFile.writestr
    [3] http://docs.python.org/library/zipfile.html#zipfile.ZipFile.write
    --
    Posted via http://www.ruby-forum.com/.
    Peter Odding, Sep 11, 2010
    #10
  11. Thomas Sondergaard, Sep 11, 2010
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Thomas

    [ANN] rubyzip-0.5.0

    Thomas, Aug 22, 2003, in forum: Ruby
    Replies:
    0
    Views:
    81
    Thomas
    Aug 22, 2003
  2. Simon Strandgaard

    RAA error - can't dup NilClass

    Simon Strandgaard, May 6, 2004, in forum: Ruby
    Replies:
    12
    Views:
    227
    Chad Fowler
    May 8, 2004
  3. José Luis
    Replies:
    2
    Views:
    325
    José Luis
    Apr 3, 2006
  4. subimage
    Replies:
    9
    Views:
    223
    Thomas Sondergaard
    Jul 29, 2006
  5. François Beausoleil

    :s.respond_to?(:dup) && :s.dup raises

    François Beausoleil, Apr 5, 2007, in forum: Ruby
    Replies:
    1
    Views:
    90
    Tim Hunter
    Apr 5, 2007
Loading...

Share This Page