zipping large file using Archive::Zip

Discussion in 'Perl Misc' started by Mark, Nov 9, 2007.

  1. Mark

    Mark Guest

    I have a 16 GB file that I zipped using perl's Archive::zip module.
    When I try to unzip this file using WinZip, it shows the "uncompressed
    size" = 4294967295.

    Upon trying to extract this file via WinZip, the winzip program
    deflates the file correctly to 16 GB, but then before it completes and
    yields control to the user, WinZip does a file size check between
    actual decompressed size and the file information (uncompressed size)
    stored with the Zip, and, since those don't match, it automatically it
    considers the decompressed file as invalid and removes it from the
    disk.

    This root cause of this problem is due to perl's ZLIB module writing
    an incorrect header information on the zip file

    Can someone please tell me why Perl's Archive::zip module writes
    incorrect information in the zip file header ? Are there any settings
    that can override such behaviour?

    Thanks
    Mark, Nov 9, 2007
    #1
    1. Advertising

  2. From: Christian Winter [mailto:]

    > Mark wrote:
    > > I have a 16 GB file that I zipped using perl's Archive::zip module.
    > > When I try to unzip this file using WinZip, it shows the "uncompressed
    > > size" = 4294967295.
    > >
    > > Upon trying to extract this file via WinZip, the winzip program
    > > deflates the file correctly to 16 GB, but then before it completes and
    > > yields control to the user, WinZip does a file size check between
    > > actual decompressed size and the file information (uncompressed size)
    > > stored with the Zip, and, since those don't match, it automatically it
    > > considers the decompressed file as invalid and removes it from the
    > > disk.
    > >
    > > This root cause of this problem is due to perl's ZLIB module writing
    > > an incorrect header information on the zip file
    > >
    > > Can someone please tell me why Perl's Archive::zip module writes
    > > incorrect information in the zip file header ? Are there any settings
    > > that can override such behaviour?

    >
    > The only way around I know of is to use a different decompression
    > tool. Archive::Zip makes use of Compress::Zlib, which in turn
    > calls the native zlib library. zlib, however, hasn't adopted the
    > somewhat proprietary deflate64 extension which PKWare introduced
    > in the pkzip sdk to make size values > 2^32 possible. In the standard
    > zip format, "compressed size" and "uncompressed size" are only
    > 4 bytes, therefore the value 4294967295.
    >
    > Unless deflate64 gets incorporated into zlib, I don't see much
    > chance that Archive::Zip will be able to produce "correct" zips
    > of this size. So the short solutions would be to either split
    > up things into separate files, each smaller than 4 GB uncompressed,
    > or use a different compression (e.g. tar/bzip2) - at least if
    > dropping WinZip is a no-go.


    That's not correct - you are confusing Deflate64 with Zip64. Here is the
    definition of Deflate64 from the Zip definition (from
    http://www.pkware.com/documents/casestudies/APPNOTE.TXT)


    Enhanced Deflating - Method 9
    -----------------------------

    The Enhanced Deflating algorithm is similar to Deflate but
    uses a sliding dictionary of up to 64K. Deflate64(tm) is supported
    by the Deflate extractor.

    To support compressed files > 2^32, your Zip implementation needs to support
    Zip64. That has nothing to do with the underlying zlib implementation.

    I don't think that Archive::Zip supports Zip64, but IO::Compress::Zip does.

    Paul
    Paul Marquess, Nov 12, 2007
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Graham Wood
    Replies:
    3
    Views:
    521
    A. Sinan Unur
    Jan 11, 2004
  2. Twig
    Replies:
    1
    Views:
    122
    A. Sinan Unur
    Jan 27, 2006
  3. MoshiachNow
    Replies:
    2
    Views:
    262
    Ilya Zakharevich
    Oct 4, 2006
  4. MoshiachNow

    Archive::Zip - zip file has "invalid" format

    MoshiachNow, Oct 5, 2006, in forum: Perl Misc
    Replies:
    1
    Views:
    156
  5. Bo Yang
    Replies:
    9
    Views:
    290
    -berlin.de
    Nov 20, 2006
Loading...

Share This Page