how to create multiple zip files that are no larger than 2 Gb each

Discussion in 'Perl Misc' started by texasreddog, May 15, 2007.

  1. texasreddog

    texasreddog Guest

    I have an interesting problem, where I am zipping up a large archive
    of jpg images. The zip file can be no larger than 2 Gb. If I try to
    zip up n number of images, where n can vary a lot, and if the zip file
    gets to 2Gb, the zip process fails.

    Would there be some way that I could track the .zip file as it is
    being zipped, so when it reaches 2 Gb in size, a new zip file could be
    created and the zip process could continue? I don't think the split
    command will work, because that runs after zip finishes.

    Or, is there a way to possibly start another zip file, when the first
    one fails, and then somehow force the zip to pick up where it left
    off?

    Any suggestions/ideas/tips are much appreciated! Thanks!
    texasreddog, May 15, 2007
    #1
    1. Advertising

  2. texasreddog

    Guest

    texasreddog <> wrote:
    > I have an interesting problem, where I am zipping up a large archive
    > of jpg images. The zip file can be no larger than 2 Gb. If I try to
    > zip up n number of images, where n can vary a lot, and if the zip file
    > gets to 2Gb, the zip process fails.
    >
    > Would there be some way that I could track the .zip file as it is
    > being zipped, so when it reaches 2 Gb in size, a new zip file could be
    > created and the zip process could continue?


    I wouldn't expect jpgs to compress by any meaningful amount, so you could
    probably assume the size of the archive will be slightly more than the sum
    of the sizes of the original files.


    > I don't think the split
    > command will work, because that runs after zip finishes.
    >
    > Or, is there a way to possibly start another zip file, when the first
    > one fails, and then somehow force the zip to pick up where it left
    > off?


    I don't know the answer, but I expect it will depend on what program/module
    you are using to do the zipping.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , May 15, 2007
    #2
    1. Advertising

  3. texasreddog

    Joe Smith Guest

    Re: how to create multiple zip files that are no larger than 2 Gbeach

    texasreddog wrote:
    > I have an interesting problem, where I am zipping up a large archive
    > of jpg images. The zip file can be no larger than 2 Gb. If I try to
    > zip up n number of images, where n can vary a lot, and if the zip file
    > gets to 2Gb, the zip process fails.
    >
    > Would there be some way that I could track the .zip file as it is
    > being zipped, so when it reaches 2 Gb in size, a new zip file could be
    > created and the zip process could continue?


    Here is what I am currently using for that purpose.
    -Joe

    our $MAXSIZE = 2047 * 1024 * 1024; # Just less than 2 GB

    sub create_zip {
    my($out_file,@files) = @_;
    for (my $N='a'; @files; $N++) {
    my $zip_file = "$out_file$N.zip";
    my $name = shift @files; # Always do at least one file per zip
    my @names = ($name);
    my $bytes = -s $name or next;
    while (@files and $bytes < $MAXSIZE) {
    $name = shift @files;
    $bytes += -s $name || 0; # Cumulative uncompressed size
    if ($bytes < $MAXSIZE) {
    push @names, $name; # OK to add this one
    } else {
    unshift @files,$name; # Put it back on list of files to do
    }
    }
    my $cmd = '|zip ' . (-f $zip_file ? '-yu' : '-y ') . " $zip_file -@";
    print "$cmd (" . @names . " files)\n"; # Number of files being zipped
    print " ",join("\n ",@names),"\n" if $verbose;
    if (open ZIP,$cmd) {
    print ZIP join "\n",@names,'' or warn "print(ZIP): $! ($?)";
    close ZIP;
    my ($err,$sig) = ($? >> 8, $? & 0xff); # 12 = "No changes"
    warn "zip returned error code $err\n" if $err and $err != 12;
    warn "At least one file is in use; cannot be backed up\n" if $err == 18;
    next if $err == 12;
    print "$zip_file = " . bytes_count(-s $zip_file) . "\n\n";
    sleep 1;
    } else {
    warn "open('$cmd') failed: $! ($?)";
    }
    }
    }
    Joe Smith, May 16, 2007
    #3
  4. texasreddog

    Peter Wyzl Guest

    "texasreddog" <> wrote in message
    news:...
    >I have an interesting problem, where I am zipping up a large archive
    > of jpg images. The zip file can be no larger than 2 Gb. If I try to
    > zip up n number of images, where n can vary a lot, and if the zip file
    > gets to 2Gb, the zip process fails.
    >
    > Would there be some way that I could track the .zip file as it is
    > being zipped, so when it reaches 2 Gb in size, a new zip file could be
    > created and the zip process could continue? I don't think the split
    > command will work, because that runs after zip finishes.
    >
    > Or, is there a way to possibly start another zip file, when the first
    > one fails, and then somehow force the zip to pick up where it left
    > off?
    >
    > Any suggestions/ideas/tips are much appreciated! Thanks!


    You probably need to update your Perl to a more recent version if you can't
    create files over 2Gb in size.

    P
    Peter Wyzl, May 16, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. grimBeast
    Replies:
    2
    Views:
    667
    MS News \(MS ILM\)
    Aug 27, 2003
  2. =?Utf-8?B?Sm9yZ2UgUmliZWlybw==?=

    Uploading files larger than 4Mb

    =?Utf-8?B?Sm9yZ2UgUmliZWlybw==?=, Feb 18, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    518
    Gopal \(FMS, Inc.\)
    Feb 18, 2005
  3. Peter Williams
    Replies:
    1
    Views:
    1,380
    Dylan Parry
    Jun 3, 2005
  4. Robert Kochem

    iostream and files larger than 4GB

    Robert Kochem, Jul 21, 2008, in forum: C++
    Replies:
    16
    Views:
    1,000
    James Kanze
    Jul 23, 2008
  5. MoshiachNow
    Replies:
    2
    Views:
    272
    Ilya Zakharevich
    Oct 4, 2006
Loading...

Share This Page