Re: Is any way to split zip archive to sections?

Discussion in 'Python' started by Tim Williams, Mar 31, 2007.

  1. Tim Williams

    Tim Williams Guest

    On 30/03/07, Durumdara <> wrote:
    > Hi!
    >
    > I want to create some backup archives with python (I want to write a backup
    > application in Python).
    > Some package managers (7z, arj, winzip) can create splitted archives (1
    > mega, 650, 700 mega, etc).
    >
    > Because I want to ftp these results to a ftp server, I want to split large
    > volumes to 15 mb sections.
    >
    > Can I do it with any python wrapper automatically (like in Cobian), or I
    > need to create the large volume, and next split it with another tool?
    >
    > Or anybody knows about a command line tool (like 7z, or arj) that can expand
    > the splitted archive (and I can add files with them from Python one by one)?
    >


    If you are iterating through a list of files to be backed up, and
    adding them to a ZIP one-by-one then you could use something like this
    which adds each file until the zip is over 15mb - then it closes the
    ZIP and creates the next one.

    Not tested or optimised :)
    ---------------------------------------------------------------------------------------------------
    import zipfile

    archive_num = 1
    outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    zsize = 0

    for full_name in filelist:
    full_name_path = os.path.join(full_name, full_path)

    if zsize > 15728640 : # 15mb
    outfile.close()
    archive_num += 1
    outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    zsize= 0

    outfile.write( full_name_path , full_name_path ,
    zipfile.ZIP_DEFLATED) # add the file
    zsize += outfile.getinfo(full_name_path).compress_size # get
    compressed size of file

    outfile.close()
    Tim Williams, Mar 31, 2007
    #1
    1. Advertising

  2. Tim Williams

    Larry Bates Guest

    Tim Williams wrote:
    > On 30/03/07, Durumdara <> wrote:
    >> Hi!
    >>
    >> I want to create some backup archives with python (I want to write a
    >> backup
    >> application in Python).
    >> Some package managers (7z, arj, winzip) can create splitted archives (1
    >> mega, 650, 700 mega, etc).
    >>
    >> Because I want to ftp these results to a ftp server, I want to split
    >> large
    >> volumes to 15 mb sections.
    >>
    >> Can I do it with any python wrapper automatically (like in Cobian), or I
    >> need to create the large volume, and next split it with another tool?
    >>
    >> Or anybody knows about a command line tool (like 7z, or arj) that can
    >> expand
    >> the splitted archive (and I can add files with them from Python one by
    >> one)?
    >>

    >
    > If you are iterating through a list of files to be backed up, and
    > adding them to a ZIP one-by-one then you could use something like this
    > which adds each file until the zip is over 15mb - then it closes the
    > ZIP and creates the next one.
    >
    > Not tested or optimised :)
    > ---------------------------------------------------------------------------------------------------
    >
    > import zipfile
    >
    > archive_num = 1
    > outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    > zsize = 0
    >
    > for full_name in filelist:
    > full_name_path = os.path.join(full_name, full_path)
    >
    > if zsize > 15728640 : # 15mb
    > outfile.close()
    > archive_num += 1
    > outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    > zsize= 0
    >
    > outfile.write( full_name_path , full_name_path ,
    > zipfile.ZIP_DEFLATED) # add the file
    > zsize += outfile.getinfo(full_name_path).compress_size # get
    > compressed size of file
    >
    > outfile.close()


    You can do slightly better as following:

    change to (not tested):


    import os
    import stat
    import zipfile

    archive_num = 1
    outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    zsize = 0

    for full_name in filelist:
    full_name_path = os.path.join(full_name, full_path)
    if outfile.fp.tell() > 15728640: # 15mb
    outfile.close()
    archive_num += 1
    outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")

    outfile.write(full_name_path,full_name_path,zipfile.ZIP_DEFLATED)


    There is still a couple of "issues":

    1) Files larger than 15Mb may not be able to be compressed to fall below the limit.

    2) If you are near the 15Mb limit and the next file is very large you have the
    same problem.


    -Larry
    Larry Bates, Apr 2, 2007
    #2
    1. Advertising

  3. Tim Williams

    Larry Bates Guest

    Tim Williams wrote:
    > On 30/03/07, Durumdara <> wrote:
    >> Hi!
    >>
    >> I want to create some backup archives with python (I want to write a
    >> backup
    >> application in Python).
    >> Some package managers (7z, arj, winzip) can create splitted archives (1
    >> mega, 650, 700 mega, etc).
    >>
    >> Because I want to ftp these results to a ftp server, I want to split
    >> large
    >> volumes to 15 mb sections.
    >>
    >> Can I do it with any python wrapper automatically (like in Cobian), or I
    >> need to create the large volume, and next split it with another tool?
    >>
    >> Or anybody knows about a command line tool (like 7z, or arj) that can
    >> expand
    >> the splitted archive (and I can add files with them from Python one by
    >> one)?
    >>

    >
    > If you are iterating through a list of files to be backed up, and
    > adding them to a ZIP one-by-one then you could use something like this
    > which adds each file until the zip is over 15mb - then it closes the
    > ZIP and creates the next one.
    >
    > Not tested or optimised :)
    > ---------------------------------------------------------------------------------------------------
    >
    > import zipfile
    >
    > archive_num = 1
    > outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    > zsize = 0
    >
    > for full_name in filelist:
    > full_name_path = os.path.join(full_name, full_path)
    >
    > if zsize > 15728640 : # 15mb
    > outfile.close()
    > archive_num += 1
    > outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    > zsize= 0
    >
    > outfile.write( full_name_path , full_name_path ,
    > zipfile.ZIP_DEFLATED) # add the file
    > zsize += outfile.getinfo(full_name_path).compress_size # get
    > compressed size of file
    >
    > outfile.close()


    You can do slightly better as following:

    change to (not tested):


    import os
    import stat
    import zipfile

    archive_num = 1
    outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")
    zsize = 0

    for full_name in filelist:
    full_name_path = os.path.join(full_name, full_path)
    if outfile.fp.tell() > 15728640: # 15mb
    outfile.close()
    archive_num += 1
    outfile = zipfile.ZipFile('/zips/archive%s.zip' % archive_num, "w")

    outfile.write(full_name_path,full_name_path,zipfile.ZIP_DEFLATED)


    There is still a couple of "issues":

    1) Files larger than 15Mb may not be able to be compressed to fall below the limit.

    2) If you are near the 15Mb limit and the next file is very large you have the
    same problem.


    -Larry
    Larry Bates, Apr 2, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Graham Wood
    Replies:
    3
    Views:
    518
    A. Sinan Unur
    Jan 11, 2004
  2. Twig
    Replies:
    1
    Views:
    121
    A. Sinan Unur
    Jan 27, 2006
  3. MoshiachNow
    Replies:
    2
    Views:
    257
    Ilya Zakharevich
    Oct 4, 2006
  4. MoshiachNow

    Archive::Zip - zip file has "invalid" format

    MoshiachNow, Oct 5, 2006, in forum: Perl Misc
    Replies:
    1
    Views:
    153
  5. Bo Yang
    Replies:
    9
    Views:
    289
    -berlin.de
    Nov 20, 2006
Loading...

Share This Page