zipfile library - problem..

[Renzo]

Hi,
I'm working to create a backup function for a software; in particular,
from a directory with 12300 files (Jpg, medium size = 250KB / total size
= 2.90GB), i have to create a zip file.

I've decided to use the standard "zipfile" library to do that.
This is the code:

zipName = path.join(config.get('server.xbakPath'), 'backup.zip')
fileExport = zipfile.ZipFile(zipName,'w',zipfile.ZIP_DEFLATED)

#Insert images
listaImg = listdir( config.get('server.iPath') )
for file in listaImg:
if not path.isdir(path.join(config.get('server.iPath'), file)):
fileExport.write(path.join(config.get('server.iPath'), file), \
file)

#close ZIP
fileExport.close()

Unfortunately, after few minutes, i see this error message:
Traceback (most recent call last):
File "C:\Python24\Lib\site-packages\cherrypy\_cphttptools.py", line
256, in run
main()
File "C:\Python24\Lib\site-packages\cherrypy\_cphttptools.py", line
452, in main
body = func(*(virtualPathList + cherrypy.request.paramList),
File "C:\pagine\utilities.py", line 162, in exportdati
fileExport.close()
File "C:\Python24\lib\zipfile.py", line 503, in close
zinfo.header_offset)
OverflowError: long int too large to convert to int

Reading the error I've known that the error is generated calling
"fileExport.close()", that calls the zipfile module,
and into that module there is an exception.

For information: when code stops, the zip file has size = 3.177.950.237
byte, but this size doesn't exceed ZIP limits:

the limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/in fozip/FAQ.html#limits

statistic limit
number of files 65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length 64 KB

Have you any idea?
Oh, the O.S. is WinXP Pro

Thank you!.
Renzo

 

[Renzo]

Renzo ha scritto:
<cut all>
I've seen that the error is a known bug:

"zipfile still has 2GB boundary bug"
<http://mail.python.org/pipermail/python-dev/2005-April/052887.html>

so I've used the module TarFile instead ZipFile, and now the script
works perfectly.

Thanks to Deelan and Manlio Perillo (i.c.l.p.)

Renzo

 

[Bryan Olson]

> Renzo ha scritto:
> <cut all>
> I've seen that the error is a known bug:
>
> "zipfile still has 2GB boundary bug"
> <http://mail.python.org/pipermail/python-dev/2005-April/052887.html>
>
> so I've used the module TarFile instead ZipFile, and now the script
> works perfectly.

Thanks; good to know. Looks like people are working on this
Python library bug. Next after that, we'll need to fix the zip
format. The 4GB limit on contained files and 256TB limit on
archives both looked astronomical at the time they were defined.
Sorry to sound like an old guy, but wow, how times change.

Incidentally, since your files are jpegs, they're already
compressed, and thus TarFile should be just as good as ZipFile
anyway.