how can I put a 1Gb file in a zipfile??

B

Bennie

Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?

souce:
zip = zipfile.ZipFile(file, 'w')

for all in os.walk(os.getcwd()):
path = all[0]
for document in all[2]:
zipaccview.write(path + os.sep + document)

zip.close()

Bennie
 
C

Christos TZOTZIOY Georgiou

Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?

AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip
and to the total size of the zip file. This limit comes from the zip file
specification (32 bit offsets).
souce:
zip = zipfile.ZipFile(file, 'w')

for all in os.walk(os.getcwd()):
path = all[0]
for document in all[2]:
zipaccview.write(path + os.sep + document)

zip.close()

Can it be that you are creating a zip file that its total size exceeds the
limit?
 
B

bennie

Christos said:
Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?


AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip
and to the total size of the zip file. This limit comes from the zip file
specification (32 bit offsets).

souce:
zip = zipfile.ZipFile(file, 'w')

for all in os.walk(os.getcwd()):
path = all[0]
for document in all[2]:
zipaccview.write(path + os.sep + document)

zip.close()


Can it be that you are creating a zip file that its total size exceeds the
limit?
That is possible.
But with Winzip program it can.
How come that is with ZipFile not works
 
J

Jeff Epler

The limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/infozip/FAQ.html#limits

statistic limit
number of files 65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length 64 KB

I had no trouble creating a zip file from a 4GB file filled with '\0'
bytes:
$ python bennie.py
$ ls -ls test.zip big
12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big
4084 -rw-rw-r-- 1 jepler jepler 4174545 Mar 20 14:14 test.zip

I'm using Python 2.3.3 on Fedora Core 2.
#------------------------------------------------------------------------
# bennie.py
def make_4gb_file(f):
f = open(f, "w")
f.seek ( 4 * 1024 * 1024 * 1024 - 1)
f.write("\0")
f.close()

import zipfile
z = zipfile.ZipFile("/tmp/test.zip", "w", zipfile.ZIP_DEFLATED)
make_4gb_file("/tmp/big")
z.write("/tmp/big")
z.close()
#------------------------------------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCPdpOJd01MZaTXX0RAngaAJ9tkc4tCUAwJmlio7/9Pn46Dyh9bgCfdRQW
4rQqWnqLljuDUH/d6NBfoeM=
=XSAF
-----END PGP SIGNATURE-----
 
B

Bengt Richter

--liOOAslEiF7prFVr
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

The limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/infozip/FAQ.html#limits

statistic limit
number of files 65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length 64 KB=20

I had no trouble creating a zip file from a 4GB file filled with '\0'
bytes:
$ python bennie.py
$ ls -ls test.zip big
12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big
4084 -rw-rw-r-- 1 jepler jepler 4174545 Mar 20 14:14 test.zip

I'm using Python 2.3.3 on Fedora Core 2.
#------------------------------------------------------------------------
# bennie.py
def make_4gb_file(f):
f =3D open(f, "w")
f.seek ( 4 * 1024 * 1024 * 1024 - 1)
f.write("\0")
f.close()
Not quite OT[?]:
This makes me think there ought to be a way of making at least python's builtin open see virtual file system objects,
analogous to StringIO creating file objects.

If we had a mountvfs('/some/unix/and/or/maybe/win/style/path/vfs', vfsclass(some, args, if_desired))
which would result in that open(/some/unix/and/or/maybe/win/style/path/vfs/morepath/filename.ext', mode)
(where vfsclass(some, args, if_desired) => vsfclass_intance) would call
vfsclass_instance.open('morepath.filename.ext', mode) which could then return an object that could support
file operations like returning 4gb of virtually read-by-read-method data, or otherwise acting like an open file
object of a real file system that python programs and library function using open and file would find
if given the mounted path. Subdirectories could be fixed for starters, but virtualizing subdirectory creation
etc would be possible if you intercepted the right interface calls and implemented it in the vfs.

This would let you define a virtual file in place of the real file above, and also would allow a lot of transparent
testing of file-using software that takes paths and file names, not open file objects.

Of course you can't affect what the underlying os sees without getting into its file system machinery, but
being able to mount virtual file systems into what os.open sees would cover a lot of ground ISTM. One
could argue pro and con about supporting virtual mounts into both unix and windows-style paths.
import zipfile
z =3D zipfile.ZipFile("/tmp/test.zip", "w", zipfile.ZIP_DEFLATED)
make_4gb_file("/tmp/big")
z.write("/tmp/big")
z.close()
#------------------------------------------------------------------------

Regards,
Bengt Richter
 
R

RM

From what I understand, Winzip uses their own proprietary version of a
Zip format. That means that you will only be able open those archives
with Winzip. (IOW, you are locked in)

I think someone mentioned having developed a Python module for the 7zip
fomat, but I may be wrong. In any case, you could simply write a
wrapper over the command line version of 7zip and your problem would be
solved. There is a Linux version as well.
(http://p7zip.sourceforge.net/)
 
?

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=

Bennie said:
How can I fix this?

Since this is the question that you actually asked, there
is an easy answer. Investigate the problem in detail, understand
the source code of the zipfile module, and the format of zip files,
and develop a change to the code to correct the behaviour.
Then submit a patch to sf.net/projects/python.

Regards,
Martin
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top