zipfile 2GB problems?

[xamdam]

Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread
http://mail.python.org/pipermail/python-dev/2005-April/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

 

[Jim]

Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a threadhttp://mail.python.org/pipermail/python-dev/2005-April/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
Thanks,
Jim

 

[Steve Holden]

Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.

.... with reading or writing *using the Python libraries*?

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[xamdam]

Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread http://mail.python.org/pipermail/python-dev/2005-April/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.

 

[xamdam]

Hi,

I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
Thanks,
Jim

Which version of Python? What platform?

thanks,
max

 

[Steve Holden]

Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.

It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[xamdam]

It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?

Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.

 

[Steve Holden]

Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.

Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[xamdam]

Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291> raise BadZipfile, "Bad magic number for file
header"

 

[Steve Holden]

It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291> raise BadZipfile, "Bad magic number for file
header"

Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

 

[xamdam]

Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!

regards
Steve

Well, Steve, I don't know what else to tell you - I certainly do not
want to cause you tooth ache! I was sort of hoping for some ZIP expert
to go, 'oh, yeah'...

 

[John Machin]

It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291> raise BadZipfile, "Bad magic number for file
header"

Suggestion 1: RTEDManual

Manual says:
"""
12.4.1 ZipFile Objects

class ZipFile( file[, mode[, compression[, allowZip64]]])
[snip]
If allowZip64 is True zipfile will create ZIP files that use the ZIP64
extensions when the zipfile is larger than 2 GB. If it is false (the
default) zipfile will raise an exception when the ZIP file would
require ZIP64 extensions. ZIP64 extensions are disabled by default
because the default zip and unzip commands on Unix (the InfoZIP
utilities) don't support these extensions.
"""

It's vague about whether allowZip64=True is needed for reading -- have
you tried it?

Suggestion 2: RTEDCode. At the line you quoted, it is checking the
first 4 bytes of the file for a signature: 'PK\x03\x04'
Do this:
print repr(open('yourfile.zip', 'rb').read(4))
Tell us what you find.

Suggestion 3: RTEDWordsOfWisdomOfSteveHolden. IOW supply the *WHOLE*
EDing traceback.

HTH,
John

 

[John Machin]

Well, Steve, I don't know what else to tell you

"post some code and a traceback"

I certainly do not
want to cause you tooth ache!

I suspect that you are inducing unwanted symptoms at the *other* end
of Steve's alimentary canal.

I was sort of hoping for some ZIP expert
to go, 'oh, yeah'...

Hope all you want, but supply the information you are asked for.