zipimport limited to 65536 files?

[Tom Graves]

Hello,

I am trying to use python (2.6.6) to read a jar file that contains python files.  I'm simply setting PYTHONPATH= spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar.   Unfortunately it fails to read the python files from the jar file and if run in verbose mode just shows:

import zipimport # builtin
# installed zipimport hook
# zipimport: found 0 names in spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar

I was messing around and noticed that if I reduce the number of files and directories in the jar to below 65536 then it works:

import zipimport # builtin
# installed zipimport hook
# zipimport: found 65452 names in pyspark.jar

Is this a known limitation or is this perhaps fixed in newer version or is there a work around?

Note, I'm not subscribed to the mailing list so please copy me in response if possible.

Thanks,
Tom