T
Tom Graves
Hello,
I am trying to use python (2.6.6) to read a jar file that contains python files. I'm simply setting PYTHONPATH= spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar. Unfortunately it fails to read the python files from the jar file and if run in verbose mode just shows:
import zipimport # builtin
# installed zipimport hook
# zipimport: found 0 names in spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar
I was messing around and noticed that if I reduce the number of files and directories in the jar to below 65536 then it works:
import zipimport # builtin
# installed zipimport hook
# zipimport: found 65452 names in pyspark.jar
Is this a known limitation or is this perhaps fixed in newer version or is there a work around?
Note, I'm not subscribed to the mailing list so please copy me in response if possible.
Thanks,
Tom
I am trying to use python (2.6.6) to read a jar file that contains python files. I'm simply setting PYTHONPATH= spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar. Unfortunately it fails to read the python files from the jar file and if run in verbose mode just shows:
import zipimport # builtin
# installed zipimport hook
# zipimport: found 0 names in spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar
I was messing around and noticed that if I reduce the number of files and directories in the jar to below 65536 then it works:
import zipimport # builtin
# installed zipimport hook
# zipimport: found 65452 names in pyspark.jar
Is this a known limitation or is this perhaps fixed in newer version or is there a work around?
Note, I'm not subscribed to the mailing list so please copy me in response if possible.
Thanks,
Tom