would be nice: import from archive

[Dan Perl]

Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file?
Maybe a python specific file type, let's call it a 'par' file?

It would be useful in packaging an python library. Sure, there's always the
python packages, but a single file instead of a whole directory tree would
be more convenient. I am particularly interested because I am working on a
framework/toolkit and I am using a configuration divided into several
modules. It would be nice to be able to save configurations (combinations
of several modules) into single archive files and then switch between
configurations by pointing to one such archive file.

I am quite new to python so I should ask first whether there is already
something like that, although I did a search already. Or maybe such a
feature has already been discussed somewhere? If this is an original idea,
how can I propose it for future releases?

Dan Perl
(yes, I'm Mr. Perl, but I'm using Python, after all Perl is not my middle
name, it's my last name)

 

[Anthony Baxter]

Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file?
Maybe a python specific file type, let's call it a 'par' file?

Or a zip file, perhaps? See PEP-0273, which was implemented in Python 2.3.

(Hm. 273 isn't up-to-date. This is bad :-(

 

[Dan Perl]

My bad. Right after posting the message I found the 'Index of Python
Enhancement Proposals (PEPs)'
and there is PEP 273, 'Import Modules from Zip Archives', submitted by
James C. Ahlstrom. Thank you, Jim! And there are even two implementations
already.

Dan

 

[Sean Ross]

Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file? [snip]
I am quite new to python so I should ask first whether there is already
something like that, although I did a search already. Or maybe such a
feature has already been discussed somewhere? If this is an original idea,
how can I propose it for future releases?

Well, you can use .zip files
http://www.python.org/doc/2.3.4/whatsnew/node5.html
HTH,
Sean

 

[Dan Perl]

Yes, I saw that after posting my initial message. Thanks though, Anthony!

Dan

 

[Alex Martelli]

Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file?

Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.

I am quite new to python so I should ask first whether there is already
something like that, although I did a search already. Or maybe such a
feature has already been discussed somewhere? If this is an original idea,
how can I propose it for future releases?

It's there already, and has been for over a year now;-).


Alex

 

[Anthony Baxter]

My bad. Right after posting the message I found the 'Index of Python
Enhancement Proposals (PEPs)'
and there is PEP 273, 'Import Modules from Zip Archives', submitted by
James C. Ahlstrom. Thank you, Jim! And there are even two implementations
already.

Note that the PEP is not up-to-date. zipimport "just works" in Python
2.3 and Python 2.4. In the following example, we use the -v flag to
show where imports are coming from.

bonanza% cat hello.py
def hello():
print 'hello world'
bonanza% zip hello.zip hello.py
adding: hello.py (deflated 8%)
bonanza% rm hello.py
bonanza% python2.4 -v
Python 2.4a2 (#3, Aug 24 2004, 01:25:51)
[GCC 3.4.0 20040613 (Red Hat Linux 3.4.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
... lots and lots of lines showing default imports snipped ...Traceback (most recent call last):
# zipimport: found 1 names in hello.zip
dlopen("/usr/local/lib/python2.4/lib-dynload/zlib.so", 2);
import zlib # dynamically loaded from
/usr/local/lib/python2.4/lib-dynload/zlib.so
# zipimport: zlib available
import hello # loaded from Zip hello.zip/hello.pyhello world

At least on this box (linux) there's even a zip file on the default sys.path!

Anthony

 

[Paul Rubin]

Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.

Jar files are -signed- zip files. Is there some reason to not do that
for Python?

 

[Dan Perl]

I am using 2.3 and I tried importing from a zip file in a script of my own
and, yes, it works.

I never used the -v flag before but it's great for debugging some problems.
Thanks for mentioning it!

Dan

 

[Dan Perl]

----- Original Message -----
From: "Alex Martelli" <[email protected]>
Newsgroups: comp.lang.python
Sent: Friday, August 27, 2004 1:37 PM
Subject: Re: would be nice: import from archive

Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.

[...]

It's there already, and has been for over a year now;-).

Alex

Alright, alright! And given the fact that it was added only in 2.3, that
explains why I couldn't find any mention of the feature in 'Python
Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
2nd ed. ;-)

Dan

 

[Alex Martelli]

Jar files are -signed- zip files. Is there some reason to not do that
for Python?

I know of no reason to forbid support for such "signing", no. If you
want to offer a patch to zipimport to let it support whatever signing,
encryption, or other devilry appeals to you, this is definitely the
right moment if you hope to see it happen in Python 2.4.

If you mean patching zipimport to _forbid_ importing from any zipfile
whatsoever, including a plain vanilla one, I think it's too late for
THAT for Python 2.4 -- not sure what backwards incompatibilities that
might cause, but it's definitely not the kind of thing you can spring on
the world in a release that's already fast moving towards its
hopefully-last alpha release. (Adding functionality is one thing,
breaking compatibility with something that a previous release allowed is
quite another...).


Alex

 

[Alex Martelli]

Alright, alright! And given the fact that it was added only in 2.3, that
explains why I couldn't find any mention of the feature in 'Python
Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
2nd ed. ;-)

If somebody submits a good recipe about it, I'll be overjoyed to add it
(not sure what chapter -- 'files'? 'system administration'? 'programs
about programs'? -- ah well, I'll find a spot!-).

In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
breath!-), I'll write it up just like I will for all the new delights
since 2.2 -- but for the Cookbook I'm supposed to use recipes that
people do submit to the online cookbook site... OK, I and my co-editors
_do_ perform a lot of editing and merging, and occasionally do add a
recipe or three, but mostly it IS the book you all have written, with
over a hundred authors covering the subjects THEY think matter...


Alex

 

[Paul Rubin]

I know of no reason to forbid support for such "signing", no. If you
want to offer a patch to zipimport to let it support whatever signing,
encryption, or other devilry appeals to you, this is definitely the
right moment if you hope to see it happen in Python 2.4.

I think the simplest is to just have zipimport understand jar files
and their signatures. There's enough supporting infrastructure needed
that getting it in 2.4 is probably asking a bit much, though.

If you mean patching zipimport to _forbid_ importing from any zipfile
whatsoever, including a plain vanilla one, I think it's too late for
THAT for Python 2.4

If signing is supported, then there has to be a way to reject imports
whose signatures don't verify. It could be a runtime option or
something, I guess. How does zipimport work anyway? I don't see it
in the library doc index for 2.3.

 

[Jorge Godoy]

If somebody submits a good recipe about it, I'll be overjoyed to add it
(not sure what chapter -- 'files'? 'system administration'? 'programs
about programs'? -- ah well, I'll find a spot!-).

In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
breath!-), I'll write it up just like I will for all the new delights
since 2.2 -- but for the Cookbook I'm supposed to use recipes that
people do submit to the online cookbook site... OK, I and my co-editors
_do_ perform a lot of editing and merging, and occasionally do add a
recipe or three, but mostly it IS the book you all have written, with
over a hundred authors covering the subjects THEY think matter...

It would be great to have one example with more than one file.

From the discussion I got curious and tested it here and -- since
Python's so efficient I wasn't surprised that -- it worked.



$ cat test.py
def test():
print "Test from file 1"

$ cat test2.py
def test():
print "Test from file 2"



I also noticed that there was no '.pyc' created for that import, as is
usually done for uncompressed modules.



Be seeing you,

 

[Dan Perl]

I thought about submitting a recipe but I couldn't think of a way to use it
in a good code 'snippet'. I am using the zipimport feature now to save
several configuration files together in a zip file (so I can have many
configurations saved in a convenient format, better than a new directory for
each configuration). That would be more of an example for using multiple
configurations but, anyway, it doesn't make for a short, well-contained
example. Sorry, Alex.

BTW, there are two books on Python that I keep on my Safari bookshelf:
'Python Cookbook' and 'Learning Python'. So my jab was made will all the
respect I can muster. I am really looking forward to the second edition,
recipe on zipimport or not.

Dan

 

[Alex Martelli]

I think the simplest is to just have zipimport understand jar files
and their signatures. There's enough supporting infrastructure needed
that getting it in 2.4 is probably asking a bit much, though.


If signing is supported, then there has to be a way to reject imports
whose signatures don't verify. It could be a runtime option or
something, I guess. How does zipimport work anyway? I don't see it
in the library doc index for 2.3.

While the overall way the new import hooks work is well documented in
their PEP, zipimport is admittedly underdocumented. I suggest peeking
at the Python source distribution, files:
dist/src/Modules/zipimport.c
dist/src/Lib/test/test_zipimport.py

The excu^H^H^H^H reason for the documentation scarcity, you can read at
the end of the docstring for zipimport...:
"""
It is usually not needed to use the zipimport module explicitly; it is
used by the builtin import mechanism for sys.path items that are paths
to Zip archives.
"""


Alex

 

[Alex Martelli]

I thought about submitting a recipe but I couldn't think of a way to use it
in a good code 'snippet'. I am using the zipimport feature now to save
several configuration files together in a zip file (so I can have many
configurations saved in a convenient format, better than a new directory for
each configuration). That would be more of an example for using multiple
configurations but, anyway, it doesn't make for a short, well-contained
example. Sorry, Alex.

Alas! Your analysis seems spot-on and is pretty close to why I haven't
done a zipimport recipe myself -- it's a very useful feature but it's
not easy to think of a way to show it off in a recipe that's just how
recipes should be -- short, self-contained, readable. We can still hope
that somebody else does think of something...

BTW, there are two books on Python that I keep on my Safari bookshelf:
'Python Cookbook' and 'Learning Python'. So my jab was made will all the
respect I can muster. I am really looking forward to the second edition,
recipe on zipimport or not.

I did take your friendly jab as just that, friendly and pleasant, and
answered in just the same vein. (I do hope the reason the Nutshell
isn't on your Safari bookshelf is that you have the paper copy always at
hand, right?-)


Alex

 

[Paul Rubin]

While the overall way the new import hooks work is well documented in
their PEP, zipimport is admittedly underdocumented. I suggest peeking

OK. I'll look at the PEP. I think adding signing is hairy enough
that it should have its own round of discussion; should I see about
editing the PEP to add something about signing?

 

[Alex Martelli]

OK. I'll look at the PEP. I think adding signing is hairy enough
that it should have its own round of discussion; should I see about
editing the PEP to add something about signing?

I think that would be an excellent idea. If it was just about allowing
import from signed zipfiles it might not be needed, but how best to let
the user optionally DIS-allow imports from UN-signed files does appear
to be something requiring a little debate. An environment variable
would have the advantage of letting the disallowing work even for the
early imports that Python does before application code gets control, but
some people dislike relying on environment variables particularly for
security-related configuration tasks. Would it make sense to rely on a
naming convention instead? I.e. foo.zip would be unsigned but bar.jar
would have to be signed or else no go. This would have the advantage of
allowing substantial granularity in controlling this.

Side question, does module zipfile already have the code to allow
reading such signed files? Otherwise a first, uncontentious step, in
parallel with the PEP, might be a patch to add that ability to zipfile.


Alex

 

[Paul Rubin]

Would it make sense to rely on a naming convention instead?
I.e. foo.zip would be unsigned but bar.jar would have to be signed
or else no go. This would have the advantage of allowing
substantial granularity in controlling this.

I think this is reasonable, except what does the import statement look
like? Do you say something like "import frob from bar.jar"?

Side question, does module zipfile already have the code to allow
reading such signed files?

I think jar files are just zip files containing an extra file (called
"manifest") that has signatures in it. So you can import from a jar
as if it were a zip.