py2exe and library.zip

[Timothy Smith]

is it possible instead of py2exe putting all library's in a zip file, to
just put them in a sub dir?

 

[Peter Hansen]

is it possible instead of py2exe putting all library's in a zip file, to
just put them in a sub dir?

Anything's possible. Maybe you could explain what you're actually
trying to accomplish (or the reasons) so we can better understand where
you're going with this...

-Peter

 

[Timothy Smith]

Timothy Smith wrote:



Anything's possible. Maybe you could explain what you're actually
trying to accomplish (or the reasons) so we can better understand where
you're going with this...

-Peter

exactly what i just stated, i don't want py2exe to zip up it's
library's, but to put them in a sub dir.
the reason for this, is so that when users login and update from svn,
they only have to download some tiny pyc files, not a great big zip file
with everything in it ( 99% of which never changes)

 

[Simon Brunning]

exactly what i just stated, i don't want py2exe to zip up it's
library's, but to put them in a sub dir.
the reason for this, is so that when users login and update from svn,
they only have to download some tiny pyc files, not a great big zip file
with everything in it ( 99% of which never changes)

The zip file is a generated artifact. I've always found it a good rule
of thumb that you should keep source artifacts in your version control
system, not generated artifacts.

 

[Simon Brunning]

version control systems are used for many purposes. it's just as
important to keep track of your packaged releases and documentation as
it is source code.
its also a very convenient method of keeping people up with the lastest
version.

Well, they may be used for many things, but what they are *good* at is
what they were designed for. Use them to do a job that they *weren't*
designed for, and they don't do so well.

The zip file essentially contains the whole system in on lump. Change
the system, and naturally your users will have to download the whole
lump again. (In my 'day job', I'm a Java web app developer. We keep
the source for our systems in SVN, any 3rd party JARs, and our build
scripts. We *used* to keep the installable WAR files in there too, but
we've stopped doing that now, for this reason amongst others.)

i know i've seen what i'm asking for before, but no one seems to be able
to tell me.

Sorry, but I'm not aware that py2exe can do this, or has ever been
able to. Again, it's designed to build you a minimum-file-count
distributable, and that's what it's good at.

 

[Timothy Smith]

Well, they may be used for many things, but what they are *good* at is
what they were designed for. Use them to do a job that they *weren't*
designed for, and they don't do so well.

don't do so well how. all it does is keep of files and tracks their
revisions.

The zip file essentially contains the whole system in on lump. Change
the system, and naturally your users will have to download the whole
lump again. (In my 'day job', I'm a Java web app developer. We keep
the source for our systems in SVN, any 3rd party JARs, and our build
scripts. We *used* to keep the installable WAR files in there too, but
we've stopped doing that now, for this reason amongst others.)

but if it was just a dir, when they update from the svn at log in, all
they do is download the extra\changed files. much much quicker then
downloading a 4 meg uncompressed zip file.
there's no technical reason for this not to work that i can see, apart
from py2exe's ability/inablity to not use a zip file.

 

[Peter Hansen]

exactly what i just stated, i don't want py2exe to zip up it's
library's, but to put them in a sub dir.
the reason for this, is so that when users login and update from svn,
they only have to download some tiny pyc files, not a great big zip file
with everything in it ( 99% of which never changes)

More detail on this version control thing would probably help. (Things
like that are why I asked fo more background... "exactly what I stated"
isn't as useful as something like (for example) "I want my app users to
use Subversion to retrieve updates to the application without having to
download a big zip file".)

Do you know that Subversion has (as I understand it) a fairly
intelligent binary file comparison routine, and it will (again, as I
understand it) not transmit the entire contents of the zip file but
would actually send only the portions that have changed? At least,
that's if the file isn't compressed in some way that prevents this
algorithm from working well. (Note to self: check if zip files that can
be in sys.path can be compressed, and if py2exe compresses them.)

Anyway, while this sort of thing isn't directly supported (probably
since almost nobody would want to do it, for reasons similar to what
Simon has explained), you should be able to work it out yourself. The
..exe produced by py2exe contains a "stub" executable that basically sets
sys.path to point to the .zip file, then runs the main .py which has
been integrated into the .exe file (not even compiled to a .pyc, just
like normal). If you modify your main .py to adjust sys.path to point
to an external subdirectory, you should be able to leave .py files (or
..pyc files) there and use svn to update them.

I'll leave it as an exercise for the reader (mainly because I don't know
how to do it offhand) what you should do to prevent py2exe from putting
all your files in its zip. It should probably still put the standard
library modules there, since otherwise you'd have little reason to be
using py2exe in the first place...

-Peter

 

[Timothy Smith]

Timothy Smith wrote:



More detail on this version control thing would probably help. (Things
like that are why I asked fo more background... "exactly what I stated"
isn't as useful as something like (for example) "I want my app users to
use Subversion to retrieve updates to the application without having to
download a big zip file".)

Do you know that Subversion has (as I understand it) a fairly
intelligent binary file comparison routine, and it will (again, as I
understand it) not transmit the entire contents of the zip file but
would actually send only the portions that have changed? At least,
that's if the file isn't compressed in some way that prevents this
algorithm from working well. (Note to self: check if zip files that can
be in sys.path can be compressed, and if py2exe compresses them.)

Anyway, while this sort of thing isn't directly supported (probably
since almost nobody would want to do it, for reasons similar to what
Simon has explained), you should be able to work it out yourself. The
.exe produced by py2exe contains a "stub" executable that basically sets
sys.path to point to the .zip file, then runs the main .py which has
been integrated into the .exe file (not even compiled to a .pyc, just
like normal). If you modify your main .py to adjust sys.path to point
to an external subdirectory, you should be able to leave .py files (or
.pyc files) there and use svn to update them.

I'll leave it as an exercise for the reader (mainly because I don't know
how to do it offhand) what you should do to prevent py2exe from putting
all your files in its zip. It should probably still put the standard
library modules there, since otherwise you'd have little reason to be
using py2exe in the first place...

-Peter

ideally i'd love to not to have to use py2exe at all, then they can just
update from the stable branch of my svn. the problem with THAT is how to
install python, wx,pysvn,reportlab etc etc on their systems with no
fuss(py2exe convienently packages all dll's together for me). if you
know of a way i can achieve that i'd be your best friend forever.

the whole reason i'm going with the svn updating method is there is no
user interaction required. even getting users to click a single button
to update has proven too much to ask of them :/

the other reason is my particular project gets updated very often, so
you can imagine the combination of users resistant to any kind of
computer use with an environment requiring frequent updating.


i have some other problems to solve before i get to that point, mainly
svn related eg. pysvn whinges because files exist in the working
directory that it wants to update ( with the same name ), only they
aren't under version control.

 

[Just]

[ ... ] (Note to self: check if zip files that can
be in sys.path can be compressed,
Yes.

and if py2exe compresses them.)

Don't know, but I assume yes.

Just

 

[Thomas Heller]

is it possible instead of py2exe putting all library's in a zip file,
to just put them in a sub dir?

You can subclass the build_exe command (the extending sample shows how
to do this, althoutgh for a different purpose), and copy the files into
a sub dir instead of building an archive. You still have to make sure
that your exe has this directory on sys.path.

Thomas

 

[Thomas Heller]

[ ... ] (Note to self: check if zip files that can
be in sys.path can be compressed,
Yes.

and if py2exe compresses them.)

Don't know, but I assume yes.

There's an option for that, because it slows down the imports.

Thomas

 

[David Bolen]

Do you know that Subversion has (as I understand it) a fairly
intelligent binary file comparison routine, and it will (again, as I
understand it) not transmit the entire contents of the zip file but
would actually send only the portions that have changed? At least,
that's if the file isn't compressed in some way that prevents this
algorithm from working well. (Note to self: check if zip files that
can be in sys.path can be compressed, and if py2exe compresses them.)

Even if the files were compressed, which has a net result similar to
randomizing the contents and will certainly extend the portion that
appears "changed", the worst that would happen is that subversion
(which does use a binary delta algorithm) would end up downloading the
single file portion of the zip file rather than the smaller change
within the file. It should still be efficient.

But to be honest, for something like the OPs purpose, it's not clear
that an SCM is needed, since all he's trying to accomplish is bring a
remote copy up to date with the central one. For that you could just
publish a location containing the necessary files and have the users
use something like rsync directly (which is just as efficient in terms
of a binary delta) to update their own local version.

Of course, if the Subversion server is already in place so it's a
convenient server, or if more of the user base already has the client
in place, it should work just about as well.

-- David

 

[Peter Hansen]

Even if the files were compressed, which has a net result similar to
randomizing the contents and will certainly extend the portion that
appears "changed", the worst that would happen is that subversion
(which does use a binary delta algorithm) would end up downloading the
single file portion of the zip file rather than the smaller change
within the file. It should still be efficient.

Good point. When I wrote that I was picturing the form of compression
that a .tar.gz file would have, not what is actually used inside a .zip
file which is -- quite logically now that you point it out -- done on a
file-by-file basis. (Clearly to do otherwise would risk your data and
make changing compressed zips highly inefficient.)

-Peter

 

[Timothy Smith]

Even if the files were compressed, which has a net result similar to
randomizing the contents and will certainly extend the portion that
appears "changed", the worst that would happen is that subversion
(which does use a binary delta algorithm) would end up downloading the
single file portion of the zip file rather than the smaller change
within the file. It should still be efficient.

But to be honest, for something like the OPs purpose, it's not clear
that an SCM is needed, since all he's trying to accomplish is bring a
remote copy up to date with the central one. For that you could just
publish a location containing the necessary files and have the users
use something like rsync directly (which is just as efficient in terms
of a binary delta) to update their own local version.

Of course, if the Subversion server is already in place so it's a
convenient server, or if more of the user base already has the client
in place, it should work just about as well.

-- David

I've got this working now, and fyi it downloads the entire zip every
time. and svn appears to be very slow at it to.

 

[David Bolen]

Good point. When I wrote that I was picturing the form of compression
that a .tar.gz file would have, not what is actually used inside a
.zip file which is -- quite logically now that you point it out --
done on a file-by-file basis. (Clearly to do otherwise would risk
your data and make changing compressed zips highly inefficient.)

Right, and yes, .tar.gz files are very problematic for such
algorithms, such as rsync. In fact, there was a patch made available
for gzip (never made it ito the actual package I believe) that
permitted resetting the compression engine at selected block
boundaries - thus effectively bounding the "noise" generated by a
single change. The output would grow a bit since resetting the engine
dropped overall efficiency, but you got a tremendous gain back in
terms of "rsyncability" of the file.

-- David

 

[David Bolen]

I've got this working now, and fyi it downloads the entire zip every
time. and svn appears to be very slow at it to.

Hmm, not what I would have expected, and certainly unfortunate for
your desired use case.

I just tried some experiments with rsync (easier to test locally than
Subversion), and found that taking an existing zip, unpacking it and
then repacking it with some rearrangement was in fact sending
everything, even though the source files were unchanged. Since py2exe
is effectively rebuilding that library.zip on each run, that probably
is a fair representation of the generation process.

I'm not familiar enough with zip file compression, but perhaps it
includes the use of something that is file specific to seed the
compression engine, which would mean that making a new zip file even
with the same files in it might not yield precisely the same internal
compressed storage. Both versions would be proper and decompressible,
just not binary identical even for unchanged sources.

If I disabled compression for the zip files (just did a store only),
and rebuilt the zip even with a rearranged file order, rsync was
able to detect just the changes.

So you might want to try ensuring that your py2exe generated file is
not compressing the individual modules (a verbose zip listing of the
library.zip should just show that they were "Stored"). Your
library.zip will get larger, but it should become more efficient to
transfer - hopefully as well with Subversion as I was seeing with
rsync.

(In fact, I remember doing just something like this with a project of mine
that I was using py2exe with, and then using rsync to push out the resultant
files to remote sites - I had originally compressed the library.zip but
rsync was pushing the whole thing out, so I stopped using the compression)

-- David

 

[brian]

The zip file essentially contains the whole system in on lump. Change
the system, and naturally your users will have to download the whole
lump again. [...]

but if it was just a dir, when they update from the svn at log in, all
they do is download the extra\changed files. much much quicker then
downloading a 4 meg uncompressed zip file.

Not clear what you are really asking for, but maybe this will help. I
have a application that is 95% normal code, packaged up via py2exe.
Other 5% is a few .py files in a separate directory, modified regularly
by the users (report specifications in some lists/dicts). These files
are *excluded* from the py2exe build, and I manually add this directory
to sys.path during the application startup. A change to one of the
spec files is picked up like any normal python session, and I don't
need to rebuild the entire app. Could you do the same type of thing?

Brian.