Re: py2exe + svn - the final drama

Discussion in 'Python' started by Timothy Smith, May 6, 2005.

  1. Timothy Smith wrote:

    >ok, i am updating my program from my svn - this works, however, i am
    >getting the following error when i close the program. the zip file
    >successfully updates, so i'm curious as to the meaning of this.
    >
    >Traceback (most recent call last):
    > File "Main.pyo", line 820, in ValidateLogin
    > File "Main.pyo", line 863, in ShowMainFrameItems
    > File "glob.pyo", line 22, in glob
    > File "glob.pyo", line 50, in glob1
    > File "fnmatch.pyo", line 42, in filter
    >zipimport.ZipImportError: bad local file header in Z:\temp\library.zip
    >
    >not that once i have finished client.update(''), it has successfully
    >updated the zipfile, i open a dialoge box saying "click ok and restart
    >program" AFTER i click on the above error pops up and my app shuts down
    >as intended.
    >
    >ideas?
    >
    >

    ok i have done some digging and i cound this

    /* Check to make sure the local file header is correct */
    fseek(fp, file_offset, 0);
    l = PyMarshal_ReadLongFromFile(fp);
    if (l != 0x04034B50) {
    /* Bad: Local File Header */
    PyErr_Format(ZipImportError,
    "bad local file header in %s",
    archive);
    fclose(fp);


    can anyone explain to me about zip file headers and why it would be different/incorrect and give me this error?
    return NULL;
     
    Timothy Smith, May 6, 2005
    #1
    1. Advertising

  2. Timothy Smith

    David Bolen Guest

    Timothy Smith <> writes:

    > Timothy Smith wrote:

    (...)
    > >zipimport.ZipImportError: bad local file header in Z:\temp\library.zip
    > >
    > > not that once i have finished client.update(''), it has successfully
    > > updated the zipfile, i open a dialoge box saying "click ok and
    > > restart program" AFTER i click on the above error pops up and my app
    > > shuts down as intended.
    > >
    > >ideas?
    > >

    > ok i have done some digging and i cound this
    >
    > /* Check to make sure the local file header is correct */
    > fseek(fp, file_offset, 0);
    > l = PyMarshal_ReadLongFromFile(fp);
    > if (l != 0x04034B50) {
    > /* Bad: Local File Header */
    > PyErr_Format(ZipImportError,
    > "bad local file header in %s",
    > archive);
    > fclose(fp);
    >
    >
    > can anyone explain to me about zip file headers and why it would be
    > different/incorrect and give me this error?


    Are you perhaps trying to update the zip file in-place while it is still
    being used by the application? I'm not sure that's a safe operation. A
    quick peek at the same module where I think you found the above code shows
    that when a zip importer instance is associated with a zip file, the
    directory for that zip file is read in and cached. So the importer is
    holding onto offset information for each file based on the contents of the
    zip directory at initialization time.

    If you then change the file contents (such as updating it with svn), those
    offsets will no longer be valid. I then expect that during your process
    exit, some bit of code is performing an extra import, which accesses the
    wrong (based on the new file contents) portion of the zip file, and the
    above safety check prevents it from loading an erroneous set of bytes
    thinking its a valid module.

    I expect you need to work on a mechanism to update the file
    independently of the running copy, and then arrange to have it moved
    into place for a subsequent execution. Or find some way to have the
    zip importer refresh its directory information or make a new importer
    instance once the zip file is updated.

    One (untested) thought ... before the update, make a copy of your
    current library.zip as some other name, and adjust your sys.path to
    reference that name (rather than the default pointer to the main
    library.zip that py2exe initializes things with). That should force
    any future imports to access the old copy of the zip file and not the
    one that svn will be updating. Since you need to leave that zip file
    copy in place through the exit (to satisfy any trailing imports),
    arrange for your application to check for that copy on startup and
    remove it if present.

    Or, after looking through import.c handling for zip file imports,
    there might be a simpler way. ZIP imports are handled by a
    zipimporter installed in sys.path_hooks, and once a specific path
    element has a path hook instantiated for it (based on the sys.path
    element name) it is cached in sys.path_hooks_cache.

    So, simply clearing out the path_hooks_cache entry for your main
    library.zip file should cause the next import attempt to re-create a
    new zipimporter instance and thus re-open the file and re-load the
    directory information.

    I don't know if py2exe installs the library.zip into sys.path just as
    "library.zip" or with some path information, but try checking out the
    keys in sys.path_hooks_cache from your application when it is running.
    You should find an entry (probably the only one unless you explicitly
    augment sys.path yourself) for library.zip - clear out that key after
    the update and see how it works.

    Heck, since you're the efficiency hit is likely not an issue, just
    flush all of sys.path_hooks_cache and don't even worry about the
    actual key name for library.zip. So a simple:
    sys.path_importer_cache.clear()
    call after your update completes may do the trick.

    -- David

    PS: In the same way that updating the library.zip under the running
    application is tricky, you might run into issues if you end up trying
    to update one of the extension modules. svn might not be able to
    update it (depending on how it deals with "in use" files).
     
    David Bolen, May 6, 2005
    #2
    1. Advertising

  3. Timothy Smith

    Just Guest

    In article <>,
    David Bolen <> wrote:

    > Are you perhaps trying to update the zip file in-place while it is still
    > being used by the application? I'm not sure that's a safe operation.


    I'm sure it's not :)

    [lots of useful help snipped]

    the zipimport module has an attr called _zip_directory_cache, which is a
    dict you can .clear(). Still, reloading modules is hairy at best, its
    probably easiest to relaunch your app when the .zip file has changed.

    Just
     
    Just, May 6, 2005
    #3
  4. Just wrote:

    >In article <>,
    > David Bolen <> wrote:
    >
    >
    >
    >>Are you perhaps trying to update the zip file in-place while it is still
    >>being used by the application? I'm not sure that's a safe operation.
    >>
    >>

    >
    >I'm sure it's not :)
    >
    >[lots of useful help snipped]
    >
    >the zipimport module has an attr called _zip_directory_cache, which is a
    >dict you can .clear(). Still, reloading modules is hairy at best, its
    >probably easiest to relaunch your app when the .zip file has changed.
    >
    >Just
    >
    >

    what i do is as soon as the update is complete i close the app, but it
    still gives the error, i tried clear() after update and before it, it
    still got the same error. it's be nice to not have to fiddle around with
    the zip file, i really think making py2exe create a dir instead of a zip
    will be much better
    here what i do anyway

    if (os.name == 'nt') or (os.name == 'win32'):

    client = pysvn.Client()
    #get current revision number
    CurrentRev = client.info('').revision.number

    Check = client.update('')
    sys.path_importer_cache.clear()

    if Check.number > CurrentRev:
    self.Popup('Update installed, click ok and restart
    ','Update installed')
    self.Destroy()

    else:
    InfoMsg.Update(3,'No Updates needed')
     
    Timothy Smith, May 7, 2005
    #4
  5. Timothy Smith

    David Bolen Guest

    Just <> writes:

    > the zipimport module has an attr called _zip_directory_cache, which is a
    > dict you can .clear(). Still, reloading modules is hairy at best, its
    > probably easiest to relaunch your app when the .zip file has changed.


    Except that he's getting an error during the process exit of the
    current execution, which is needed to restart. And if he updates to a
    different copy, there's the bootstrap problem of how to get it back
    into the standard location for the next restart since his application
    will need to have it to restart in the first place.

    -- David
     
    David Bolen, May 7, 2005
    #5
  6. Timothy Smith

    David Bolen Guest

    Timothy Smith <> writes:

    > what i do is as soon as the update is complete i close the app, but it
    > still gives the error, i tried clear() after update and before it, it
    > still got the same error. it's be nice to not have to fiddle around
    > with the zip file, i really think making py2exe create a dir instead
    > of a zip will be much better


    Well, you'd still potentially have a problem if the update changed a
    file in that directory that hadn't been imported yet, but now depended
    on other updated files that your application had already loaded old
    versions for. That's a general problem of updating modules beneath
    the executing application, and not really specific to the zip file,
    although you're getting a zip importer specific error related to that
    in this case.

    > here what i do anyway
    >
    > if (os.name == 'nt') or (os.name == 'win32'):
    > client = pysvn.Client()
    > #get current revision number
    > CurrentRev = client.info('').revision.number
    > Check = client.update('')
    > sys.path_importer_cache.clear()
    > if Check.number > CurrentRev:
    > self.Popup('Update installed, click ok and restart
    > ','Update installed')
    > self.Destroy()
    > else:
    > InfoMsg.Update(3,'No Updates needed')


    Ah, it's more devious than I thought. Just pointed out the other
    missing piece in his response.

    Apparently there are two levels of caching that you've got to defeat
    if you change the underlying zip:

    1. A global file set of file directory cache information for any opened
    zip file (for all files in the zip). This is held in the zipimport
    module global _zip_directory_cache.
    2. Individual file cached information within the zipimporter instance
    that is kept in the path importer cache (sys.path_importer_cache).
    Technically these are just references to the same individual entries
    being held in the dictionary from (1).

    So when you cleared out (2), it still found the cached directory at
    the zipimport module level and re-used that information. But if you only
    clear out (1), then the reference in (2) to the directory entries for
    currently imported modules remains and still gets used.

    I tried testing this with a small zip file that I first built with normal
    compression on the entries, then imported one from a running interpreter,
    and then rebuilt the zip without compression. I couldn't seem to get the
    precise error you were getting, but doing this gave me a decompression
    error upon an attempted reload of an imported module, since the cached
    information still thought it was compressed.

    After clearing both sys.path_importer_cache and
    zipimport._zip_directory_cache, the reload went fine.

    It's sort of unfortunate that you have to cheat with the "private"
    cache clearing in this case. It might be worth an enhancement request
    to see if zipimport could know to update itself if the timestamp on
    the zip file changes, but this is sort of a very specialized scenario.
    Although maybe just a public way to cleanly flush import cache
    information would be useful.

    -- David
     
    David Bolen, May 7, 2005
    #6
  7. David Bolen wrote:

    >Timothy Smith <> writes:
    >
    >
    >
    >>what i do is as soon as the update is complete i close the app, but it
    >>still gives the error, i tried clear() after update and before it, it
    >>still got the same error. it's be nice to not have to fiddle around
    >>with the zip file, i really think making py2exe create a dir instead
    >>of a zip will be much better
    >>
    >>

    >
    >Well, you'd still potentially have a problem if the update changed a
    >file in that directory that hadn't been imported yet, but now depended
    >on other updated files that your application had already loaded old
    >versions for. That's a general problem of updating modules beneath
    >the executing application, and not really specific to the zip file,
    >although you're getting a zip importer specific error related to that
    >in this case.
    >
    >
    >
    >>here what i do anyway
    >>
    >>if (os.name == 'nt') or (os.name == 'win32'):
    >> client = pysvn.Client()
    >> #get current revision number
    >> CurrentRev = client.info('').revision.number
    >> Check = client.update('')
    >> sys.path_importer_cache.clear()
    >> if Check.number > CurrentRev:
    >> self.Popup('Update installed, click ok and restart
    >>','Update installed')
    >> self.Destroy()
    >> else:
    >> InfoMsg.Update(3,'No Updates needed')
    >>
    >>

    >
    >Ah, it's more devious than I thought. Just pointed out the other
    >missing piece in his response.
    >
    >Apparently there are two levels of caching that you've got to defeat
    >if you change the underlying zip:
    >
    >1. A global file set of file directory cache information for any opened
    > zip file (for all files in the zip). This is held in the zipimport
    > module global _zip_directory_cache.
    >2. Individual file cached information within the zipimporter instance
    > that is kept in the path importer cache (sys.path_importer_cache).
    > Technically these are just references to the same individual entries
    > being held in the dictionary from (1).
    >
    >So when you cleared out (2), it still found the cached directory at
    >the zipimport module level and re-used that information. But if you only
    >clear out (1), then the reference in (2) to the directory entries for
    >currently imported modules remains and still gets used.
    >
    >I tried testing this with a small zip file that I first built with normal
    >compression on the entries, then imported one from a running interpreter,
    >and then rebuilt the zip without compression. I couldn't seem to get the
    >precise error you were getting, but doing this gave me a decompression
    >error upon an attempted reload of an imported module, since the cached
    >information still thought it was compressed.
    >
    >After clearing both sys.path_importer_cache and
    >zipimport._zip_directory_cache, the reload went fine.
    >
    >It's sort of unfortunate that you have to cheat with the "private"
    >cache clearing in this case. It might be worth an enhancement request
    >to see if zipimport could know to update itself if the timestamp on
    >the zip file changes, but this is sort of a very specialized scenario.
    >Although maybe just a public way to cleanly flush import cache
    >information would be useful.
    >
    >-- David
    >
    >

    awesome it looks like it's working now!
    it's a very convenient why of keeping them all up today, now instead of
    stuffing around making setup packages i can just run my makeexe.bat file
    to create the py2exe files, svn commit -m "new package" and i'm done.
    also there's no way for the staff to bugger it up ( well, it's fairly safe)

    thanks very very much to everyone who made suggestions, this is what
    makes OSS so good - the community input and support.
     
    Timothy Smith, May 7, 2005
    #7
  8. Timothy Smith

    Just Guest

    In article <>,
    David Bolen <> wrote:

    > Just <> writes:
    >
    > > the zipimport module has an attr called _zip_directory_cache, which is a
    > > dict you can .clear(). Still, reloading modules is hairy at best, its
    > > probably easiest to relaunch your app when the .zip file has changed.

    >
    > Except that he's getting an error during the process exit of the
    > current execution, which is needed to restart. And if he updates to a
    > different copy, there's the bootstrap problem of how to get it back
    > into the standard location for the next restart since his application
    > will need to have it to restart in the first place.


    Right. It sounds like a module is imported at exit that wasn't imported
    before. If that's the case, it may help to make sure all modules needed
    for exit are imported beforehand.

    Just
     
    Just, May 7, 2005
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. JFCM
    Replies:
    4
    Views:
    5,756
  2. F. GEIGER
    Replies:
    0
    Views:
    930
    F. GEIGER
    Aug 16, 2004
  3. Timothy Smith

    py2exe + svn - the final drama

    Timothy Smith, May 6, 2005, in forum: Python
    Replies:
    0
    Views:
    477
    Timothy Smith
    May 6, 2005
  4. python
    Replies:
    3
    Views:
    1,824
    Simon Brunning
    May 22, 2006
  5. Replies:
    5
    Views:
    532
    Chris Uppal
    Nov 17, 2006
Loading...

Share This Page