Re: leftover pyc files

Discussion in 'Python' started by Andrea Crotti, Nov 2, 2011.

  1. On 11/02/2011 03:03 PM, Peter Otten wrote:
    > Switch to Python3.2 ;)


    Yes I saw that and it would be great, unfortunately we're stuck with
    Python 2.5 :O
    for some more time.

    Anyway now the code that does it is a recursive thing ilke
    def _clean_orphaned_pycs(self, directory, simulate=False):
    """Remove all the pyc files without a correspondent module
    """
    files = listdir(directory)
    # going down the tree recursively (like os.walk)
    for x in files:
    if x.endswith('.pyc'):
    py_file = x.split('.pyc')[0] + '.py'

    if (not simulate) and (py_file not in files):
    deletable_pyc = path.join(directory, x)
    print 'DELETING pyc %s as it has no py %s' % \
    (deletable_pyc, py_file)
    remove(deletable_pyc)

    #TODO: move out these special cases
    if path.isdir(path.join(directory, x)) \
    and x != '.svn' \
    and not x.endswith('.egg-info'):

    self._clean_orphaned_pycs(path.join(directory, x))


    Can be done much better probably with os.walk or os.path.walk,
    any suggestions?
     
    Andrea Crotti, Nov 2, 2011
    #1
    1. Advertising

  2. A task like this is more suited to bash than Python:

    find . -name '*.pyc' -exec rm '{}' ';'
     
    Jonathan Hartley, Nov 3, 2011
    #2
    1. Advertising

  3. A task like this is more suited to bash than Python:

    find . -name '*.pyc' -exec rm '{}' ';'
     
    Jonathan Hartley, Nov 3, 2011
    #3
  4. This can be added to git as a post-checkout hook:

    In your project's .git/hooks/post-checkout:

    #!/usr/bin/env bash
    cd ./$(git rev-parse --show-cdup)
    find . -name '*.pyc' -exec rm '{}' ';'
     
    Jonathan Hartley, Nov 3, 2011
    #4
  5. This can be added to git as a post-checkout hook:

    In your project's .git/hooks/post-checkout:

    #!/usr/bin/env bash
    cd ./$(git rev-parse --show-cdup)
    find . -name '*.pyc' -exec rm '{}' ';'
     
    Jonathan Hartley, Nov 3, 2011
    #5
  6. This can be added to your project's .git/hooks/post-checkout:

    #!/usr/bin/env bash
    cd ./$(git rev-parse --show-cdup)
    find . -name '*.pyc' -exec rm '{}' ';'
     
    Jonathan Hartley, Nov 3, 2011
    #6
  7. This can be added to your project's .git/hooks/post-checkout:

    #!/usr/bin/env bash
    cd ./$(git rev-parse --show-cdup)
    find . -name '*.pyc' -exec rm '{}' ';'
     
    Jonathan Hartley, Nov 3, 2011
    #7
  8. Andrea Crotti

    Peter Otten Guest

    Jonathan Hartley wrote:

    > A task like this is more suited to bash than Python:
    >
    > find . -name '*.pyc' -exec rm '{}' ';'


    You forgot to exclude the .svn directory from the search and didn't limit
    the deletion to pyc-files whose corresponding py-file doesn't exist.

    How would your line look with these modifications?
     
    Peter Otten, Nov 3, 2011
    #8
  9. All these ideas (shell and git hooks) are nice, but unfortunately
    - it's svn not git
    - it's windows not *nix
    - we have to remove only the ones without the corresponding *py...
     
    Andrea Crotti, Nov 3, 2011
    #9
  10. On Fri, Nov 4, 2011 at 4:54 AM, Andrea Crotti <> wrote:
    > All these ideas (shell and git hooks) are nice, but unfortunately
    > - it's svn not git
    > - it's windows not *nix
    > - we have to remove only the ones without the corresponding *py...


    Windows? Well, Windows shell scripting isn't quite as rich as
    bash/csh/etc, but it's possible. I'll leave recursion as an exercise
    for the reader, but this (untested) should do it for one directory:

    for %d in (*.pyc) do (
    set fn=%d
    if not exist !fn:~0,-1! del %d
    )

    This needs to be run with 'cmd /v' to enable delayed variable
    evaluation. Of course, it'd be really easy to do if Windows Batch had
    anything like decent string manipulation.

    ChrisA
     
    Chris Angelico, Nov 3, 2011
    #10
  11. On Thu, Nov 3, 2011 at 1:54 PM, Andrea Crotti <> wrote:
    > All these ideas (shell and git hooks) are nice, but unfortunately
    > - it's svn not git
    > - it's windows not *nix
    > - we have to remove only the ones without the corresponding *py...
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >

    Barely tested. Change the print functions to removes. Pass the top
    directory as the argument.

    import os, sys
    for dirpath, dirnames, filenames in os.walk(sys.argv[1]):
    if dirpath.endswith("__pycache__"):
    thispath = dirpath[:-11]
    for f in filenames:
    if f.endswith(".pyc"):
    if not os.path.exists(os.path.join(thispath,f[:-1])):
    print("delete " + os.path.join(thispath,f))

    else:
    for f in filenames:
    if f.endswith(".pyc"):
    if not os.path.exists(os.path.join(dirpath,f[:-1])):
    print("delete " + os.path.join(dirpath,f))
    #os.remove(os.path.join(dirpath,f))
     
    David Robinow, Nov 4, 2011
    #11
  12. On Thu, 03 Nov 2011 17:54:52 +0000, Andrea Crotti wrote:

    > All these ideas (shell and git hooks) are nice, but unfortunately - it's
    > svn not git
    > - it's windows not *nix
    > - we have to remove only the ones without the corresponding *py...


    Does it matter? The other .pyc files will be recreated when they are next
    needed. Just nuke all the .pyc files and be done with it. Or if you can't
    bear having to wait for Python to compile them as needed, you can force
    compilation by running your test suite (you do have a test suite, don't
    you?) or by using the compileall.py script.

    python -m compileall some_directory

    should do the trick.


    --
    Steven
     
    Steven D'Aprano, Nov 4, 2011
    #12
  13. I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this.

    svn also has hooks, but sadly not a checkout hook:
    http://svnbook.red-bean.com/en/1.1/ch05s02.html
    I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files.

    I don't understand why deleting only some pyc files is important. Can't youjust delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects.

    Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpage.
     
    Jonathan Hartley, Nov 4, 2011
    #13
  14. I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this.

    svn also has hooks, but sadly not a checkout hook:
    http://svnbook.red-bean.com/en/1.1/ch05s02.html
    I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files.

    I don't understand why deleting only some pyc files is important. Can't youjust delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects.

    Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpage.
     
    Jonathan Hartley, Nov 4, 2011
    #14
  15. Apologies for all my messasges appearing twice. I'm using google groups web ui and have no idea why it's doing that. I'll stop using it.
     
    Jonathan Hartley, Nov 4, 2011
    #15
  16. On 11/04/2011 09:27 AM, Jonathan Hartley wrote:
    > I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this.
    >
    > svn also has hooks, but sadly not a checkout hook:
    > http://svnbook.red-bean.com/en/1.1/ch05s02.html
    > I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files.
    >
    > I don't understand why deleting only some pyc files is important. Can't you just delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects.
    >
    > Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpa


    Uhm yes it makes sense also to just remove all of them, I don't know why
    it was done like this but
    probably for "performance" reasons.

    I will try both ways, but I would definitively avoid the shell
    scripting, because it's mainly windows
    but it has to be multi-platform..
     
    Andrea Crotti, Nov 4, 2011
    #16
  17. On Fri, Nov 4, 2011 at 9:34 PM, Andrea Crotti <> wrote:
    > Uhm yes it makes sense also to just remove all of them, I don't know why it
    > was done like this but
    > probably for "performance" reasons.
    >
    > I will try both ways, but I would definitively avoid the shell scripting,
    > because it's mainly windows
    > but it has to be multi-platform..


    If you're removing them all, you don't need to use a powerful shell.
    Much much easier! Just recursively del *.pyc and you're done.

    ChrisA
     
    Chris Angelico, Nov 4, 2011
    #17
  18. On 11/04/2011 10:39 AM, Chris Angelico wrote:
    > If you're removing them all, you don't need to use a powerful shell.
    > Much much easier! Just recursively del *.pyc and you're done. ChrisA


    Discussing with the guy that did it I think it's actually a good idea
    instead, because removing them *all* means.
    - remove a few thousand files every time where maybe 0 would be removed
    otherwise
    - recompile the same thousand of files

    It might not be so relevant but for 10 lines of code more I guess it's
    fine..
     
    Andrea Crotti, Nov 4, 2011
    #18
  19. Uhn, thanks for the easy way Just delete all *.pyc recursively. spend another 5-20
    minutes to recompile all to get everything sync.. That is trivial!
     
    88888 Dihedral, Nov 4, 2011
    #19
  20. Uhn, thanks for the easy way Just delete all *.pyc recursively. spend another 5-20
    minutes to recompile all to get everything sync.. That is trivial!
     
    88888 Dihedral, Nov 4, 2011
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rim
    Replies:
    8
    Views:
    970
  2. Ladvánszky Károly

    pyc files

    Ladvánszky Károly, Feb 3, 2004, in forum: Python
    Replies:
    3
    Views:
    403
    Rene Pijlman
    Feb 3, 2004
  3. Andrea Crotti

    leftover pyc files

    Andrea Crotti, Nov 2, 2011, in forum: Python
    Replies:
    0
    Views:
    222
    Andrea Crotti
    Nov 2, 2011
  4. Hal Fulton
    Replies:
    9
    Views:
    214
    Gregory Millam
    Apr 28, 2004
  5. Mircescu Andrei
    Replies:
    3
    Views:
    104
    Mircescu Andrei
    Feb 20, 2014
Loading...

Share This Page