__pycache__, one more good reason to stck with Python 2?

Discussion in 'Python' started by jmfauth, Jan 17, 2011.


  1. When has that ever been possible?


    ..pyc files are Python byte-code. You can't run them directly using Python
    (except via the import machinery), you can't run them as a script,
    they're not machine code. Unless you write a wrapper to import the file
    as a module, you can't directly execute .pyc files.

    I don't understand what you're getting at there.

    I will say that in general I'm not very happy with what seems like every
    second application creating huge disk caches in random locations. It's a
    PITA, and I wish they'd standardize on a single cache location, to make
    it easy to exclude them from backups, to sanitize the bred-crumbs and
    data trails applications leave, and so forth.

    But having said that, the __pycache__ idea isn't too bad. If you have
    this directory structure:

    ../module.py
    ../module.pyc

    and import module, the top-level .pyc file will continue to be used. If
    you delete module.py, leaving only the top-level .pyc, it will still
    continue to be used. That much hasn't changed.

    The only difference is if you start with only a .py file:

    ../module.py

    and import it, the cached version will be placed into __pycache__
    instead. Unlike orphaned .pyc files in the top level, orphaned .pyc in
    the __pycache__ are considered stale and will be ignored. So they're safe
    to ignore, and safe to delete, at any time.

    Nothing is stopping you from moving the .pyc file into the top level and
    deleting the .py file, if you so desire.
     
    Steven D'Aprano, Jan 19, 2011
    #21
    1. Advertisements

  2. It seems to work here:

    $ echo "print 'foo'" > toto.py
    $ python -m compileall -l .
    Listing . ...
    Compiling ./toto.py ...
    $ rm toto.py
    $ python toto.pyc
    foo


    But it still works under 3.2 even though the incantation is less pretty:

    $ echo "import sys; print(sys.version)" > toto.py
    $ __svn__/python -m compileall -l .
    Listing . ...
    Compiling ./toto.py ...
    $ rm toto.py
    $ __svn__/python __pycache__/toto.cpython-32.pyc
    3.2rc1+ (py3k:88095M, Jan 18 2011, 17:12:15)
    [GCC 4.4.3]


    Regards

    Antoine.
     
    Antoine Pitrou, Jan 19, 2011
    #22
    1. Advertisements

  3. jmfauth

    jmfauth Guest

    My "nightmare" was mainly due, because when I read the
    the "What's new?", I did not understand too much this
    caching stuff. It's only later, after testing some
    applications, I really got the surprise to understand
    it. (Py3.1 and Py3.2 pyc's mixture).

    Having said this, to tell you the truth. I do really
    not feel comfortable with it -- two "working directories",
    a subdir in a working dir containing only a few scripts,
    filenames, cache multiplications (I would have 1202 caches
    on my hd now).

    There are certainly advantages, just wondering if the
    balance is positive ou negative.

    Just for the story, I'm already somwehow experencing some
    funny stuff. My test working dir has already a full bloated
    cache. You may argue, that's my fault. I may reply: bof,
    that Python which is doing it...)

    (As a Windows users, I just wondering how this will
    impact tools like py2exe or cx_freeze).

    -----

    Antoine Pitrou

    Yes, I can launch a pyc, when I have a single file.
    But what happens, if one of your cached .pyc file import
    a module with a name as defined in the parent directory?
    The machinery is broken. The parent dir is not in the
    sys.path.

    (Unless I'm understanding nothing or I have done a huge
    mistake in my tests).

    jmf
     
    jmfauth, Jan 19, 2011
    #23
  4. Well, you don't have to launch a pyc file anyway. Put all your code in
    some (pyc) modules on the standard Python path, and use a clear-text
    script with some trivial code to invoke a function from the compiled
    modules.

    Otherwise you can customize sys.path a little from your script,
    using __file__ and os.path.dirname. Nothing complicated AFAICT.

    (by the way, the fact that pyc files are version-specific should
    discourage any use outside of version-specific directories,
    e.g. /usr/lib/pythonX.Y/site-packages)

    Regards

    Antoine.
     
    Antoine Pitrou, Jan 19, 2011
    #24
  5. jmfauth

    jmfauth Guest

    That's not the point. I'm toying. And the "behaviour" from now
    is deeply different from what it was.
    That's not as simple. You have too prepare the py file in such a way,
    that it recognizes the path of its ancestor py file. The "home dir"
    is no more the same.

    Here we are. I'm just wondering if all this stuff is not just
    here in order to satisfy the missmatched Python installation on
    *x platforms compared to Windows whre each Python version
    lives cleanely in its isolated space. (Please no os war).

    jmf



    compared to
     
    jmfauth, Jan 19, 2011
    #25
  6. jmfauth

    Carl Banks Guest

    Nope. PEP 3147 says it now always uses __pycache__.


    Carl Banks
     
    Carl Banks, Jan 19, 2011
    #26
  7. [snip incantation]

    Then I stand corrected, thank you.

    I know I've seen problems executing .pyc files from the shell in the
    past... perhaps I was conflating details of something else. Ah, I know!

    [[email protected] ~]$ chmod u+x toto.pyc
    [[email protected] ~]$ ./toto.pyc
    : command not found ��
    ../toto.pyc: line 2: syntax error near unexpected token `('
    ../toto.pyc: line 2: `[email protected] dGHdS(tfooN((((s ./
    toto.py<module>s'
     
    Steven D'Aprano, Jan 19, 2011
    #27
  8. Steven D'Aprano, Jan 19, 2011
    #28
  9. .... don't do that. I do not know why that would be expected to work,
    ever. (Unless you're crafty and wrap a real shell script around the
    ..pyc, in which case it's no longer a .pyc.)

    — Alice.
     
    Alice Bevan–McGregor, Jan 19, 2011
    #29
  10. jmfauth

    John Pinner Guest

    Hi

    You have disturbe my slumber, Steven ;-)

    Not true. 'python myprog.pyc' has worked for as long as I have been
    using Python. Whether or not it is a good idea is another matter.

    Probably it would be best to check what you're saying before posting
    such a bald assertion, even though you were 110% sure of what you were
    saying. I've been there, done that, bought the tee-shirt.

    Best wishes,

    John
    --
     
    John Pinner, Jan 20, 2011
    #30
  11. Yes, thank you, I've already been corrected over this :)

    But you got me thinking... how far back does this behaviour go?
    Apparently it goes back as long as I've been using Python too:

    [[email protected] ~]$ echo "print 'spam spam spam'" > spam.py
    [[email protected] ~]$ python1.5
    Python 1.5.2 (#1, Apr 1 2009, 22:55:54) [GCC 4.1.2 20070925 (Red Hat
    4.1.2-27)] on linux2
    Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam[[email protected] ~]$ python1.5 spam.pyc
    spam spam spam


    But of course, .pyc files1.5 are version specific:


    [[email protected] ~]$ python2.5 spam.pyc
    RuntimeError: Bad magic number in .pyc file
     
    Steven D'Aprano, Jan 21, 2011
    #31
  12. I know I've seen problems executing .pyc files from the shell in the
    Works fine here:

    [email protected]:/tmp$ cat >a.py
    print "Hello, world"
    [email protected]:/tmp$ python
    Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
    [GCC 4.4.5] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    py> import a
    Hello, world
    py>
    [email protected]:/tmp$ chmod +x a.pyc
    [email protected]:/tmp$ ./a.pyc
    Hello, world

    Notice that it is a Linux feature that it can support Python bytecode
    as a "native" executable when properly configured.

    Windows has a very similar feature, though. I think OSX can support
    it as well, but I haven't tried.

    Regards,
    Martin
     
    Martin v. Loewis, Jan 21, 2011
    #32
  13. But you got me thinking... how far back does this behaviour go?

    =================================
    ==> Release 1.1 (11 Oct 1994) <==
    =================================

    - Passing the interpreter a .pyc file as script argument will execute
    the code in that file. (On the Mac such files can be double-clicked!)
    - New module compileall generates .pyc files for all modules in a
    directory (tree) without also executing them

    [Notice that "on the Mac" refers to Mac System 7 here]

    =======================================
    ==> Release 1.0.0 (26 January 1994) <==
    =======================================

    * It is now possible to have a .pyc file without a corresponding .py
    file. (Warning: this may break existing installations if you have an
    old .pyc file lingering around somewhere on your module search path
    without a corresponding .py file, when there is a .py file for a
    module of the same name further down the path -- the new interpreter
    will find the first .pyc file and complain about it, while the old
    interpreter would ignore it and use the .py file further down.)

    Regards,
    Martin
     
    Martin v. Loewis, Jan 21, 2011
    #33
  14. I didn't expect it to work, but I have seen others do it and be surprised
    that it doesn't. This is why I was pleasantly surprised to learn that
    `python toto.pyc` does work -- I was conflating the above failure with
    the issue under discussion.
     
    Steven D'Aprano, Jan 21, 2011
    #34
  15. jmfauth

    systemBuilder

    Joined:
    Jul 24, 2013
    Messages:
    1
    Likes Received:
    0
    Why give it such a goofy name ??

    I for one have no idea why they chose the goofy name __pycache__ for the directory. It seems the python architects have a serious misunderstanding of the difference between a compiler token and a file name.

    why not .pycache ?? At least that way it would be hidden on *NIX systems (all future computer systems, IMHO ...)

    - DWG
     
    systemBuilder, Jul 24, 2013
    #35
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.