__pycache__, one more good reason to stck with Python 2?

[Steven D'Aprano]

It is now practically impossible to launch a Python application via a
.pyc file.

When has that ever been possible?


..pyc files are Python byte-code. You can't run them directly using Python
(except via the import machinery), you can't run them as a script,
they're not machine code. Unless you write a wrapper to import the file
as a module, you can't directly execute .pyc files.

(For the fun, try to add the "parent directory" of a cached
file to the sys.path).

I don't understand what you're getting at there.

About the caches, I'am just fearing, they will become finally garbage
collectors of orphan .pyc files, Python has seed/seeded(?). The .pyc
files may not be very pleasant, but at least you can use them and you
have that "feeling" of their existence. I my "computer experience", once
you start to cache/hide something for simplicity, the problems start.

I will say that in general I'm not very happy with what seems like every
second application creating huge disk caches in random locations. It's a
PITA, and I wish they'd standardize on a single cache location, to make
it easy to exclude them from backups, to sanitize the bred-crumbs and
data trails applications leave, and so forth.

But having said that, the __pycache__ idea isn't too bad. If you have
this directory structure:

../module.py
../module.pyc

and import module, the top-level .pyc file will continue to be used. If
you delete module.py, leaving only the top-level .pyc, it will still
continue to be used. That much hasn't changed.

The only difference is if you start with only a .py file:

../module.py

and import it, the cached version will be placed into __pycache__
instead. Unlike orphaned .pyc files in the top level, orphaned .pyc in
the __pycache__ are considered stale and will be ignored. So they're safe
to ignore, and safe to delete, at any time.

Nothing is stopping you from moving the .pyc file into the top level and
deleting the .py file, if you so desire.

 

[Antoine Pitrou]

When has that ever been possible?


.pyc files are Python byte-code. You can't run them directly using Python
(except via the import machinery), you can't run them as a script,
they're not machine code. Unless you write a wrapper to import the file
as a module, you can't directly execute .pyc files.

It seems to work here:

$ echo "print 'foo'" > toto.py
$ python -m compileall -l .
Listing . ...
Compiling ./toto.py ...
$ rm toto.py
$ python toto.pyc
foo


But it still works under 3.2 even though the incantation is less pretty:

$ echo "import sys; print(sys.version)" > toto.py
$ __svn__/python -m compileall -l .
Listing . ...
Compiling ./toto.py ...
$ rm toto.py
$ __svn__/python __pycache__/toto.cpython-32.pyc
3.2rc1+ (py3k:88095M, Jan 18 2011, 17:12:15)
[GCC 4.4.3]


Regards

Antoine.

 

[jmfauth]

My "nightmare" was mainly due, because when I read the
the "What's new?", I did not understand too much this
caching stuff. It's only later, after testing some
applications, I really got the surprise to understand
it. (Py3.1 and Py3.2 pyc's mixture).

Having said this, to tell you the truth. I do really
not feel comfortable with it -- two "working directories",
a subdir in a working dir containing only a few scripts,
filenames, cache multiplications (I would have 1202 caches
on my hd now).

There are certainly advantages, just wondering if the
balance is positive ou negative.

Just for the story, I'm already somwehow experencing some
funny stuff. My test working dir has already a full bloated
cache. You may argue, that's my fault. I may reply: bof,
that Python which is doing it...)

(As a Windows users, I just wondering how this will
impact tools like py2exe or cx_freeze).

-----

Antoine Pitrou

Yes, I can launch a pyc, when I have a single file.
But what happens, if one of your cached .pyc file import
a module with a name as defined in the parent directory?
The machinery is broken. The parent dir is not in the
sys.path.

(Unless I'm understanding nothing or I have done a huge
mistake in my tests).

jmf

 

[Antoine Pitrou]

Yes, I can launch a pyc, when I have a single file.
But what happens, if one of your cached .pyc file import
a module with a name as defined in the parent directory?
The machinery is broken. The parent dir is not in the
sys.path.

Well, you don't have to launch a pyc file anyway. Put all your code in
some (pyc) modules on the standard Python path, and use a clear-text
script with some trivial code to invoke a function from the compiled
modules.

Otherwise you can customize sys.path a little from your script,
using __file__ and os.path.dirname. Nothing complicated AFAICT.

(by the way, the fact that pyc files are version-specific should
discourage any use outside of version-specific directories,
e.g. /usr/lib/pythonX.Y/site-packages)

Regards

Antoine.

 

[jmfauth]

Well, you don't have to launch a pyc file anyway. Put all your code in
some (pyc) modules on the standard Python path, and use a clear-text
script with some trivial code to invoke a function from the compiled
modules.

That's not the point. I'm toying. And the "behaviour" from now
is deeply different from what it was.

Otherwise you can customize sys.path a little from your script,
using __file__ and os.path.dirname. Nothing complicated AFAICT.

That's not as simple. You have too prepare the py file in such a way,
that it recognizes the path of its ancestor py file. The "home dir"
is no more the same.

(by the way, the fact that pyc files are version-specific should
discourage any use outside of version-specific directories,
e.g. /usr/lib/pythonX.Y/site-packages)

Here we are. I'm just wondering if all this stuff is not just
here in order to satisfy the missmatched Python installation on
*x platforms compared to Windows whre each Python version
lives cleanely in its isolated space. (Please no os war).

jmf



compared to

 

[Carl Banks]

But having said that, the __pycache__ idea isn't too bad. If you have
this directory structure:

./module.py
./module.pyc

and import module, the top-level .pyc file will continue to be used.

Nope. PEP 3147 says it now always uses __pycache__.


Carl Banks

 

[Steven D'Aprano]

It seems to work here:

[snip incantation]

Then I stand corrected, thank you.

I know I've seen problems executing .pyc files from the shell in the
past... perhaps I was conflating details of something else. Ah, I know!

[steve@sylar ~]$ chmod u+x toto.pyc
[steve@sylar ~]$ ./toto.pyc
: command not found ��
../toto.pyc: line 2: syntax error near unexpected token `('
../toto.pyc: line 2: `P7Mc@s dGHdS(tfooN((((s ./
toto.py<module>s'

 

[Steven D'Aprano]

Nope. PEP 3147 says it now always uses __pycache__.

Looks like the PEP is outdated then.

[steve@sylar ~]$ rm __pycache__/*
[steve@sylar ~]$ echo "print('spam spam spam')" > spam.py
[steve@sylar ~]$ python3.2 -m compileall spam.py
Compiling spam.py ...
[steve@sylar ~]$ mv __pycache__/spam.cpython-32.pyc ./spam.pyc
[steve@sylar ~]$ python3.2 -m spam
spam spam spam
[steve@sylar ~]$ ls __pycache__/
[steve@sylar ~]$

 

[Alice Bevan–McGregor]

I know I've seen problems executing .pyc files from the shell in the
past... perhaps I was conflating details of something else. Ah, I know!

[steve@sylar ~]$ chmod u+x toto.pyc
[steve@sylar ~]$ ./toto.pyc
: command not found ��
./toto.pyc: line 2: syntax error near unexpected token `('
./toto.pyc: line 2: `P7Mc@s dGHdS(tfooN((((s ./
toto.py<module>s'

.... don't do that. I do not know why that would be expected to work,
ever. (Unless you're crafty and wrap a real shell script around the
..pyc, in which case it's no longer a .pyc.)

— Alice.

 

[John Pinner]

Hi

You have disturbe my slumber, Steven ;-)

When has that ever been possible?

.pyc files are Python byte-code. You can't run them directly using Python
(except via the import machinery), you can't run them as a script,
they're not machine code. Unless you write a wrapper to import the file
as a module, you can't directly execute .pyc files.

Not true. 'python myprog.pyc' has worked for as long as I have been
using Python. Whether or not it is a good idea is another matter.

Probably it would be best to check what you're saying before posting
such a bald assertion, even though you were 110% sure of what you were
saying. I've been there, done that, bought the tee-shirt.

Best wishes,

John
--

 

[Steven D'Aprano]

Hi

You have disturbe my slumber, Steven ;-)



Not true. 'python myprog.pyc' has worked for as long as I have been
using Python. Whether or not it is a good idea is another matter.

Probably it would be best to check what you're saying before posting
such a bald assertion, even though you were 110% sure of what you were
saying. I've been there, done that, bought the tee-shirt.

Yes, thank you, I've already been corrected over this

But you got me thinking... how far back does this behaviour go?
Apparently it goes back as long as I've been using Python too:

[steve@sylar ~]$ echo "print 'spam spam spam'" > spam.py
[steve@sylar ~]$ python1.5
Python 1.5.2 (#1, Apr 1 2009, 22:55:54) [GCC 4.1.2 20070925 (Red Hat
4.1.2-27)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam[steve@sylar ~]$ python1.5 spam.pyc
spam spam spam


But of course, .pyc files1.5 are version specific:


[steve@sylar ~]$ python2.5 spam.pyc
RuntimeError: Bad magic number in .pyc file

 

[Martin v. Loewis]

I know I've seen problems executing .pyc files from the shell in the

past... perhaps I was conflating details of something else. Ah, I know!

[steve@sylar ~]$ chmod u+x toto.pyc
[steve@sylar ~]$ ./toto.pyc
: command not found ��
./toto.pyc: line 2: syntax error near unexpected token `('
./toto.pyc: line 2: `P7Mc@s dGHdS(tfooN((((s ./
toto.py<module>s'

Works fine here:

martin@mira:/tmp$ cat >a.py
print "Hello, world"
martin@mira:/tmp$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
py> import a
Hello, world
py>
martin@mira:/tmp$ chmod +x a.pyc
martin@mira:/tmp$ ./a.pyc
Hello, world

Notice that it is a Linux feature that it can support Python bytecode
as a "native" executable when properly configured.

Windows has a very similar feature, though. I think OSX can support
it as well, but I haven't tried.

Regards,
Martin

 

[Martin v. Loewis]

But you got me thinking... how far back does this behaviour go?

=================================
==> Release 1.1 (11 Oct 1994) <==
=================================

- Passing the interpreter a .pyc file as script argument will execute
the code in that file. (On the Mac such files can be double-clicked!)
- New module compileall generates .pyc files for all modules in a
directory (tree) without also executing them

[Notice that "on the Mac" refers to Mac System 7 here]

=======================================
==> Release 1.0.0 (26 January 1994) <==
=======================================

* It is now possible to have a .pyc file without a corresponding .py
file. (Warning: this may break existing installations if you have an
old .pyc file lingering around somewhere on your module search path
without a corresponding .py file, when there is a .py file for a
module of the same name further down the path -- the new interpreter
will find the first .pyc file and complain about it, while the old
interpreter would ignore it and use the .py file further down.)

Regards,
Martin

 

[Steven D'Aprano]

I know I've seen problems executing .pyc files from the shell in the
past... perhaps I was conflating details of something else. Ah, I know!

[steve@sylar ~]$ chmod u+x toto.pyc
[steve@sylar ~]$ ./toto.pyc
: command not found ��
./toto.pyc: line 2: syntax error near unexpected token `(' ./toto.pyc:
line 2: `P7Mc@s dGHdS(tfooN((((s ./ toto.py<module>s'

... don't do that. I do not know why that would be expected to work,
ever. (Unless you're crafty and wrap a real shell script around the
.pyc, in which case it's no longer a .pyc.)

I didn't expect it to work, but I have seen others do it and be surprised
that it doesn't. This is why I was pleasantly surprised to learn that
`python toto.pyc` does work -- I was conflating the above failure with
the issue under discussion.

 

[systemBuilder]

Why give it such a goofy name ??

I for one have no idea why they chose the goofy name __pycache__ for the directory. It seems the python architects have a serious misunderstanding of the difference between a compiler token and a file name.

why not .pycache ?? At least that way it would be hidden on *NIX systems (all future computer systems, IMHO ...)

- DWG