Disable use of pyc file with no matching py file

[Roy Smith]

Every so often (typically when refactoring), I'll remove a .py file and forget to remove the corresponding .pyc file. If I then import the module, python finds the orphaned .pyc and happily imports it. Usually leading to confusing and hard to debug failures.

Is there some way to globally tell python, "Never import a .pyc unless the corresponding .py file exits"? Perhaps some environment variable I could set in my .login file?

 

[Miki Tebeka]

Not that I'm aware of.

I have a script that run the test suite, one of the first commands (before calling nosetests) is:
find . -name '*.py[co]' -exec rm {} \;

This makes sure the tests run in a "clean" environment.

 

[Terry Reedy]

Every so often (typically when refactoring), I'll remove a .py file
and forget to remove the corresponding .pyc file. If I then import
the module, python finds the orphaned .pyc and happily imports it.
Usually leading to confusing and hard to debug failures.

Is there some way to globally tell python, "Never import a .pyc
unless the corresponding .py file exits"?

Upgrade to 3.2.

 

[Jean-Michel Pichavant]

Upgrade to 3.2.

No.

JM

 

[Andrea Crotti]

Every so often (typically when refactoring), I'll remove a .py file and forget to remove the corresponding .pyc file. If I then import the module, python finds the orphaned .pyc and happily imports it. Usually leading to confusing and hard to debug failures.

Is there some way to globally tell python, "Never import a .pyc unless the corresponding .py file exits"? Perhaps some environment variable I could set in my .login file?

If you want to stay python only I have this function:

def clean_orphaned_pycs(directory, simulate=False):
"""Remove all .pyc files without a correspondent module
and return the list of the removed files.
If simulate=True don't remove anything but still return
the list of pycs
"""

exists, join = path.exists, path.join
rm = (lambda _: None) if simulate else remove
removed = []

for root, dirs, files in walk(directory):
for f in files:
if f.endswith(".pyc"):
pyc = join(root, f)
if not exists(pyc[:-1]):
logger.debug("DELETING orphaned file %s" % pyc)
removed.append(pyc)
rm(pyc)

# ignore certain sub-dirs
for i in reversed(range(len(dirs))):
d = dirs
if d == ".svn" or d.endswith(".egg-info"):
dirs.pop(i) # dirs.remove(d)

return removed

 

[Steven D'Aprano]

No.

Is that intended as "No, I won't upgrade" or "No, Python 3.2 doesn't do
the job"?

If the first one, then that's your decision, of course, but there really
is no other straight-forward way to have Python ignore left-over .pyc
files. Like it or not, there is no way to disable the use of .pyc files
in older versions of Python.

(I suppose you could write a patch for Python, and use your custom
version, but that's it.)

The ability to use .pyc files without providing source code is considered
by some people a feature, not a bug, and in fact even in Python 3.2 the
same applies. The difference in 3.2 is that you can't *accidentally* use
old left-over .pyc files, you have to move and rename them before they
can be used as sourceless modules.

 

[Jean-Michel Pichavant]

Is that intended as "No, I won't upgrade" or "No, Python 3.2 doesn't do
the job"?

To answer Ben's mail as well, the "No" would be more of the "don't do
it". My answer was as argued as Terry's one anyway (it was quite intended).
Steven, you often use analogies/similarities, here's one:

A: "My wheel is flat"
B: "Buy a new car"

Buying a new car would solve A's problem : yes
Should A buy a new car : probably no

Python 3.2 is fine, but someone could run into several issues while
migrating. This is quite a pretty huge decision to make (I dedicate this
sentence to Rick, I hope he's trolling fine).

JM

 

[John Roth]

Upgrade to 3.2.

Terry,

I've noticed that the tutorial (section 6.1.3) hasn't been updated for
PEP 3147; there's no way of telling that this is the behavior from
reading the tutorial. The development doc for 3.3 hasn't been updated
either.

 

[Terry Reedy]

I tested before writing this. The caveat is that x.pyc in the same
directly as x.py will not be ignored (for back compatibility). However,
this only happens intentionally as .pyc files are put in __pycache__/

I've noticed that the tutorial (section 6.1.3) hasn't been updated for
PEP 3147; there's no way of telling that this is the behavior from
reading the tutorial. The development doc for 3.3 hasn't been updated
either.

You are right. An oversight. Thanks for noticing.
http://bugs.python.org/issue13915
Suggested rewrites are welcome.

 

[Terry Reedy]

A: "My wheel is flat"
B: "Buy a new car"

A better analogy would be

Q. "How do I make my old model car do something (it cannot do)?"
A. "Get the free new model that has that feature added."

Of course, there is a cost to giving up the old and familiar and
learning and adjusting to the new, even when it is available gratis. A
husband wearing an old sweater after his wife gives him a new one, and
even retrieving it from the trash when she tosses it out, is a classic
(and true) cartoon joke.

But I am sure that 95% of readers here will be using 3.x withing 10
years. The only question for them is "When?". This not-well-known new
feature is one straw that some will put on the 'sooner' side of the balance.

 

[Roy Smith]

But I am sure that 95% of readers here will be using 3.x withing 10
years. The only question for them is "When?". This not-well-known new
feature is one straw that some will put on the 'sooner' side of the balance.

We would love to move to 3.x, for the better unicode support, if nothing
else. What's keeping us from doing so is the host of third-party
modules and tools we depend on that don't yet support 3.x. Off the top
of my head (it's possible some of these already have support):

django (I understand it's been done, if not yet officially supported)
pymongo
mongoengine
tornado
pybeanstalk
dateutils
requests
lxml
blinker
gunicorn
I'm sure I've missed a few

It's getting to the point where any open-source python project needs to
either jump on the 3.x bandwagon or get consigned to obscurity. I'm
guessing that's going to start happening in a big way in 2012.

 

[Jean-Michel Pichavant]

A better analogy would be

Q. "How do I make my old model car do something (it cannot do)?"
A. "Get the free new model that has that feature added."

Of course, there is a cost to giving up the old and familiar and
learning and adjusting to the new, even when it is available gratis. A
husband wearing an old sweater after his wife gives him a new one, and
even retrieving it from the trash when she tosses it out, is a classic
(and true) cartoon joke.

But I am sure that 95% of readers here will be using 3.x withing 10
years. The only question for them is "When?". This not-well-known new
feature is one straw that some will put on the 'sooner' side of the
balance.

A simple solution to that problem has been provided by Miki. Someone
should read at least http://wiki.python.org/moin/Python2orPython3 before
migrating to python 3.

Speaking for myself, I'm stuck with python 2.5 :-/

JM

 

[Terry Reedy]

We would love to move to 3.x, for the better unicode support, if nothing
else. What's keeping us from doing so is the host of third-party
modules and tools we depend on that don't yet support 3.x.

Tell that to the authors of packages you use so they no longer say that
they have not converted for lack of demand ;-)

 

[Devin Jeanpierre]

Q. "How do I make my old model car do something (it cannot do)?"
A. "Get the free new model that has that feature added."

Of course, there is a cost to giving up the old and familiar and learning
and adjusting to the new, even when it is available gratis. A husband
wearing an old sweater after his wife gives him a new one, and even
retrieving it from the trash when she tosses it out, is a classic (and true)
cartoon joke.

It really bothers me that you imagine that there are no other problems
than the newness. It's disheartening, because the problems are not
that trivial and the world would be better if people were less callous
about it, and realized that they exist. Python 3 is not very different
from Python 2, as far as humans are concerned
semantically/syntactically -- but, hell, just pick any project that
uses PyPy, or Jython, or IronPython, or Twisted, or Zope, etc. -- it
can be a lot of effort (sometimes infeasibly much) to port something
dependent on these things, and it's taken years to get the (smallish)
set of dependencies ported that we have now [and we literally paid
people to do it, too!], and still many large projects haven't made the
transition, and many small projects never will.

Anyone that relies on those projects is stuck, and your "free car"
metaphor completely ignores the true cost of wasting that much time
porting everything for a tiny little feature. Evaluating only the
monetary amounts can be misleading as to what the rational decision is
(in particular when there are no monetary amounts). The only true
notion of cost is the alternatives you sacrifice in making a decision:
opportunity cost. The car is not free.

-- Devin

 

[John Roth]

I tested before writing this. The caveat is that x.pyc in the same
directly as x.py will not be ignored (for back compatibility). However,
this only happens intentionally as .pyc files are put in __pycache__/


You are right. An oversight. Thanks for noticing.http://bugs.python.org/issue13915
Suggested rewrites are welcome.

I'll see if I can put a suggestion in the bug report.

One other point: I'm unclear if a compiled module in the source
directory would be named spam.pyc or spam.cpython-32.pyc. I'd think
the latter to allow two versions of a compiled-only distribution.

John Roth

 

[Terry Reedy]

It really bothers me that you imagine that there are no other problems
than the newness.

And it bothers me that you imput such ignorance to me. You made what I
think was a bad analogy and I made a better one of the same type, though
still imperfect. I acknowledged that the transition will take years.

 

[Terry Reedy]

One other point: I'm unclear if a compiled module in the source
directory would be named spam.pyc or spam.cpython-32.pyc. I'd think
the latter to allow two versions of a compiled-only distribution.

By test, it has to be spam.pyc, as before.

 

[Devin Jeanpierre]

And it bothers me that you imput such ignorance to me. You made what I think
was a bad analogy and I made a better one of the same type, though still
imperfect. I acknowledged that the transition will take years.

Ah. It is a common attitude among those that make these sorts of
comments about Python 3, and I hadn't read anything in what you said
that made me think that you were considering more than the superficial
costs of moving. I am sorry that I did not give you the benefit of the
doubt.

-- Devin

 

[Terry Reedy]

Ah. It is a common attitude among those that make these sorts of
comments about Python 3, and I hadn't read anything in what you said
that made me think that you were considering more than the superficial
costs of moving.

I thought '95% in 10 years' would be a hint that I know upgrading is not
trivial for every one ;-).

I am sorry that I did not give you the benefit of the doubt.

Apology accepted.

 

[Brian]

I thought '95% in 10 years' would be a hint that I know
upgrading is not trivial for every one ;-).


Apology accepted.

This is OT, but relevant to the side discussion. Also note that
the community respects and appreciates the T.J. Reedy
contributions and hard work.

Reality. It was a monumental task to convert ATE driver dev and
related factory automation stuff from C to Python starting in
2000. Most was done guerrilla-style. My employer now appreciates
the resultant infrastructure that even the Mexico factory
engineering team can use, and is good for their pride because
they are no longer totally dependent on gringo engineers.

Am currently being disruptive to their little world with my
recent (1Q 2011) switch to 3.x. The corporate natives are
restless and there is talk of an armed insurrection. Humans, at
the tribal level, do not adapt to change. Expect a long series
of battles that are ruthless, bloody, and have a high body
count. Vive la Revolution.