Module baldy compiled to pyc?

[Laszlo Nagy]

Today I had a strange experience. I have copied some updated py files
(modules) to a directory on a remote server, overwritting the old ones.
The pyc files on the server were older. Many programs are importing
these modules, and most of them are started as background jobs (from
cron). They started to throw all kinds of errors. I checked the py
files, and they did have class definitions inside. However when I tried
to use them I got AttributeError exceptions. Telling that those things
are not in the module.

I checked their contents by importing them, and they were not defined
indeed. Finally I have checked the module.__file__ attributes to see
that they are imported from the right place. The __file__ contained the
path to the compiled pyc file, but the path was correct. So finally I
have deleted all pyc files, and suddenly every program was working
again. (Starting the interpreter again and importing the modules again
did not solve the problem.)

I suspect that there were two (or more) programs starting at the same
time, writting the same pyc file at the same time. It happened with two
modules today. Over the years, I have always copied files to this
server, and let background programs compile the pyc files as needed. I
have never experienced anything like this before, and I cannot reproduce
the error.

The question is this: do you think this could happen? Is it possible
that something else caused the problem? What else could it be? What are
the chances that it will happen again, and how can I prevent it?

Thanks,

Laszlo

 

[Steven D'Aprano]

Today I had a strange experience. I have copied some updated py files
(modules) to a directory on a remote server, overwritting the old ones.
The pyc files on the server were older. Many programs are importing
these modules, and most of them are started as background jobs (from
cron). They started to throw all kinds of errors. I checked the py
files, and they did have class definitions inside. However when I tried
to use them I got AttributeError exceptions. Telling that those things
are not in the module.

It sounds to me like the timestamps on the .py files were messed up when
you copied them, causing Python to prefer old, broken .pyc files instead
of the newer .py files.

I suspect that there were two (or more) programs starting at the same
time, writting the same pyc file at the same time.

I suppose that's not impossible, although you would need to check the
Python source code to be sure. But I would expect that each process
writes the .pyc file atomically, so even if two processes both try to
create the .pyc file, one will harmlessly overwrite the other's output. I
would be surprised if it were possible for two processes to
simultaneously write to the same file -- that would require the Python
compiler to explicitly open the .pyc file in shared writable mode, and
why would it do that?

Far more likely is a file system error. I recommend you run a disk check
on the file server and see if it comes up with any errors.