Race condition when generating .pyc files

[yogamatt1970]

I have a large body of Python code which runs on many different (Unix)
machines concurrently. Part of the code lives in one place, but most
of it lives in directories which I find at runtime. I only have one
copy of each Python source file and I think I'm hitting a race
condition where two hosts attempt to import the same module at the
same time. My import fails on one of the machines and the following
exception is thrown:
EOFError: EOF read where object expected
My hypothesis is that there's contention between the two (or more)
hosts when the module's .pyc file is generated.

Possible solutions I see:
1) Running a cron job before my code executes which compiles all the
python source first.
2) Making separate copies of all the .py files for each host running
the code - I'd rather not do this, it seems like a big pain.
3) Inhibiting the generation of .pyc files altogether if that's even
possible - I saw a PEP for a possible -R flag (http://www.python.org/
dev/peps/pep-0304/) but I don't think it has been added to a released
version of Python yet plus I am stuck with Python 2.4 for the time
being.

Any and all help appreciated.
Thanks.

 

[Gerhard Häring]

I have a large body of Python code which runs on many different (Unix)
machines concurrently. Part of the code lives in one place, but most
of it lives in directories which I find at runtime. I only have one
copy of each Python source file and I think I'm hitting a race
condition where two hosts attempt to import the same module at the
same time. My import fails on one of the machines and the following
exception is thrown:
EOFError: EOF read where object expected
My hypothesis is that there's contention between the two (or more)
hosts when the module's .pyc file is generated.

Possible solutions I see:
1) Running a cron job before my code executes which compiles all the
python source first.
2) Making separate copies of all the .py files for each host running
the code - I'd rather not do this, it seems like a big pain.
3) Inhibiting the generation of .pyc files altogether if that's even
possible [...]

If you don't want pyc files to be created, you could set Unix
permissions such that Python cannot write to the directory.

-- Gerhard

 

[Aaron \Castironpi\ Brady]

I have a large body of Python code which runs on many different (Unix)
machines concurrently.  Part of the code lives in one place, but most
of it lives in directories which I find at runtime.  I only have one
copy of each Python source file and I think I'm hitting a race
condition where two hosts attempt to import the same module at the
same time.  My import fails on one of the machines and the following
exception is thrown:
EOFError: EOF read where object expected
My hypothesis is that there's contention between the two (or more)
hosts when the module's .pyc file is generated.

Acquire an flock on the .py file, then compile it.

 

[Bruno Desthuilliers]

(e-mail address removed) a écrit :

I have a large body of Python code which runs on many different (Unix)
machines concurrently. Part of the code lives in one place, but most
of it lives in directories which I find at runtime. I only have one
copy of each Python source file and I think I'm hitting a race
condition where two hosts attempt to import the same module at the
same time. My import fails on one of the machines and the following
exception is thrown:
EOFError: EOF read where object expected
My hypothesis is that there's contention between the two (or more)
hosts when the module's .pyc file is generated.

Possible solutions I see:
1) Running a cron job before my code executes which compiles all the
python source first.

If you package your apps using setup, pyc should be automatically
generated. Don't know if it can apply to your problem. But surely I'd go
this way (ie : automating pyc creation one way or another).

2) Making separate copies of all the .py files for each host running
the code - I'd rather not do this, it seems like a big pain.
yeps.

3) Inhibiting the generation of .pyc files altogether if that's even
possible - I saw a PEP for a possible -R flag (http://www.python.org/
dev/peps/pep-0304/) but I don't think it has been added to a released
version of Python yet plus I am stuck with Python 2.4 for the time
being.

Not having pyc means the modules will have to be recompiled on each
import. Not sure this is what you want.

 

[yogamatt1970]

If you package your apps using setup, pyc should be automatically

generated. Don't know if it can apply to your problem. But surely I'd go
this way (ie : automating pyc creation one way or another).

Yeah, I don't package up my code, it's all integrated into my build
system,
not an actual deliverable.

Not having pyc means the modules will have to be recompiled on each
import. Not sure this is what you want.

I realize that that would be slower, but I think it would avoid the
problem
I'm seeing because no host would actually be creating a .pyc file.
But is it even possible in Python 2.4?

 

[Terry Reedy]

Yeah, I don't package up my code, it's all integrated into my build
system,
not an actual deliverable.

Can you integrate compileall into the build process (probably used by
setup)?
Help on module compileall:

NAME
compileall - Module/script to "compile" all .py files to .pyc (or
..pyo) file.

FILE
c:\programs\python30\lib\compileall.py

DESCRIPTION
When called as a script with arguments, this compiles the directories
given as arguments recursively; the -l option prevents it from
recursing into directories.

Without arguments, if compiles all modules on sys.path, without
recursing into subdirectories. (Even though it should do so for
packages -- for now, you'll have to deal with packages separately.)

See module py_compile for details of the actual byte-compilation.

FUNCTIONS
compile_dir(dir, maxlevels=10, ddir=None, force=0, rx=None, quiet=0)
Byte-compile all modules in the given directory tree.

Arguments (only dir is required):

dir: the directory to byte-compile
maxlevels: maximum recursion level (default 10)
ddir: if given, purported directory name (this is the
directory name that will show up in error messages)
force: if 1, force compilation, even if timestamps are
up-to-date
quiet: if 1, be quiet during compilation

compile_path(skip_curdir=1, maxlevels=0, force=0, quiet=0)
Byte-compile all module on sys.path.

Arguments (all optional):

skip_curdir: if true, skip current directory (default true)
maxlevels: max recursion level (default 0)
force: as for compile_dir() (default 0)
quiet: as for compile_dir() (default 0)

DATA
__all__ = ['compile_dir', 'compile_path']

 

[Gabriel Genellina]

En Tue, 07 Oct 2008 12:21:40 -0300, (e-mail address removed)

I have a large body of Python code which runs on many different (Unix)
machines concurrently. Part of the code lives in one place, but most
of it lives in directories which I find at runtime. I only have one
copy of each Python source file and I think I'm hitting a race
condition where two hosts attempt to import the same module at the
same time. My import fails on one of the machines and the following
exception is thrown:
EOFError: EOF read where object expected
My hypothesis is that there's contention between the two (or more)
hosts when the module's .pyc file is generated.

Possible solutions I see:
1) Running a cron job before my code executes which compiles all the
python source first.

You could make your source directories not writeable by the user under
those processes run. Then make the cron job at 1) compile all new sources
(of course *this* process should have write permission). Or better, watch
those directories for changes and compile when needed.
Use the py_compile module for that.

2) Making separate copies of all the .py files for each host running
the code - I'd rather not do this, it seems like a big pain.
3) Inhibiting the generation of .pyc files altogether if that's even
possible - I saw a PEP for a possible -R flag (http://www.python.org/
dev/peps/pep-0304/) but I don't think it has been added to a released
version of Python yet plus I am stuck with Python 2.4 for the time
being.

Yes, with Python 2.6 (and 3.0) you may use the -B option or set the
PYTHONDONTWRITEBYTECODE env. var., but 2.4 doesn't support it. And having
to recompile every module when it is imported may be time consuming.

 

[Steve Holden]

I have a large body of Python code which runs on many different (Unix)
machines concurrently. Part of the code lives in one place, but most
of it lives in directories which I find at runtime. I only have one
copy of each Python source file and I think I'm hitting a race
condition where two hosts attempt to import the same module at the
same time. My import fails on one of the machines and the following
exception is thrown:
EOFError: EOF read where object expected
My hypothesis is that there's contention between the two (or more)
hosts when the module's .pyc file is generated.

Possible solutions I see:
1) Running a cron job before my code executes which compiles all the
python source first.
2) Making separate copies of all the .py files for each host running
the code - I'd rather not do this, it seems like a big pain.
3) Inhibiting the generation of .pyc files altogether if that's even
possible - I saw a PEP for a possible -R flag (http://www.python.org/
dev/peps/pep-0304/) but I don't think it has been added to a released
version of Python yet plus I am stuck with Python 2.4 for the time
being.

Any and all help appreciated.

Are you using the same version of Python on all the hosts that are using
the code? If not, the different versions will be forever arguing about
whether the .pyc files need recompiling.

Just a thought ...

regards
Steve

 

[yogamatt1970]

I think this is my best option for now - I'm going to give it a shot.
Thanks.

 

[yogamatt1970]

Ugggh, I'm not using the exact same version everywhere. Of course,
the mystery to me is that this just started failing recently,
everything has been fine until
last week.
Anyway, thanks for the info.