Cross compile generation of .pyc from .py files...

[venkatbo]

Hi all,

We've managed to cross-compile (from i686 targeting ppc) python (2.4.2)
binaries and
extension modules.

However we cannot figure out how to cross-compile the .py (of
distribution) files and
generate the .pyc files for the target ppc from an i686 system. Is it
possible to cross
compile .pyc files from .py files?

Some of the errors we get when we run 'python -v' on the target system
include:
.....
# /usr/lib/python2.4/site.pyc has bad magic
import site # from /usr/lib/python2.4/site.py

# /usr/lib/python2.4/types.pyc has bad magic
import types # from /usr/lib/python2.4/types.py
......

Thanks,
/venkat

 

[Terry Reedy]

However we cannot figure out how to cross-compile the .py (of
distribution) files and
generate the .pyc files for the target ppc from an i686 system. Is it
possible to cross
compile .pyc files from .py files?

I am under the impression that .pyc files are system independent.
Have you tried simply copying them over?

tjr

 

[ookoi]

You should look at this blog entry:
http://www.voidspace.org.uk/python/weblog/arch_d7_2006_02_11.shtml#e222

But if you have the original .py it's quite insane to want to hack the
..pyc only for porting it under others architectures. Instead, use
setuptools.

 

[venkatbo]

...
I am under the impression that .pyc files are system independent.
Have you tried simply copying them over?

tjr

Hi Terry,

It appears that python on the target ppc system is expecting to see
ppc-related info in the .pyc files. These .pyc were generated at cross
compile time (on i686 system) and packaged, deployed, installed on the
ppc system. The "...has bad magic..." appears to indicate that
ppc-version of python is expecting to see ppc-specific .pyc files, but
is encountering i686-specific .pyc files... For some reason, the
cross-compile step that results in the .pyc files is not generating
them for the proper ppc-target, but is building them for the i686
system where they were being cross-compiled...

Thanks,
/venkat

 

[Just]

Hi Terry,

It appears that python on the target ppc system is expecting to see
ppc-related info in the .pyc files.

There is no such thing.

These .pyc were generated at cross
compile time (on i686 system) and packaged, deployed, installed on the
ppc system. The "...has bad magic..." appears to indicate that
ppc-version of python is expecting to see ppc-specific .pyc files, but
is encountering i686-specific .pyc files... For some reason, the
cross-compile step that results in the .pyc files is not generating
them for the proper ppc-target, but is building them for the i686
system where they were being cross-compiled...

..pyc files are only compatible with the same major Python version, so it
sounds like you're using different versions on both platforms.

Just

 

[venkatbo]

Hi Sebastian,

Thanks for that link and your notes.

I was under the impression, that the .pyc files will be used (if found)
by python to speed up execution of scripts... and so we packaged,
deployed and installed the .py/.pyc files on to the ppc-target system.
That package includes, site.py(c), types.py(c) etc., among others.

Though I see these errors when I invokde 'python -v', I'm able to
execute the .py files... but was hoping to use the .pyc files to
benefit from the enhanced speed of execution. Like I mentioned to
Terry, for some reason only the cross compile generation of .pyc for
the ppc-target is not getting done right, whereas the actual
ppc-specific python (2.4.2) binaries and extension (.so) modules are
getting created properly.

We were not trying to just ship the .pyc files and reverse-engineer the
..py files from them on the target ppc system.

Thanks,
/venkat

 

[Terry Reedy]

ppc system. The "...has bad magic..." appears to indicate that

The format of .pyc files, which are generated for greater speed of repeat
runs, is considered an internal implementation detail subject to change.
The exact details are generally specific to each x,y version. (For
instance, byte codes are occasionally added.) So each version generally
has a version-specific magic number and will only run .pyc files with the
same magic number. If you get 'bad magic' from the interpreter, then you
have a mismatch.

If you are shipping .py files, there is no need to also ship .pyc files.
Let them be generated as needed or run compileall at installation.

Terry Jan Reedy

 

[David Wahler]

I was under the impression, that the .pyc files will be used (if found)
by python to speed up execution of scripts... and so we packaged,
deployed and installed the .py/.pyc files on to the ppc-target system.
That package includes, site.py(c), types.py(c) etc., among others.

Though I see these errors when I invokde 'python -v', I'm able to
execute the .py files... but was hoping to use the .pyc files to
benefit from the enhanced speed of execution.

[snip...]

..pyc files do not increase the speed of execution -- only the speed of
importing modules. Even then, the benefit is only seen the first time
the modules are loaded, as the .pyc files will be generated
automatically. I'd say the tiny, one-time speedup isn't worth it; and
if you really want the modules to be precompiled, it's better to do it
at installation time using distutils.

-- David