Building python packages for the correct architecture on OSX 10.5

[Arnaud Delobelle]

Hi fellow python enthusiasts.

Having recently acquired a MacBook Pro (Intel Core 2 Duo) which comes
with python2.5, I have been installing some modules that I need (PIL,
psycopg2, PyXML ...).

The problem is that [$python setup.py build] compiles all the binaries
to universal files for i386 and ppc32, but not x86_64 or ppc64. It
does not appear to be a problem when running scripts from the shell
(as python seems to run as a 32 bits problems), but it is a problem
from apache2/mod_python as the included apache2 runs as 64 bits
processes.

This means the modules need to be compiles for at least both i386 and
x86_64 in my case. I have been looking at the setup.py files of
various modules but I cannot see a suitable way to indicate what
architectures I want them compiled for. So far, I have managed by
adding the following lines in setup.py just after the Extension class
is imported:

OrigExtension = Extension
def Extension(*args, **kwargs):
extra_args = ['-arch', 'ppc', '-arch', 'ppc64',
'-arch', 'i386', '-arch', 'x86_64 ']
kwargs['extra_compile_args'] = extra_args +
kwargs.get('extra_compile_args', [])
kwargs['extra_link_args'] = extra_args +
kwargs.get('extra_link_args', [])
return OrigExtension(*args, **kwargs)


Obviously this is a dirty hack, and I would like to know how to do
this the right way. How can this be done better?

 

[Kevin Walzer]

Hi fellow python enthusiasts.

Having recently acquired a MacBook Pro (Intel Core 2 Duo) which comes
with python2.5, I have been installing some modules that I need (PIL,
psycopg2, PyXML ...).

The problem is that [$python setup.py build] compiles all the binaries
to universal files for i386 and ppc32, but not x86_64 or ppc64. It
does not appear to be a problem when running scripts from the shell
(as python seems to run as a 32 bits problems), but it is a problem
from apache2/mod_python as the included apache2 runs as 64 bits
processes.

This means the modules need to be compiles for at least both i386 and
x86_64 in my case. I have been looking at the setup.py files of
various modules but I cannot see a suitable way to indicate what
architectures I want them compiled for. So far, I have managed by
adding the following lines in setup.py just after the Extension class
is imported:

OrigExtension = Extension
def Extension(*args, **kwargs):
extra_args = ['-arch', 'ppc', '-arch', 'ppc64',
'-arch', 'i386', '-arch', 'x86_64 ']
kwargs['extra_compile_args'] = extra_args +
kwargs.get('extra_compile_args', [])
kwargs['extra_link_args'] = extra_args +
kwargs.get('extra_link_args', [])
return OrigExtension(*args, **kwargs)


Obviously this is a dirty hack, and I would like to know how to do
this the right way. How can this be done better?

You may want to post this on the MacPython list--there are plenty of
experts there on building Python mudles for OS X.

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

This means the modules need to be compiles for at least both i386 and

x86_64 in my case.

Building Python in 64-bit mode as a universal (fat) binary is not
supported in Python 2.5, period. So any solution you come necessarily
has to be a work-around.

The only solution I can see is to make a plain, non-fat installation
of Python in 64-bit mode, and then use that installation to build
64-bit extension modules.

def Extension(*args, **kwargs):
extra_args = ['-arch', 'ppc', '-arch', 'ppc64',
'-arch', 'i386', '-arch', 'x86_64 ']

This cannot really work, for two reasons:
a) even if your extension module becomes x86_64 with that mechanism,
the Python interpreter itself (i.e. the Python framework) will be
purely 32-bit code. So it should not link correctly.
b) During configure, Python generates a pyconfig.h which has the
computed sizes of data types (such as int, long, size_t). It only
has a single such file, and the file is generated only during
configure. Therefore, the data in it cannot work both for 32-bit
and 64-bit architectures. When you compile for a 64-bit target
using the 32-bit pyconfig.h, the code may work incorrectly
(provided it makes use of the computed values somewhere) (*)

(*) It is surprising that pyconfig.h actually works for both
big-endian (ppc) and little-endian (i386) systems, even though
it computes the endianness during configure only once. This is
due to an OSX-specific hack in pyconfig.h, which hides the
definition of the computed endianness value, and uses the
value that the compiler provides as a macro instead.

Regards,
Martin

 

[Arnaud Delobelle]

This means the modules need to be compiles for at least both i386 and
x86_64 in my case.

Building Python in 64-bit mode as a universal (fat) binary is not
supported in Python 2.5, period. So any solution you come necessarily
has to be a work-around.

The only solution I can see is to make a plain, non-fat installation
of Python in 64-bit mode, and then use that installation to build
64-bit extension modules.

def Extension(*args, **kwargs):
extra_args = ['-arch', 'ppc', '-arch', 'ppc64',
'-arch', 'i386', '-arch', 'x86_64 ']

This cannot really work, for two reasons:
a) even if your extension module becomes x86_64 with that mechanism,
the Python interpreter itself (i.e. the Python framework) will be
purely 32-bit code. So it should not link correctly.

My machine disagrees:

marigold:~ arno$ file /System/Library/Frameworks/Python.framework/
Python
/System/Library/Frameworks/Python.framework/Python: Mach-O universal
binary with 4 architectures
/System/Library/Frameworks/Python.framework/Python (for architecture
ppc7400): Mach-O dynamically linked shared library ppc
/System/Library/Frameworks/Python.framework/Python (for architecture
ppc64): Mach-O 64-bit dynamically linked shared library ppc64
/System/Library/Frameworks/Python.framework/Python (for architecture
i386): Mach-O dynamically linked shared library i386
/System/Library/Frameworks/Python.framework/Python (for architecture
x86_64): Mach-O 64-bit dynamically linked shared library x86_64

b) During configure, Python generates a pyconfig.h which has the
computed sizes of data types (such as int, long, size_t). It only
has a single such file, and the file is generated only during
configure. Therefore, the data in it cannot work both for 32-bit
and 64-bit architectures. When you compile for a 64-bit target
using the 32-bit pyconfig.h, the code may work incorrectly
(provided it makes use of the computed values somewhere) (*)

(*) It is surprising that pyconfig.h actually works for both
big-endian (ppc) and little-endian (i386) systems, even though
it computes the endianness during configure only once. This is
due to an OSX-specific hack in pyconfig.h, which hides the
definition of the computed endianness value, and uses the
value that the compiler provides as a macro instead.

Thanks for the details.
I have had no problems with the modules I have compiled so far, they
have been working in 32 and 64 bits. Maybe I was just lucky? I'll
have to look into this more, then. Python on OSX 10.5 has been a
challenge so far

Regards,
Martin

Thanks

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

a) even if your extension module becomes x86_64 with that mechanism,

My machine disagrees:

I see. I guess Apple has implemented that somehow; the official Python
release does not support such an operation.

Regards,
Martin