LinearAlgebra incredibly slow for eigenvalue problems

[drife]

Hello,

I need to calculate the eigenvectors and eigenvalues for a 3600 X 3600
covariance matrix.

The LinearAlgebra package in Python is incredibly slow to perform the
above calculations (about 1.5 hours). This in spite of the fact that
I have installed Numeric with the full ATLAS and LAPACK libraries.

Also note that my computer has dual Pentium IV (3.1 GHz) processors
with 2Gb ram.

Every Web discussion I have seen about such issues indicates that
one can expect huge speed ups if one compiles and installs Numeric
linked against the ATLAS and LAPACK libraries.

Even more perplexing is that the same calculation takes a mere 7 min
in Matlab V6.5. Matlab uses both ATLAS and LAPACK.

Moreover, the above calculation takes the same amount of time for
Numeric to complete with --and-- without ATLAS and PACK. I am certain
that I have done the install correctly.

Can anyone provide some insight?
Thanks in advance for your help.


Daran

 

[David M. Cooke]

Hello,

I need to calculate the eigenvectors and eigenvalues for a 3600 X 3600
covariance matrix.

The LinearAlgebra package in Python is incredibly slow to perform the
above calculations (about 1.5 hours). This in spite of the fact that
I have installed Numeric with the full ATLAS and LAPACK libraries.

Also note that my computer has dual Pentium IV (3.1 GHz) processors
with 2Gb ram.

Every Web discussion I have seen about such issues indicates that
one can expect huge speed ups if one compiles and installs Numeric
linked against the ATLAS and LAPACK libraries.

Are you *sure* that Numeric is linked against these?

Even more perplexing is that the same calculation takes a mere 7 min
in Matlab V6.5. Matlab uses both ATLAS and LAPACK.

Moreover, the above calculation takes the same amount of time for
Numeric to complete with --and-- without ATLAS and PACK. I am certain
that I have done the install correctly.

This is good evidence that Numeric *isn't* linked to them.

If you're on a Linux system, you can check with ldd:
cookedm@arbutus$ ldd /usr/lib/python2.3/site-packages/Numeric/lapack_lite.so
liblapack.so.3 => /usr/lib/atlas/liblapack.so.3 (0x0000002a95677000)
libblas.so.3 => /usr/lib/atlas/libblas.so.3 (0x0000002a95e55000)
libg2c.so.0 => /usr/lib/libg2c.so.0 (0x0000002a96721000)
libpthread.so.0 => /lib/libpthread.so.0 (0x0000002a96842000)
libc.so.6 => /lib/libc.so.6 (0x0000002a96957000)
libm.so.6 => /lib/libm.so.6 (0x0000002a96b96000)
/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

You can see that lapack and blas (the Atlas versions) are linked to
the lapack_lite.so.

To install Numeric using Lapack:
- remove the build/ directory in your Numeric sources, so you don't
any old binaries
- edit setup.py and follow the comments on using Lapack (you need to
comment out a few lines, and set some directories)
Also set use_dotblas to 1.
- do the 'python setup.py build', 'python setup.py install' dance.

 

[drife]

Hi David,

I performed the above check, and sure enough, Numeric
is --not-- linked to the ATLAS libraries.

I followed each of your steps outlined above, and Numeric
still is not linking to the ATLAS libraries.

My setup.py file is attached below.

Thanks ,

Daran

--#!/usr/bin/env python
# To use:
# python setup.py install
# or:
# python setup.py bdist_rpm (you'll end up with RPMs in dist)
#
import os, sys, string, re
from glob import glob
if not hasattr(sys, 'version_info') or sys.version_info <
(2,0,0,'alpha',0):
raise SystemExit, "Python 2.0 or later required to build Numeric."
import distutils
from distutils.core import setup, Extension

# Get all version numbers
execfile(os.path.join('Lib','numeric_version.py'))
numeric_version = version

execfile(os.path.join('Packages', 'MA', 'Lib', 'MA_version.py'))
MA_version = version

headers = glob (os.path.join ("Include","Numeric","*.h"))
extra_compile_args = [] # You could put "-O4" etc. here.
mathlibs = ['m']
define_macros = [('HAVE_INVERSE_HYPERBOLIC',None)]
undef_macros = []
# You might need to add a case here for your system
if sys.platform in ['win32']:
mathlibs = []
define_macros = []
undef_macros = ['HAVE_INVERSE_HYPERBOLIC']
elif sys.platform in ['mac', 'beos5']:
mathlibs = []

# delete all but the first one in this list if using your own
LAPACK/BLAS
sourcelist = [os.path.join('Src', 'lapack_litemodule.c')]
# set these to use your own BLAS;

library_dirs_list = ['/d2/lib/atlas']
libraries_list = ['lapack', 'ptcblas', 'ptf77blas', 'atlas', 'g2c']

# set to true (1), if you also want BLAS optimized
matrixmultiply/dot/innerproduct
use_dotblas = 1
include_dirs = ['/d2/include']
# You may need to set this to find cblas.h
# e.g. on UNIX using ATLAS this should be
['/usr/include/atlas']
extra_link_args = []

# for MacOS X to link against vecLib if present
VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework'
if os.path.exists(VECLIB_PATH):
extra_link_args = ['-framework', 'vecLib']
include_dirs = [os.path.join(VECLIB_PATH, 'Headers')]

# The packages are split in this way to allow future optional inclusion
# Numeric package
packages = ['']
package_dir = {'': 'Lib'}
include_dirs.append('Include')
ext_modules = [
Extension('_numpy',
[os.path.join('Src', '_numpymodule.c'),
os.path.join('Src', 'arrayobject.c'),
os.path.join('Src', 'ufuncobject.c')],
extra_compile_args = extra_compile_args),
Extension('multiarray',
[os.path.join('Src', 'multiarraymodule.c')],
extra_compile_args = extra_compile_args),
Extension('umath',
[os.path.join('Src', 'umathmodule.c')],
libraries = mathlibs,
define_macros = define_macros,
undef_macros = undef_macros,
extra_compile_args = extra_compile_args),
Extension('arrayfns',
[os.path.join('Src', 'arrayfnsmodule.c')],
extra_compile_args = extra_compile_args),
Extension('ranlib',
[os.path.join('Src', 'ranlibmodule.c'),
os.path.join('Src', 'ranlib.c'),
os.path.join('Src', 'com.c'),
os.path.join('Src', 'linpack.c')],
extra_compile_args = extra_compile_args),
Extension('lapack_lite', sourcelist,
library_dirs = library_dirs_list,
libraries = libraries_list,
extra_link_args = extra_link_args,
extra_compile_args = extra_compile_args)
]

# add FFT package (optional)
packages.append('FFT')
package_dir['FFT'] = os.path.join('Packages','FFT','Lib')
include_dirs.append(os.path.join('Packages','FFT','Include'))
ext_modules.append(Extension('FFT.fftpack',
[os.path.join('Packages','FFT','Src',
'fftpackmodule.c'),
os.path.join('Packages', 'FFT', 'Src',
'fftpack.c')],
extra_compile_args = extra_compile_args))

# add MA package (optional)
packages.append('MA')
package_dir['MA'] = os.path.join('Packages', 'MA', 'Lib')

# add RNG package (optional)
packages.append('RNG')
packages.append('RNG')
package_dir['RNG'] = os.path.join('Packages', 'RNG', 'Lib')
include_dirs.append(os.path.join('Packages', 'RNG', 'Include'))
ext_modules.append(Extension('RNG.RNG',
[os.path.join('Packages', 'RNG', 'Src',
'RNGmodule.c'),
os.path.join('Packages', 'RNG', 'Src',
'ranf.c'),
os.path.join('Packages', 'RNG', 'Src',
'pmath_rng.c')],
extra_compile_args = extra_compile_args))
# add dotblas package (optional)
if use_dotblas:
packages.append('dotblas')
package_dir['dotblas'] = os.path.join('Packages', 'dotblas',
'dotblas')
ext_modules.append(Extension('_dotblas',
[os.path.join('Packages', 'dotblas',
'dotblas', '_dotblas.c')],
library_dirs = library_dirs_list,
libraries = libraries_list,

extra_compile_args=extra_compile_args))




long_description = """
Numerical Extension to Python with subpackages.

The authors and maintainers of the subpackages are:

FFTPACK-3.1
maintainer = "Numerical Python Developers"
maintainer_email = "(e-mail address removed)"
description = "Fast Fourier Transforms"
url = "http://numpy.sourceforge.net"

MA-%s
author = "Paul F. Dubois"
description = "Masked Array facility"
maintainer = "Paul F. Dubois"
maintainer_email = "(e-mail address removed)"
url = "http://sourceforge.net/projects/numpy"

RNG-3.1
author = "Lee Busby, Paul F. Dubois, Fred Fritsch"
maintainer = "Paul F. Dubois"
maintainer_email = "(e-mail address removed)"
description = "Cray-like Random number package."
""" % (MA_version, )

# Oops, another bug in Distutils!?
# Write rpm_build.sh pointing to this python
rpm_build_text="""env CFLAGS="$RPM_OPT_FLAGS" %setup.py build\n""" %
sys.executable
rpm_script = open("rpm_build.sh", "w")
rpm_script.write(rpm_build_text)
rpm_script.close()

# Write rpm_install.sh pointing to this python
rpm_install_text=sys.executable +""" setup.py install
--root=$RPM_BUILD_ROOT

cat >INSTALLED_FILES <<EOF
%doc Demo
EOF
find $RPM_BUILD_ROOT -type f | sed -e "s|$RPM_BUILD_ROOT||g"
"""
rpm_script = open("rpm_install.sh", "w")
rpm_script.write(rpm_install_text)
rpm_script.close()

setup (name = "Numeric",
version = numeric_version,
maintainer = "Numerical Python Developers",
maintainer_email = "(e-mail address removed)",
description = "Numerical Extension to Python",
long_description = long_description,
url = "http://numpy.sourceforge.net",

# distutils allows you to fix or fudge anything
extra_path = 'Numeric',
packages = packages,
package_dir = package_dir,
headers = headers,
include_dirs = include_dirs,
ext_modules = ext_modules
)

print 'MA Version', MA_version
print 'Numeric Version', numeric_version

 

[John Hunter]

drife> Hi David, I performed the above check, and sure enough,
drife> Numeric is --not-- linked to the ATLAS libraries.

drife> I followed each of your steps outlined above, and Numeric
drife> still is not linking to the ATLAS libraries.

drife> My setup.py file is attached below.

Are you sure that you

1) 'rm -rf build' before rebuilding

2) are using the python / Numeric that you think you are, eg, might
you have one python in /usr/bin and another in /usr/local. root
paths can differ from user paths which may effect which python is
used at install time versus run time

 

[drife]

Hi John,

I do have more than one version of Python laying around.

To do the build and install I am typing:

/d2/python/bin/python setup.by build > &! build.out
/d2/python/bin/python setup.by install > &! install.out
Should I be doing something different?

Daran

 

[David M. Cooke]

Hi David,

I performed the above check, and sure enough, Numeric
is --not-- linked to the ATLAS libraries.

I followed each of your steps outlined above, and Numeric
still is not linking to the ATLAS libraries.

My setup.py file is attached below.

# delete all but the first one in this list if using your own LAPACK/BLAS
sourcelist = [os.path.join('Src', 'lapack_litemodule.c')]
# set these to use your own BLAS;

library_dirs_list = ['/d2/lib/atlas']
libraries_list = ['lapack', 'ptcblas', 'ptf77blas', 'atlas', 'g2c']

# set to true (1), if you also want BLAS optimized matrixmultiply/dot/innerproduct
use_dotblas = 1
include_dirs = ['/d2/include']

This all look right (assuming you've got the right stuff in /d2).

When it compiles, does it look like it's actually doing the linking?
After doing python setup.py build, you can run ldd on the libraries in
the build directory (something like build/lib.linux-i386-2.3/lapack_lite.so).
If that's linked, then it's not being installed right.

You don't have a previous Numeric installation that's being picked up
instead of the one you're trying to install, do you?

At the interpreter prompt, check that
gives you something you're expecting, and not something else.

 

[drife]

Hi David,

Yes, when Numeric compiles it does look like the linking is being
done properly. I captured the build to a file and see a few lines
similar to:


gcc -pthread -shared build/temp.linux-i686-2.4/Src/lapack_litemodule.o
-L/d2/lib/atlas -llapack -lptcblas -lptf77blas -latlas -lg2c -o
build/lib.linux-i686-2.4/lapack_lite.so

I checked the files in build/lib.linux-i686-2.4 and none have
dependencies on ATLAS.

When I run Python interpreter with:

I get the answer I expect.
I am totall baffled.

Any other ideas?

Thanks for your help,

Daran

 

[drife]

David,

One more thing. I checked to see if the SciPy libraries had
dependencies on ATLAS. They do not, however, the eigenvector
calculation is still much faster than Numeric?
This is very strange.

Daran

 

[drife]

David,

I noticed that the libraries that ATLAS builds are not shared
objects (e.g., liblapack.a). Should these be shared objects?
I see nothing in the ATLAS documentation about building
things as shared objects.
Wondering if this is why the Numeric install is failing.


Daran

 

[Robert Kern]

David,

I noticed that the libraries that ATLAS builds are not shared
objects (e.g., liblapack.a). Should these be shared objects?
I see nothing in the ATLAS documentation about building
things as shared objects.
Wondering if this is why the Numeric install is failing.

No, they should work just fine as static libraries.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter