2d array slicing problem

[jeg]

dear all,

i'm an astronomer working with 2d images -- 2d numarrays. i have a
script which basically does some operations on some images, and one of
the first steps is to find a galaxy on an image (at, say, a known x,y
coord), and create a sub-image by slicing out part of the larger array
to create a more managable smaller one (e.g. 50x50 pixels from a
1000x1000 array).

i was running this fine under python 2.3.4, but due to the slowness of
my machine, moved to a speedy 64-bit linux box running version 2.4.2.
arrays were no longer being sliced in the correct way, for example:

B = A[3:7,6:10]

generates different Bs if i run an identical script on the same image
on the different machines.

note: we deal with FITS format images, which are read into numarrays
using the pyfits.py module. does anyone know any obvious reason this
should be happening?

many thanks if you can help,

jim

 

[Robert Kern]

dear all,

i'm an astronomer working with 2d images -- 2d numarrays. i have a
script which basically does some operations on some images, and one of
the first steps is to find a galaxy on an image (at, say, a known x,y
coord), and create a sub-image by slicing out part of the larger array
to create a more managable smaller one (e.g. 50x50 pixels from a
1000x1000 array).

i was running this fine under python 2.3.4, but due to the slowness of
my machine, moved to a speedy 64-bit linux box running version 2.4.2.
arrays were no longer being sliced in the correct way, for example:

B = A[3:7,6:10]

generates different Bs if i run an identical script on the same image
on the different machines.

note: we deal with FITS format images, which are read into numarrays
using the pyfits.py module. does anyone know any obvious reason this
should be happening?

Not without having the data and the machines in front of me. You'll get
better luck asking on numarray's mailing list.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[jepler]

Do you have a simple program that demonstrates the problem?

I have an x86 machine with Python 2.3, and an x86_64 machine with Python 2.4
available. I wrote a simple test program which performs a slice operation,
but it behaves the same on both platforms.

Here's the program:
#------------------------------------------------------------------------
import numarray, sys, os
print "python: ", sys.version_info
print "numarray:", numarray.__version__
print "CPU: ", os.uname()[-1]

n = numarray.arange(10)
o = numarray.outerproduct(n,n)
p = o[3:7,6:10].copy()
q = numarray.outerproduct([3,4,5,6], [6,7,8,9])
print "Success: ", numarray.alltrue(p.ravel() == q.ravel())
#------------------------------------------------------------------------

Here are the two results I gathered:

python: (2, 4, 1, 'final', 0)
numarray: 1.3.3
CPU: x86_64
Success: True

python: (2, 3, 2, 'final', 0)
numarray: 0.9
CPU: i686
Success: 1

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDRsYYJd01MZaTXX0RAq35AJ477QdItft2GAaL+ElmqtbEE8fHQQCgpjpO
Vl4i9wURTugz61Xfo5Puc1w=
=X7lq
-----END PGP SIGNATURE-----

 

[jeg]

thanks, i ran it -- the only difference i got was the numarray version:
1.1.1 on the 686, and 1.3.3 on the 64bit... but i wouldn't have thought
that would make too much difference.

 

[Robert Kern]

I am thinking to upgrade my desktop to 64bit cpu with 16GB memory to handle
large astronomical images and data. I wonder if
1) the latest numarry (besides 2d slicing) has been tested with one or more
64 bit CPU and Linux distributions
Certainly.

2) with 64 bit address space, will numarray be able to handle larger arrays
and matrices (many 3d-arrays 100x100x100 and matrices 50k x 50k)?
(with 32 bit CPU I was limited by the memory.)

I would appreciate knowing about your experience.

You'll probably want to ask on the appropriate mailing list[1], but
since you ask here, I will say that there are limitations that prevent
numarray from fully utilizing 64-bit systems. numarray uses the Python
buffer interface which is addressed by 32-bit integers even on 64-bit
platforms[2]. There has been some work on numarray's replacement,
scipy_core, to address this deficiency, but more work needs to be done
and more volunteers with 64-bit systems are needed. 100x100x100 arrays
should work with numarray; 50000x50000 perhaps not. You could always try
and let us know.

[1] http://lists.sourceforge.net/lists/listinfo/numpy-discussion
[2] http://permalink.gmane.org/gmane.comp.python.numeric.general/2690

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter