Numarray: Using sum() within functions

[Jim Cser]

Hello-
I have a function to generate a multi-dimensional array, which then
gets summed over one axis. The problem is that the dimensions
are large, and I run out of memory when I create the entire array,
so I'm trying to do the sum *within* the function.

Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
functions f1(x,y,z,t), f2(y,z,t); want to calculate f1*f2 and
sum over t to get out[x,y,z].

With loops, I could do it like--
out = zeros((numX,numY,numZ))
for x in range(numX):
for y in range(numY):
for z in range(numZ):
for t in range(numT):
tempval = f1(x,y,z,t) * f2(y,z,t)
out[x,y,z] = out[x,y,z] + tempval

With numarray, if I had enough memory, I could just do--
temp1 = fromfunction(f1,(numX,numY,numZ,numT))
temp2 = resize(fromfunction(f2,(numY,numZ,numT)),(numX,numY,numZ,numT))
out = sum(temp1 * temp2, axis = 3)

Instead, I'm trying to do something like--
def f3(x,y,z):
for t in range(numT):
tempval = f1(x,y,z,t) * f2(y,z,t)

outval = sum(tempval,axis = 3)
return outval

out = fromfunction(f3,(numX,numY,numZ))

I've been trying various slicing and indexing, but I can't seem to
get that *extra* dimension within the 3-D function. I've scoured the
documentation and list archives, but haven't found exactly what I need.
Any suggestions? Am I stuck with generating the entire 4-D array?

Thanks in advance,
Jim Cser

 

[Jeff Epler]

Instead, I'm trying to do something like--
def f3(x,y,z):
for t in range(numT):
tempval = f1(x,y,z,t) * f2(y,z,t)

outval = sum(tempval,axis = 3)
return outval

Here, "tempval" takes on a series of values during the for loop, then
"sum" is used on value from the final iteration of the loop.

Perhaps you want something like
def f3(x, y, z):
temps = []
for t in range(numT):
temps.append(f1(...) * f(...))
return sum(temps, axis=3)

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBH3LzJd01MZaTXX0RAgTnAJwMvn8IgLCswgQXuRnonHZ1DHiObACfZSFv
adV/zmpFDyZiy5S+Nuf2Syo=
=znkn
-----END PGP SIGNATURE-----

 

[Jim Cser]

Instead, I'm trying to do something like--
def f3(x,y,z):
for t in range(numT):
tempval = f1(x,y,z,t) * f2(y,z,t)

outval = sum(tempval,axis = 3)
return outval

Here, "tempval" takes on a series of values during the for loop, then
"sum" is used on value from the final iteration of the loop.

Perhaps you want something like
def f3(x, y, z):
temps = []
for t in range(numT):
temps.append(f1(...) * f(...))
return sum(temps, axis=3)

Jeff

Thanks, that works, although someone gave me one that is faster:

def f3(x,y,z):
tempval = 0.*x
for t in range(numT):
tempval += f1(x,y,z,t) * f2(y,z,t)
return tempval

In either case, unfortunately, looping over t is extremely slow.
Ideally, there would be a way to use fromfunction() with slices as
arguments.

-Jim

 

[Christopher T King]

I have a function to generate a multi-dimensional array, which then
gets summed over one axis. The problem is that the dimensions
are large, and I run out of memory when I create the entire array,
so I'm trying to do the sum *within* the function.

Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
functions f1(x,y,z,t), f2(y,z,t); want to calculate f1*f2 and
sum over t to get out[x,y,z].

Would something like the following work?

def f3(x, y, z, t=arange(numT)):
return sum(f1(x,y,z,t)*f2(y,z,t))

This assumes that f1() and f2() can operate on array objects. If not, the
following (much slower) method should work:

def f3(x, y, z):
return sum(fromfunction(lambda t: f1(x,y,z,t)*f2(y,z,t),(numT,)))

To make the output matrix, you could use either this:

out = fromfunction(f3, (numX, numY, numZ))

which will save memory, but will be slow, or (assuming f1 and f2 support
arrays):

out = f3(arange(numX), arange(numY), arange(numZ))

which will eat lots of memory (more that you say you have, if you use the
first f3()), but will be very fast.

Hope this helps.

 

[duke0uke]

I have a function to generate a multi-dimensional array, which then
gets summed over one axis. The problem is that the dimensions
are large, and I run out of memory when I create the entire array,
so I'm trying to do the sum *within* the function.

Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
functions f1(x,y,z,t), f2(y,z,t); want to calculate f1*f2 and
sum over t to get out[x,y,z].

Cobbling together a number of suggestions, what finally worked was--

def f3(x, y, z, t_range=arange(numT)):
tempval = 0.* x
for t in t_range:
tempval += f1(x,y,z,t) + f2(y,z,t)
return tempval

out = fromfunction(f3,(numX,numY,numZ,1))


I couldn't quite get sum() to work inside the function, but this
is definitely good enough for now. Thanks to all for your help.
-Jim Cser

 

[Jim Cser]

I have a function to generate a multi-dimensional array, which then
gets summed over one axis. The problem is that the dimensions
are large, and I run out of memory when I create the entire array,
so I'm trying to do the sum *within* the function.

Example-- variables x,y,z,t; dimensions numX, numY, numZ, numT;
functions f1(x,y,z,t), f2(y,z,t); want to calculate f1*f2 and
sum over t to get out[x,y,z].

Cobbling together a number of suggestions, what finally worked was--

def f3(x, y, z, t_range=arange(numT)):
tempval = 0.* x
for t in t_range:
tempval += f1(x,y,z,t) + f2(y,z,t)
return tempval

out = fromfunction(f3,(numX,numY,numZ,1))


I couldn't quite get sum() to work inside the function, but this
is definitely good enough for now. Thanks to all for your help.
-Jim Cser

[wrong identity on last post, sorry]