quickly looping over a 2D array?

Discussion in 'Python' started by Martin, Jul 27, 2009.

1. MartinGuest

Hi,

I am new to python and I was wondering if there was a way to speed up
the way I index 2D arrays when I need to check two arrays
simultaneously? My current implementations is (using numpy) something
like the following...

for i in range(numrows):
for j in range(numcols):

if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
j] * some_other_value
array_1[i, j] = some_new_value

Many Thanks,

Martin

Martin, Jul 27, 2009

2. Peter OttenGuest

Martin wrote:

> I am new to python and I was wondering if there was a way to speed up
> the way I index 2D arrays when I need to check two arrays
> simultaneously? My current implementations is (using numpy) something
> like the following...
>
> for i in range(numrows):
> for j in range(numcols):
>
> if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> j] * some_other_value
> array_1[i, j] = some_new_value

array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
some_new_value

maybe?

Peter Otten, Jul 27, 2009

3. MartinGuest

On Jul 27, 12:42 pm, Peter Otten <> wrote:
> Martin wrote:
> > I am new to python and I was wondering if there was a way to speed up
> > the way I index 2D arrays when I need to check two arrays
> > simultaneously? My current implementations is (using numpy) something
> > like the following...

>
> > for i in range(numrows):
> > for j in range(numcols):

>
> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> > j] * some_other_value
> > array_1[i, j] = some_new_value

>
> array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
> some_new_value
>
> maybe?

So I tried...

band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

which led to

ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

so not sure that works?

Martin, Jul 27, 2009
4. Peter OttenGuest

Martin wrote:

> On Jul 27, 12:42 pm, Peter Otten <> wrote:
>> Martin wrote:
>> > I am new to python and I was wondering if there was a way to speed up
>> > the way I index 2D arrays when I need to check two arrays
>> > simultaneously? My current implementations is (using numpy) something
>> > like the following...

>>
>> > for i in range(numrows):
>> > for j in range(numcols):

>>
>> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
>> > j] * some_other_value
>> > array_1[i, j] = some_new_value

>>
>> array_1[(array_1 == some_value) | (array_2 >= array_1 *
>> some_other_value)] = some_new_value
>>
>> maybe?

>
> So I tried...
>
> band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0
>
> which led to
>
> ValueError: The truth value of an array with more than one element is
> ambiguous. Use a.any() or a.all()
>
> so not sure that works?

Copy and paste -- or replace "or" with "|".

Peter Otten, Jul 27, 2009
5. MartinGuest

On Jul 27, 1:46 pm, Peter Otten <> wrote:
> Martin wrote:
> > On Jul 27, 12:42 pm, Peter Otten <> wrote:
> >> Martin wrote:
> >> > I am new to python and I was wondering if there was a way to speed up
> >> > the way I index 2D arrays when I need to check two arrays
> >> > simultaneously? My current implementations is (using numpy) something
> >> > like the following...

>
> >> > for i in range(numrows):
> >> > for j in range(numcols):

>
> >> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> >> > j] * some_other_value
> >> > array_1[i, j] = some_new_value

>
> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
> >> some_other_value)] = some_new_value

>
> >> maybe?

>
> > So I tried...

>
> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>
> > which led to

>
> > ValueError: The truth value of an array with more than one element is
> > ambiguous. Use a.any() or a.all()

>
> > so not sure that works?

>
> Copy and paste -- or replace "or" with "|".

apologies - I mistook that for a type for "or"

I now get the following error...

ValueError: shape mismatch: objects cannot be broadcast to a single
shape

Martin, Jul 27, 2009
6. Peter OttenGuest

Martin wrote:

> On Jul 27, 1:46 pm, Peter Otten <> wrote:
>> Martin wrote:
>> > On Jul 27, 12:42 pm, Peter Otten <> wrote:
>> >> Martin wrote:
>> >> > I am new to python and I was wondering if there was a way to speed
>> >> > up the way I index 2D arrays when I need to check two arrays
>> >> > simultaneously? My current implementations is (using numpy)
>> >> > something like the following...

>>
>> >> > for i in range(numrows):
>> >> > for j in range(numcols):

>>
>> >> > if array_1[i, j] == some_value or array_2[i, j] >=
>> >> > array_1[i,
>> >> > j] * some_other_value
>> >> > array_1[i, j] = some_new_value

>>
>> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
>> >> some_other_value)] = some_new_value

>>
>> >> maybe?

>>
>> > So I tried...

>>
>> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>>
>> > which led to

>>
>> > ValueError: The truth value of an array with more than one element is
>> > ambiguous. Use a.any() or a.all()

>>
>> > so not sure that works?

>>
>> Copy and paste -- or replace "or" with "|".

>
> apologies - I mistook that for a type for "or"
>
> I now get the following error...
>
> ValueError: shape mismatch: objects cannot be broadcast to a single
> shape

It seems array_1 and array_2 have a -- dada! -- different shape.
Assuming array_1 is the smaller one:

numrows, numcols = array_1.shape
array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
some_other_value)] = some_new_value

There may be other options, but I'm not a numpy expert.

Peter

Peter Otten, Jul 27, 2009
7. MartinGuest

On Jul 27, 2:17 pm, Peter Otten <> wrote:
> Martin wrote:
> > On Jul 27, 1:46 pm, Peter Otten <> wrote:
> >> Martin wrote:
> >> > On Jul 27, 12:42 pm, Peter Otten <> wrote:
> >> >> Martin wrote:
> >> >> > I am new to python and I was wondering if there was a way to speed
> >> >> > up the way I index 2D arrays when I need to check two arrays
> >> >> > simultaneously? My current implementations is (using numpy)
> >> >> > something like the following...

>
> >> >> > for i in range(numrows):
> >> >> > for j in range(numcols):

>
> >> >> > if array_1[i, j] == some_value or array_2[i, j] >=
> >> >> > array_1[i,
> >> >> > j] * some_other_value
> >> >> > array_1[i, j] = some_new_value

>
> >> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
> >> >> some_other_value)] = some_new_value

>
> >> >> maybe?

>
> >> > So I tried...

>
> >> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>
> >> > which led to

>
> >> > ValueError: The truth value of an array with more than one element is
> >> > ambiguous. Use a.any() or a.all()

>
> >> > so not sure that works?

>
> >> Copy and paste -- or replace "or" with "|".

>
> > apologies - I mistook that for a type for "or"

>
> > I now get the following error...

>
> > ValueError: shape mismatch: objects cannot be broadcast to a single
> > shape

>
> It seems array_1 and array_2 have a -- dada! -- different shape.
> Assuming array_1 is the smaller one:
>
> numrows, numcols = array_1.shape
> array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
> some_other_value)] = some_new_value
>
> There may be other options, but I'm not a numpy expert.
>
> Peter

My mistake - the incorrect size in the arrays was my error.

The statement works now, but it doesn't give the same results as my
original logic, strangely!?

in my logic:

data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

for i in range(numrows):
for j in range(numcols):
if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
data[i,j] = 0
else:
data[i,j] = 1

to do the same with what you suggested...

data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
data[(band3 == 255) | (band6 >= band3 * factor)] = 0

Thanks

Martin, Jul 27, 2009
8. Peter OttenGuest

Martin wrote:

> The statement works now, but it doesn't give the same results as my
> original logic, strangely!?
>
> in my logic:
>
> data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')
>
> for i in range(numrows):
> for j in range(numcols):
> if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
> data[i,j] = 0
> else:
> data[i,j] = 1
>
> to do the same with what you suggested...
>
> data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
> data[(band3 == 255) | (band6 >= band3 * factor)] = 0

Did you swap band3 and band6? If that's the case, it is an error you should
be able to find yourself.

Please be a bit more careful.

Also, it is good practice to write a few test cases where you have
precalculated the result. How would you otherwise know which version is
correct? Writing a third one to break the tie?

Peter

Peter Otten, Jul 27, 2009
9. MartinGuest

On Jul 27, 4:12 pm, Peter Otten <> wrote:
> Martin wrote:
> > The statement works now, but it doesn't give the same results as my
> > original logic, strangely!?

>
> > in my logic:

>
> > data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

>
> > for i in range(numrows):
> > for j in range(numcols):
> > if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
> > data[i,j] = 0
> > else:
> > data[i,j] = 1

>
> > to do the same with what you suggested...

>
> > data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
> > data[(band3 == 255) | (band6 >= band3 * factor)] = 0

>
> Did you swap band3 and band6? If that's the case, it is an error you should
> be able to find yourself.
>
> Please be a bit more careful.
>
> Also, it is good practice to write a few test cases where you have
> precalculated the result. How would you otherwise know which version is
> correct? Writing a third one to break the tie?
>
> Peter

apologies... it was the way I typed it up, I wasn't copying and
pasting from my text editor!
Thanks Peter.

Martin, Jul 27, 2009
10. Robert KernGuest

On 2009-07-27 06:24, Martin wrote:
> Hi,
>
> I am new to python and I was wondering if there was a way to speed up
> the way I index 2D arrays when I need to check two arrays
> simultaneously? My current implementations is (using numpy) something
> like the following...
>
> for i in range(numrows):
> for j in range(numcols):
>
> if array_1[i, j] == some_value or array_2[i, j]>= array_1[i,
> j] * some_other_value
> array_1[i, j] = some_new_value

Peter has given you good answers, but you will probably want to ask future numpy
questions on the numpy mailing list.

http://www.scipy.org/Mailing_Lists

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma