quickly looping over a 2D array?

Discussion in 'Python' started by Martin, Jul 27, 2009.

  1. Martin

    Martin Guest

    Hi,

    I am new to python and I was wondering if there was a way to speed up
    the way I index 2D arrays when I need to check two arrays
    simultaneously? My current implementations is (using numpy) something
    like the following...

    for i in range(numrows):
    for j in range(numcols):

    if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
    j] * some_other_value
    array_1[i, j] = some_new_value

    Many Thanks,

    Martin
    Martin, Jul 27, 2009
    #1
    1. Advertising

  2. Martin

    Peter Otten Guest

    Martin wrote:

    > I am new to python and I was wondering if there was a way to speed up
    > the way I index 2D arrays when I need to check two arrays
    > simultaneously? My current implementations is (using numpy) something
    > like the following...
    >
    > for i in range(numrows):
    > for j in range(numcols):
    >
    > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
    > j] * some_other_value
    > array_1[i, j] = some_new_value


    array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
    some_new_value

    maybe?
    Peter Otten, Jul 27, 2009
    #2
    1. Advertising

  3. Martin

    Martin Guest

    On Jul 27, 12:42 pm, Peter Otten <> wrote:
    > Martin wrote:
    > > I am new to python and I was wondering if there was a way to speed up
    > > the way I index 2D arrays when I need to check two arrays
    > > simultaneously? My current implementations is (using numpy) something
    > > like the following...

    >
    > > for i in range(numrows):
    > > for j in range(numcols):

    >
    > > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
    > > j] * some_other_value
    > > array_1[i, j] = some_new_value

    >
    > array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
    > some_new_value
    >
    > maybe?


    So I tried...

    band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

    which led to

    ValueError: The truth value of an array with more than one element is
    ambiguous. Use a.any() or a.all()

    so not sure that works?
    Martin, Jul 27, 2009
    #3
  4. Martin

    Peter Otten Guest

    Martin wrote:

    > On Jul 27, 12:42 pm, Peter Otten <> wrote:
    >> Martin wrote:
    >> > I am new to python and I was wondering if there was a way to speed up
    >> > the way I index 2D arrays when I need to check two arrays
    >> > simultaneously? My current implementations is (using numpy) something
    >> > like the following...

    >>
    >> > for i in range(numrows):
    >> > for j in range(numcols):

    >>
    >> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
    >> > j] * some_other_value
    >> > array_1[i, j] = some_new_value

    >>
    >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
    >> some_other_value)] = some_new_value
    >>
    >> maybe?

    >
    > So I tried...
    >
    > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0
    >
    > which led to
    >
    > ValueError: The truth value of an array with more than one element is
    > ambiguous. Use a.any() or a.all()
    >
    > so not sure that works?


    Copy and paste -- or replace "or" with "|".
    Peter Otten, Jul 27, 2009
    #4
  5. Martin

    Martin Guest

    On Jul 27, 1:46 pm, Peter Otten <> wrote:
    > Martin wrote:
    > > On Jul 27, 12:42 pm, Peter Otten <> wrote:
    > >> Martin wrote:
    > >> > I am new to python and I was wondering if there was a way to speed up
    > >> > the way I index 2D arrays when I need to check two arrays
    > >> > simultaneously? My current implementations is (using numpy) something
    > >> > like the following...

    >
    > >> > for i in range(numrows):
    > >> > for j in range(numcols):

    >
    > >> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
    > >> > j] * some_other_value
    > >> > array_1[i, j] = some_new_value

    >
    > >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
    > >> some_other_value)] = some_new_value

    >
    > >> maybe?

    >
    > > So I tried...

    >
    > > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

    >
    > > which led to

    >
    > > ValueError: The truth value of an array with more than one element is
    > > ambiguous. Use a.any() or a.all()

    >
    > > so not sure that works?

    >
    > Copy and paste -- or replace "or" with "|".


    apologies - I mistook that for a type for "or"

    I now get the following error...

    ValueError: shape mismatch: objects cannot be broadcast to a single
    shape
    Martin, Jul 27, 2009
    #5
  6. Martin

    Peter Otten Guest

    Martin wrote:

    > On Jul 27, 1:46 pm, Peter Otten <> wrote:
    >> Martin wrote:
    >> > On Jul 27, 12:42 pm, Peter Otten <> wrote:
    >> >> Martin wrote:
    >> >> > I am new to python and I was wondering if there was a way to speed
    >> >> > up the way I index 2D arrays when I need to check two arrays
    >> >> > simultaneously? My current implementations is (using numpy)
    >> >> > something like the following...

    >>
    >> >> > for i in range(numrows):
    >> >> > for j in range(numcols):

    >>
    >> >> > if array_1[i, j] == some_value or array_2[i, j] >=
    >> >> > array_1[i,
    >> >> > j] * some_other_value
    >> >> > array_1[i, j] = some_new_value

    >>
    >> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
    >> >> some_other_value)] = some_new_value

    >>
    >> >> maybe?

    >>
    >> > So I tried...

    >>
    >> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

    >>
    >> > which led to

    >>
    >> > ValueError: The truth value of an array with more than one element is
    >> > ambiguous. Use a.any() or a.all()

    >>
    >> > so not sure that works?

    >>
    >> Copy and paste -- or replace "or" with "|".

    >
    > apologies - I mistook that for a type for "or"
    >
    > I now get the following error...
    >
    > ValueError: shape mismatch: objects cannot be broadcast to a single
    > shape


    It seems array_1 and array_2 have a -- dada! -- different shape.
    Assuming array_1 is the smaller one:

    numrows, numcols = array_1.shape
    array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
    some_other_value)] = some_new_value

    There may be other options, but I'm not a numpy expert.

    Peter
    Peter Otten, Jul 27, 2009
    #6
  7. Martin

    Martin Guest

    On Jul 27, 2:17 pm, Peter Otten <> wrote:
    > Martin wrote:
    > > On Jul 27, 1:46 pm, Peter Otten <> wrote:
    > >> Martin wrote:
    > >> > On Jul 27, 12:42 pm, Peter Otten <> wrote:
    > >> >> Martin wrote:
    > >> >> > I am new to python and I was wondering if there was a way to speed
    > >> >> > up the way I index 2D arrays when I need to check two arrays
    > >> >> > simultaneously? My current implementations is (using numpy)
    > >> >> > something like the following...

    >
    > >> >> > for i in range(numrows):
    > >> >> > for j in range(numcols):

    >
    > >> >> > if array_1[i, j] == some_value or array_2[i, j] >=
    > >> >> > array_1[i,
    > >> >> > j] * some_other_value
    > >> >> > array_1[i, j] = some_new_value

    >
    > >> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
    > >> >> some_other_value)] = some_new_value

    >
    > >> >> maybe?

    >
    > >> > So I tried...

    >
    > >> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

    >
    > >> > which led to

    >
    > >> > ValueError: The truth value of an array with more than one element is
    > >> > ambiguous. Use a.any() or a.all()

    >
    > >> > so not sure that works?

    >
    > >> Copy and paste -- or replace "or" with "|".

    >
    > > apologies - I mistook that for a type for "or"

    >
    > > I now get the following error...

    >
    > > ValueError: shape mismatch: objects cannot be broadcast to a single
    > > shape

    >
    > It seems array_1 and array_2 have a -- dada! -- different shape.
    > Assuming array_1 is the smaller one:
    >
    > numrows, numcols = array_1.shape
    > array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
    > some_other_value)] = some_new_value
    >
    > There may be other options, but I'm not a numpy expert.
    >
    > Peter


    My mistake - the incorrect size in the arrays was my error.

    The statement works now, but it doesn't give the same results as my
    original logic, strangely!?

    in my logic:

    data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

    for i in range(numrows):
    for j in range(numcols):
    if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
    data[i,j] = 0
    else:
    data[i,j] = 1

    to do the same with what you suggested...

    data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
    data[(band3 == 255) | (band6 >= band3 * factor)] = 0

    Thanks
    Martin, Jul 27, 2009
    #7
  8. Martin

    Peter Otten Guest

    Martin wrote:

    > The statement works now, but it doesn't give the same results as my
    > original logic, strangely!?
    >
    > in my logic:
    >
    > data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')
    >
    > for i in range(numrows):
    > for j in range(numcols):
    > if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
    > data[i,j] = 0
    > else:
    > data[i,j] = 1
    >
    > to do the same with what you suggested...
    >
    > data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
    > data[(band3 == 255) | (band6 >= band3 * factor)] = 0


    Did you swap band3 and band6? If that's the case, it is an error you should
    be able to find yourself.

    Please be a bit more careful.

    Also, it is good practice to write a few test cases where you have
    precalculated the result. How would you otherwise know which version is
    correct? Writing a third one to break the tie?

    Peter
    Peter Otten, Jul 27, 2009
    #8
  9. Martin

    Martin Guest

    On Jul 27, 4:12 pm, Peter Otten <> wrote:
    > Martin wrote:
    > > The statement works now, but it doesn't give the same results as my
    > > original logic, strangely!?

    >
    > > in my logic:

    >
    > > data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

    >
    > > for i in range(numrows):
    > > for j in range(numcols):
    > > if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
    > > data[i,j] = 0
    > > else:
    > > data[i,j] = 1

    >
    > > to do the same with what you suggested...

    >
    > > data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
    > > data[(band3 == 255) | (band6 >= band3 * factor)] = 0

    >
    > Did you swap band3 and band6? If that's the case, it is an error you should
    > be able to find yourself.
    >
    > Please be a bit more careful.
    >
    > Also, it is good practice to write a few test cases where you have
    > precalculated the result. How would you otherwise know which version is
    > correct? Writing a third one to break the tie?
    >
    > Peter


    apologies... it was the way I typed it up, I wasn't copying and
    pasting from my text editor!
    Thanks Peter.
    Martin, Jul 27, 2009
    #9
  10. Martin

    Robert Kern Guest

    On 2009-07-27 06:24, Martin wrote:
    > Hi,
    >
    > I am new to python and I was wondering if there was a way to speed up
    > the way I index 2D arrays when I need to check two arrays
    > simultaneously? My current implementations is (using numpy) something
    > like the following...
    >
    > for i in range(numrows):
    > for j in range(numcols):
    >
    > if array_1[i, j] == some_value or array_2[i, j]>= array_1[i,
    > j] * some_other_value
    > array_1[i, j] = some_new_value


    Peter has given you good answers, but you will probably want to ask future numpy
    questions on the numpy mailing list.

    http://www.scipy.org/Mailing_Lists

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Jul 27, 2009
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. lazy
    Replies:
    5
    Views:
    641
  2. C C++ C++
    Replies:
    11
    Views:
    5,541
    James Kanze
    Jan 17, 2008
  3. KevinSimonson
    Replies:
    31
    Views:
    1,368
  4. Replies:
    5
    Views:
    261
  5. Replies:
    3
    Views:
    115
    Roy Smith
    Sep 25, 2013
Loading...

Share This Page