List of lists surprising behaviour

Discussion in 'Python' started by candide, Jun 17, 2010.

  1. candide

    candide Guest

    Let's the following code :

    >>> t=[[0]*2]*3
    >>> t

    [[0, 0], [0, 0], [0, 0]]
    >>> t[0][0]=1
    >>> t

    [[1, 0], [1, 0], [1, 0]]

    Rather surprising, isn't it ? So I suppose all the subarrays reférence
    the same array :

    >>> id(t[0]), id(t[1]), id(t[2])

    (3077445996L, 3077445996L, 3077445996L)
    >>>



    So what is the right way to initialize to 0 a 2D array ? Is that way
    correct :


    >>> t=[[0 for _ in range(2)] for _ in range(3)]


    It seems there is no more trouble now :

    >>> t

    [[0, 0], [0, 0], [0, 0]]
    >>> t[0][0]=1
    >>> t

    [[1, 0], [0, 0], [0, 0]]
    >>>


    Correct ?
     
    candide, Jun 17, 2010
    #1
    1. Advertising

  2. candide

    Lie Ryan Guest

    On 06/17/10 20:21, candide wrote:
    > Let's the following code :
    >
    >>>> t=[[0]*2]*3
    >>>> t

    > [[0, 0], [0, 0], [0, 0]]
    >>>> t[0][0]=1
    >>>> t

    > [[1, 0], [1, 0], [1, 0]]
    >
    > Rather surprising, isn't it ? So I suppose all the subarrays reférence
    > the same array :
    >
    >>>> id(t[0]), id(t[1]), id(t[2])

    > (3077445996L, 3077445996L, 3077445996L)
    >>>>


    Yep, you're right. They share the same subarray if you uses
    multiplication to build the array.

    > So what is the right way to initialize to 0 a 2D array ? Is that way
    > correct :
    >>>> t=[[0 for _ in range(2)] for _ in range(3)]


    Right again. That's the way to go. Although if the elements are
    immutable, you can create the innermost array by multiplication:

    t=[[0]*2 for _ in range(3)]
     
    Lie Ryan, Jun 17, 2010
    #2
    1. Advertising

  3. candide

    Matteo Landi Guest

    Yes you are. List comprehension makes you create list of lists without
    reference-sharing. You should also find a recipe about that on the
    python cookbook.

    On Thu, Jun 17, 2010 at 12:21 PM, candide <> wrote:
    > Let's the following code :
    >
    >>>> t=[[0]*2]*3
    >>>> t

    > [[0, 0], [0, 0], [0, 0]]
    >>>> t[0][0]=1
    >>>> t

    > [[1, 0], [1, 0], [1, 0]]
    >
    > Rather surprising, isn't it ? So I suppose all the subarrays reférence the
    > same array :
    >
    >>>> id(t[0]), id(t[1]), id(t[2])

    > (3077445996L, 3077445996L, 3077445996L)
    >>>>

    >
    >
    > So what is the right way to initialize to 0 a 2D array ? Is that way correct
    >  :
    >
    >
    >>>> t=[[0 for _ in range(2)] for _ in range(3)]

    >
    > It seems there is no more trouble now :
    >
    >>>> t

    > [[0, 0], [0, 0], [0, 0]]
    >>>> t[0][0]=1
    >>>> t

    > [[1, 0], [0, 0], [0, 0]]
    >>>>

    >
    > Correct ?
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >




    --
    Matteo Landi
    http://www.matteolandi.net/
     
    Matteo Landi, Jun 17, 2010
    #3
  4. candide

    Boris Borcic Guest

    candide wrote:
    >
    > So what is the right way to initialize to 0 a 2D array ? Is that way
    > correct :
    >
    >
    > >>> t=[[0 for _ in range(2)] for _ in range(3)]


    That's overkill :) You can skip the inner loop by using a list display, eg

    t=[[0,0] for _ in range(3)]

    >
    > It seems there is no more trouble now :
    >
    > >>> t

    > [[0, 0], [0, 0], [0, 0]]
    > >>> t[0][0]=1
    > >>> t

    > [[1, 0], [0, 0], [0, 0]]
    > >>>

    >
    > Correct ?
     
    Boris Borcic, Jun 17, 2010
    #4
  5. candide <> writes:

    > Let's the following code :
    >
    >>>> t=[[0]*2]*3
    >>>> t

    > [[0, 0], [0, 0], [0, 0]]
    >>>> t[0][0]=1
    >>>> t

    > [[1, 0], [1, 0], [1, 0]]
    >
    > Rather surprising, isn't it ?


    Not at all, actually.

    I'd be surprised if the multiplication operator was aware of object
    constructors. Even arrays are "objects" in Python. Should the
    multiplication operator know how to instantiate three arrays from a
    single array instance? What about an instance of a user-defined class?

    > So I suppose all the subarrays reférence
    > the same array :
    >
    >>>> id(t[0]), id(t[1]), id(t[2])

    > (3077445996L, 3077445996L, 3077445996L)
    >>>>

    >


    As they should.

    >
    > So what is the right way to initialize to 0 a 2D array ? Is that way
    > correct :
    >
    >
    >>>> t=[[0 for _ in range(2)] for _ in range(3)]

    >
    > It seems there is no more trouble now :
    >
    >>>> t

    > [[0, 0], [0, 0], [0, 0]]
    >>>> t[0][0]=1
    >>>> t

    > [[1, 0], [0, 0], [0, 0]]
    >>>>

    >
    > Correct ?


    >>> 2d_zero_vector = lambda len: [[0, 0] for _ in range(len)]
    >>> t = 2d_zero_vector(3)
    >>> print t

    [[0, 0], [0, 0], [0, 0]]
    >>> t[0][0] = 1
    >>> print t

    [[1, 0], [0, 0], [0, 0], [0, 0]]

    (Of course, if you're doing matrix math you'll probably want to work
    with numpy which has a function for doing just this)
     
    J Kenneth King, Jun 17, 2010
    #5
  6. candide

    bart.c Guest

    "J Kenneth King" <> wrote in message
    news:...
    > candide <> writes:
    >
    >> Let's the following code :
    >>
    >>>>> t=[[0]*2]*3
    >>>>> t

    >> [[0, 0], [0, 0], [0, 0]]
    >>>>> t[0][0]=1
    >>>>> t

    >> [[1, 0], [1, 0], [1, 0]]
    >>
    >> Rather surprising, isn't it ?

    >
    > Not at all, actually.


    The code is clearly trying to set only t[0][0] to 1, not t[1][0] and t[2][0]
    as well.

    This behaviour is quite scary actually, especially when t[0]=42 *does* work
    as expected, while t[0][0]=42 is apparently duplicated. It appears
    inconsistent.

    > I'd be surprised if the multiplication operator was aware of object
    > constructors. Even arrays are "objects" in Python. Should the
    > multiplication operator know how to instantiate three arrays from a
    > single array instance? What about an instance of a user-defined class?


    Multiplication operators shouldn't need to be directly aware of any such
    thing; it should just request that an object be duplicated without worrying
    about how it's done.

    I don't know how Python does things, but an object should either specify a
    special way of duplicating itself, or lend itself to some standard way of
    doing so. (So for a list, it's just a question of copying the data in the
    list, then recursively duplicating each new element..)

    --
    Bartc
     
    bart.c, Jun 18, 2010
    #6
  7. On Thu, Jun 17, 2010 at 4:20 PM, bart.c <> wrote:
    >
    > "J Kenneth King" <> wrote in message
    > news:...
    >>
    >> candide <> writes:
    >>
    >>> Let's the following code :
    >>>
    >>>>>> t=[[0]*2]*3
    >>>>>> t
    >>>
    >>> [[0, 0], [0, 0], [0, 0]]
    >>>>>>
    >>>>>> t[0][0]=1
    >>>>>> t
    >>>
    >>> [[1, 0], [1, 0], [1, 0]]
    >>>
    >>> Rather surprising, isn't it ?

    >>
    >> Not at all, actually.

    >
    > The code is clearly trying to set only t[0][0] to 1, not t[1][0] and t[2][0]
    > as well.
    >
    > This behaviour is quite scary actually, especially when t[0]=42 *does* work
    > as expected, while t[0][0]=42 is apparently duplicated. It appears
    > inconsistent.
    >
    >> I'd be surprised if the multiplication operator was aware of object
    >> constructors.  Even arrays are "objects" in Python.  Should the
    >> multiplication operator know how to instantiate three arrays from a
    >> single array instance?  What about an instance of a user-defined class?

    >
    > Multiplication operators shouldn't need to be directly aware of any such
    > thing; it should just request that an object be duplicated without worrying
    > about how it's done.
    >
    > I don't know how Python does things, but an object should either specify a
    > special way of duplicating itself, or lend itself to some standard way of
    > doing so. (So for a list, it's just a question of copying the data in the
    > list, then recursively duplicating each new element..)
    >
    > --
    > Bartc


    It's the recursively duplicating each element that's the problem. How
    do you know when to stop?
     
    Benjamin Kaplan, Jun 18, 2010
    #7
  8. candide

    rantingrick Guest

    On Jun 17, 6:44 pm, Benjamin Kaplan <> wrote:

    > It's the recursively duplicating each element that's the problem. How
    > do you know when to stop?



    Thats easy, stack overflow! ;-)
     
    rantingrick, Jun 18, 2010
    #8
  9. candide

    Lie Ryan Guest

    On 06/18/10 09:20, bart.c wrote:
    >
    > "J Kenneth King" <> wrote in message
    > news:...
    >> candide <> writes:
    >>
    >>> Let's the following code :
    >>>
    >>>>>> t=[[0]*2]*3
    >>>>>> t
    >>> [[0, 0], [0, 0], [0, 0]]
    >>>>>> t[0][0]=1
    >>>>>> t
    >>> [[1, 0], [1, 0], [1, 0]]
    >>>
    >>> Rather surprising, isn't it ?

    >>
    >> Not at all, actually.

    >
    > The code is clearly trying to set only t[0][0] to 1, not t[1][0] and
    > t[2][0]
    > as well.
    >
    > This behaviour is quite scary actually, especially when t[0]=42 *does* work
    > as expected, while t[0][0]=42 is apparently duplicated. It appears
    > inconsistent.


    I agree, the behavior is often quite inconvenient, but I disagree that
    it is inconsistent. List multiplication behavior is consistent with the
    tenet: "objects are never copied unless explicitly requested"

    Peeking further:
    t = [[0] * 2] * 3
    print id(t[0]) == id(t[1]) # True
    print id(t[0][0]) == id(t[1][0]) # True

    so, it is consistent (though it is quite inconvenient).

    >> I'd be surprised if the multiplication operator was aware of object
    >> constructors. Even arrays are "objects" in Python. Should the
    >> multiplication operator know how to instantiate three arrays from a
    >> single array instance? What about an instance of a user-defined class?

    >
    > Multiplication operators shouldn't need to be directly aware of any such
    > thing; it should just request that an object be duplicated without
    > worrying about how it's done.
    >
    > I don't know how Python does things, but an object should either specify
    > a special way of duplicating itself, or lend itself to some standard way
    > of doing so. (So for a list, it's just a question of copying the data in
    > the list, then recursively duplicating each new element..)


    That is inconsistent with the tenet. Moreover, the implicit copying
    makes it quite difficult to reason about the program's behavior. How
    would you propose this list should be copied:

    class O(object):
    def __init__(self):
    self.attr = 0
    def add(self):
    self.attr += 1

    b = [[O()] * 2] * 3
     
    Lie Ryan, Jun 18, 2010
    #9
  10. On Fri, 18 Jun 2010 00:20:30 +0100, bart.c wrote:

    > The code is clearly trying to set only t[0][0] to 1, not t[1][0] and
    > t[2][0] as well.


    Trying to guess the motivation of the person writing code is tricky, but
    in this case, that's a reasonable assumption. I can't think of any reason
    why somebody would explicitly *want* that behaviour:

    # Does it make sense to talk of anonymous aliases?
    list_of_aliases = [[0]*2]*3
    list_of_aliases[0][0] = 1
    assert list_of_aliases[0][1] == 1

    so it is a safe guess that anyone writing [[0]*2]*3 has probably made a
    mistake.

    However, I've certainly done something like this:

    a = [0]*2
    my_instance.items = a
    a[0] = 1
    assert my_instance.items[0] = 1

    This is the same fundamental behaviour with the same cause: Python does
    not copy objects unless you explicitly tell it to.

    I cheerfully accept that the behaviour of [[0]*2]*3 is a Gotcha, but it
    follows logically from Python's object model and assignment rules. If you
    are surprised by it, it just goes to show that your understanding of
    Python has at least one hole in it.


    > This behaviour is quite scary actually, especially when t[0]=42 *does*
    > work as expected, while t[0][0]=42 is apparently duplicated. It appears
    > inconsistent.


    Emphasis on the word "appears". It actually displays a deep consistency
    with the language fundamentals.

    If you're ever interviewing somebody for a position as Python developer,
    this is a quick test to distinguish those who know the language from
    those who know the language *well*.


    >> I'd be surprised if the multiplication operator was aware of object
    >> constructors. Even arrays are "objects" in Python. Should the
    >> multiplication operator know how to instantiate three arrays from a
    >> single array instance? What about an instance of a user-defined class?

    >
    > Multiplication operators shouldn't need to be directly aware of any such
    > thing; it should just request that an object be duplicated without
    > worrying about how it's done.


    The multiplication operator is not a duplicator (copier). It is a
    *repetition* operator: repeat the object N times, not make N copies.


    > I don't know how Python does things,


    Well there you go :)


    > but an object should either specify
    > a special way of duplicating itself, or lend itself to some standard way
    > of doing so.


    import copy
    copy.copy(obj)

    Dicts have a copy() method as a shortcut, and for lists you can use
    slicing:

    L = [1,2,3]
    Lcopy = L[:]


    > (So for a list, it's just a question of copying the data in
    > the list, then recursively duplicating each new element..)


    There's nothing "just" about that. Consider:

    L = [1, 2]
    L.append(L)

    How would you copy that?

    The correct answer is:

    x = copy.deepcopy(L)



    --
    Steven
     
    Steven D'Aprano, Jun 18, 2010
    #10
  11. candide

    bart.c Guest

    Benjamin Kaplan wrote:
    > On Thu, Jun 17, 2010 at 4:20 PM, bart.c <> wrote:


    >> I don't know how Python does things, but an object should either
    >> specify a special way of duplicating itself, or lend itself to some
    >> standard way of doing so. (So for a list, it's just a question of
    >> copying the data in the list, then recursively duplicating each new
    >> element..)


    > It's the recursively duplicating each element that's the problem. How
    > do you know when to stop?


    When you reach a primitive object (one not comprising other objects). (I
    don't know if Python allows circular references, but that would give
    problems anyway: how would you even print out such a list?)

    --
    Bartc
     
    bart.c, Jun 18, 2010
    #11
  12. candide

    Lie Ryan Guest

    On 06/18/10 20:00, bart.c wrote:
    > (I
    > don't know if Python allows circular references, but that would give
    > problems anyway: how would you even print out such a list?)



    Python uses ellipsis to indicate recursive list:

    >>> a = [1, 2, 3]
    >>> a.append(a)
    >>> a

    [1, 2, 3, [...]]
     
    Lie Ryan, Jun 18, 2010
    #12
  13. candide

    bart.c Guest

    Lie Ryan wrote:
    > On 06/18/10 20:00, bart.c wrote:
    >> (I
    >> don't know if Python allows circular references, but that would give
    >> problems anyway: how would you even print out such a list?)

    >
    >
    > Python uses ellipsis to indicate recursive list:
    >
    >>>> a = [1, 2, 3]
    >>>> a.append(a)
    >>>> a

    > [1, 2, 3, [...]]


    Ok, perhaps whatever logic print uses to know when to stop and show "...",
    can also be used by copy handlers...

    (Although I have an issue with the way that that append works. I tried it in
    another, simpler language (which always does deep copies):

    L:=(1,2,3)
    L append:= L
    print L

    output: (1,2,3,(1,2,3))

    which is exactly what I'd expect, and not (1,2,3,(1,2,3,(1,2,3,...))) )

    --
    bartc
     
    bart.c, Jun 18, 2010
    #13
  14. On Fri, 18 Jun 2010 12:07:38 +0100, bart.c wrote:

    > (Although I have an issue with the way that that append works. I tried
    > it in another, simpler language (which always does deep copies):
    >
    > L:=(1,2,3)
    > L append:= L
    > print L
    >
    > output: (1,2,3,(1,2,3))
    >
    > which is exactly what I'd expect,
    > and not (1,2,3,(1,2,3,(1,2,3,...))) )


    I find that behaviour a big surprise. You asked to append the list L, not
    a copy of the list L. So why is this "simpler" language making a copy
    without being asked?

    If you asked for:

    L:=(1,2,3)
    M:=(0,1)
    M append:= L

    does it also append a copy of L instead of L? If so, how do you append
    the original rather than wastefully making a copy? If L is huge, making a
    copy before appending will be slow, and potentially fail.



    --
    Steven
     
    Steven D'Aprano, Jun 18, 2010
    #14
  15. candide

    bart.c Guest

    "Steven D'Aprano" <> wrote in message
    news:4c1b8ac6$0$14148$...
    > On Fri, 18 Jun 2010 12:07:38 +0100, bart.c wrote:
    >
    >> (Although I have an issue with the way that that append works. I tried
    >> it in another, simpler language (which always does deep copies):
    >>
    >> L:=(1,2,3)
    >> L append:= L
    >> print L
    >>
    >> output: (1,2,3,(1,2,3))
    >>
    >> which is exactly what I'd expect,
    >> and not (1,2,3,(1,2,3,(1,2,3,...))) )

    >
    > I find that behaviour a big surprise. You asked to append the list L, not
    > a copy of the list L. So why is this "simpler" language making a copy
    > without being asked?
    >
    > If you asked for:
    >
    > L:=(1,2,3)
    > M:=(0,1)
    > M append:= L
    >
    > does it also append a copy of L instead of L?


    It make a copy.

    > If so, how do you append
    > the original rather than wastefully making a copy?


    I don't think it can, without perhaps doing something with explicit
    pointers.

    > If L is huge, making a
    > copy before appending will be slow, and potentially fail.


    I don't know whether L append:=L requires 3 times the space of L, or 2
    times, during the operation. But it should be doable using just twice the
    space.

    I suppose there are pros and cons to both approaches; copying all the time
    at least avoids some of the odd effects and inconsistencies you get using
    Python:

    a1=[1,2,3]
    a1.append(a1)

    a2=[1,2,3]
    b=[1,2,3]
    a2.append(b)

    a3=[1,2,3]
    a3.append([1,2,3])

    print ("a1 = ",a1)
    print ("a2 = ",a2)
    print ("a3 = ",a3)

    Here, a1 ends up with a different result from a2, a3, even though the same
    value is being appended to the same list of numbers in each case. And it
    might sometimes bite you in the arse as the OP demonstrated:

    L=[1,2,3]
    M=[0,1]
    M.append(L)

    print (M) # output: [0, 1, [1, 2, 3]]

    L[1]=31416

    print (M) # output: [0, 1, [1, 31416, 3]], yikes!

    --
    Bartc
     
    bart.c, Jun 18, 2010
    #15
  16. On 6/18/10 8:40 AM, bart.c wrote:
    > I suppose there are pros and cons to both approaches; copying all the time
    > at least avoids some of the odd effects and inconsistencies you get using
    > Python:


    What inconsistencies? All your examples are perfectly consistent. Its
    just consistent to different ideals. Python never copies implicitly; and
    every object you create has a concrete identity in and of itself,
    utterly separate from its potential equality.

    > a1=[1,2,3]
    > a1.append(a1)


    The "a1" object is a distinct object unto itself; you are appending said
    distinct object onto the end of itself. Entirely doable, even if you
    don't usually want to. Recursive, but doable if that's what you want. If
    you wished to append a copy, you must-- as always, consistently--
    explicitly copy it.

    I.e.,

    a1.append(a1[:])

    > a2=[1,2,3]
    > b=[1,2,3]
    > a2.append(b)


    a2 is a distinct object from b; that the two objects are equal means
    nothing. So you're appending an object to the end of a2 which happens to
    be equal to it.

    > a3=[1,2,3]
    > a3.append([1,2,3])


    This is just another way of writing the previous example; that in one
    you are naming the object [1,2,3] and in the other you are not, doesn't
    mean anything. A named vs unnamed object in Python behaves exactly the same.

    > print ("a1 = ",a1)
    > print ("a2 = ",a2)
    > print ("a3 = ",a3)
    >
    > Here, a1 ends up with a different result from a2, a3, even though the same
    > value is being appended to the same list of numbers in each case.


    There's the rub: the VALUE is not being appended. The *actual object* is.

    Consider:

    >>> print a1 is a2

    False

    > And it
    > might sometimes bite you in the arse as the OP demonstrated:
    >
    > L=[1,2,3]
    > M=[0,1]
    > M.append(L)
    >
    > print (M) # output: [0, 1, [1, 2, 3]]
    >
    > L[1]=31416
    >
    > print (M) # output: [0, 1, [1, 31416, 3]], yikes!


    That might bite you on your arse if you think Python implicitly copies;
    but since the rule is very simple -- it /never/ implicitly copies -- and
    that it objects are discrete entities and not just names of certain
    values, it more likely then not will be beneficial to you quite often
    down the road. You'll -want- that to be how things work. Eventually.
    When you learn more Python.

    --

    Stephen Hansen
    ... Also: Ixokai
    ... Mail: me+list/python (AT) ixokai (DOT) io
    ... Blog: http://meh.ixokai.io/


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.10 (Darwin)

    iQEcBAEBAgAGBQJMG6MqAAoJEKcbwptVWx/ll/YIALfJMaQuxVQGeF0EXGllNGl4
    zAH5akcpA3nFr+X6E2I1Xc9MxN162dIbBuRJ6GrLwej2wyklkgm3snMDCQAUA0QU
    xGEzyIma0Mi6KotydyEkmpgxM6b3vQj8SSlmM6mex+zKgR+HlIXvl/zapyPc9v+8
    Kfc8glJgoFXmiv7W9fbwCu8g/w6KeIiAaaC7JepMwm6IoVsrP888p65Add8eEtKb
    PUQKuU1bSEnysmCVstzUnY4lIsByOW3ygpnnA6++uDeR3AwZ4sLRN0J33UsUiGJM
    kPV6c6LkJ9+jY3dObbnSusVPcdfvOIyUlN8i4nxKb/JddPQ9aTptLO/kupZm8xA=
    =18Pu
    -----END PGP SIGNATURE-----
     
    Stephen Hansen, Jun 18, 2010
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==

    List of lists of lists of lists...

    =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==, May 8, 2006, in forum: Python
    Replies:
    5
    Views:
    424
    =?UTF-8?B?w4FuZ2VsIEd1dGnDqXJyZXogUm9kcsOtZ3Vleg==
    May 15, 2006
  2. Joe P. Cool
    Replies:
    7
    Views:
    287
    Benjamin
    Jun 29, 2008
  3. Frank Schmitt
    Replies:
    3
    Views:
    173
    Frank Schmitt
    Dec 9, 2003
  4. Michele Simionato

    surprising behaviour of global dictionaries

    Michele Simionato, Oct 9, 2012, in forum: Python
    Replies:
    6
    Views:
    188
    Peter Otten
    Oct 9, 2012
  5. Ulrich Eckhardt

    surprising += for lists

    Ulrich Eckhardt, Nov 4, 2012, in forum: Python
    Replies:
    3
    Views:
    201
    Terry Reedy
    Nov 4, 2012
Loading...

Share This Page