a.extend(b) better than a+=b ?

Discussion in 'Python' started by candide, Apr 22, 2010.

  1. candide

    candide Guest

    Suppose a and b are lists.

    What is more efficient in order to extend the list a by appending all
    the items in the list b ?


    I imagine a.extend(b)to be more efficient for only appendinding the
    items from b while a+=b creates a copy of a before appending, right ?
    candide, Apr 22, 2010
    #1
    1. Advertising

  2. * candide:
    > Suppose a and b are lists.
    >
    > What is more efficient in order to extend the list a by appending all
    > the items in the list b ?
    >
    >
    > I imagine a.extend(b)to be more efficient for only appendinding the
    > items from b while a+=b creates a copy of a before appending, right ?


    No.

    But in general, if you're concerned about efficiency, *measure*.

    And keep in mind that measurement results may only be applicable to a given
    Python implementation, for given data, in a given environment...

    In most cases you'll do better by focusing on algorithmic efficiency rather than
    low level operational efficiency.

    Also keep in mind that when or if operational efficiency becomes an issue, then
    for Python the answer is in general to use functionality (modules) implemented
    in other languages. Python is not designed with operational efficiency as a
    goal. It's designed as an easy language.


    Cheers & hth.,

    - Alf
    Alf P. Steinbach, Apr 22, 2010
    #2
    1. Advertising

  3. candide, 22.04.2010 09:10:
    > Suppose a and b are lists.
    >
    > What is more efficient in order to extend the list a by appending all
    > the items in the list b ?
    >
    > I imagine a.extend(b)to be more efficient for only appendinding the
    > items from b while a+=b creates a copy of a before appending, right ?


    Wrong.

    Try it out yourself:

    $ python2.6 -m timeit -s "l=range(1000)" "a=l[:]; a+=l"
    100000 loops, best of 3: 9.16 usec per loop
    $ python2.6 -m timeit -s "l=range(1000)" "a=l[:]; a.extend(l)"
    100000 loops, best of 3: 9.24 usec per loop

    $ python2.6 -m timeit -s "l=range(10000)" "a=l[:]; a.extend(l)"
    10000 loops, best of 3: 96 usec per loop
    $ python2.6 -m timeit -s "l=range(10000)" "a=l[:]; a+=l"
    10000 loops, best of 3: 96.7 usec per loop

    $ python2.6 -m timeit -s "l=range(10000)" "a=l[:]; a+=l; a+=l"
    10000 loops, best of 3: 151 usec per loop
    $ python2.6 -m timeit -s "l=range(10000)" \
    "a=l[:]; a.extend(l); a.extend(l)"
    1000 loops, best of 3: 164 usec per loop

    Stefan
    Stefan Behnel, Apr 22, 2010
    #3
  4. candide

    Peter Otten Guest

    candide wrote:

    > Suppose a and b are lists.
    >
    > What is more efficient in order to extend the list a by appending all
    > the items in the list b ?
    >
    >
    > I imagine a.extend(b)to be more efficient for only appendinding the
    > items from b while a+=b creates a copy of a before appending, right ?


    No. Both append items to the original list a:

    >>> a = original_a = [1,2,3]
    >>> a.extend([4,5,6])
    >>> a is original_a # you knew that

    True
    >>> a += [7,8,9] # could rebind a
    >>> a is original_a # but effectively doesn't for lists

    True

    (Strictly speaking a += [...] rebinds a to the same value, like a = a)

    It is mostly a matter of personal preference which form you use.
    I prefer the extend() method because I think it's clearer; you don't run the
    risk of mistaking it for an arithmetic operation.

    Peter

    PS: an example where += does rebind:
    >>> a = original_a = (1,2,3)
    >>> a += (4,5,6)
    >>> a is original_a

    False
    Peter Otten, Apr 22, 2010
    #4
  5. candide

    candide Guest

    Alf P. Steinbach a écrit :
    > * candide:
    >> Suppose a and b are lists.
    >>
    >> What is more efficient in order to extend the list a by appending all
    >> the items in the list b ?
    >>
    >>
    >> I imagine a.extend(b)to be more efficient for only appendinding the
    >> items from b while a+=b creates a copy of a before appending, right ?

    >
    > No.
    >
    > But in general, if you're concerned about efficiency, *measure*.
    >



    But my question refers to memory management rather than to time
    execution. Imagine foo is a big size list and bar is a small one. It
    would be a waste of memory to copy list foo somewhere in memory before
    appending the items in bar.
    candide, Apr 22, 2010
    #5
  6. On Thu, 22 Apr 2010 09:31:12 +0200, candide wrote:

    > Alf P. Steinbach a écrit :
    >> * candide:
    >>> Suppose a and b are lists.
    >>>
    >>> What is more efficient in order to extend the list a by appending all
    >>> the items in the list b ?
    >>>
    >>>
    >>> I imagine a.extend(b)to be more efficient for only appendinding the
    >>> items from b while a+=b creates a copy of a before appending, right ?

    >>
    >> No.
    >>
    >> But in general, if you're concerned about efficiency, *measure*.
    >>
    >>

    >
    > But my question refers to memory management rather than to time
    > execution. Imagine foo is a big size list and bar is a small one. It
    > would be a waste of memory to copy list foo somewhere in memory before
    > appending the items in bar.




    Yes, unfortunately measuring Python's memory use from within Python is
    not easy... however the fact that += and extend take approximately the
    same time suggests that they are doing approximately the same amount of
    work, and therefore it's unlikely that one is radically more wasteful of
    memory than the other.

    Keep in mind that since Python uses duck-typing, you may not necessarily
    be dealing with actual built-in lists, but some other list-like object.
    If you're programming a library, and don't control your input, any
    conclusion you draw about lists may not apply in practice. If somebody
    provides you a sequence like this, what are you going to do?

    class StupidList(list):
    def extend(self, other):
    new = []
    for item in self:
    new = new[:] + item
    for item in other:
    new = new[:] + item
    return new
    def __iadd__(self, other):
    self.extend(other)
    return self

    You can't defend against the caller doing something stupid, so it's not
    worth spending too much effort trying.

    In general the warnings against premature optimization apply just as
    strongly against memory optimizations as against speed optimizations.



    --
    Steven
    Steven D'Aprano, Apr 22, 2010
    #6
  7. On Apr 22, 12:10 am, candide <> wrote:
    > Suppose a and b are lists.
    >
    > What is more efficient in order to extend the list a by appending all
    > the items in the list b ?
    >
    > I imagine a.extend(b)to be more efficient for only appendinding the
    > items from b while a+=b creates a copy of a before appending, right ?


    The a+=b form invokes list.__iadd__() which is implemented using
    list.extend(), so the two are basically the same. Looking at the
    source in http://svn.python.org/view/python/trunk/Objects/listobject.c?revision=78522&view=markup
    we see:

    static PyObject *
    list_inplace_concat(PyListObject *self, PyObject *other)
    {
    PyObject *result;

    result = listextend(self, other);
    if (result == NULL)
    return result;
    Py_DECREF(result);
    Py_INCREF(self);
    return (PyObject *)self;
    }

    There is a slight and constant difference is the overhead for making
    the call. The a.extend(b) does a dictionary lookup for the "extend"
    method and creates a bound method. Using a+=b is a little more
    direct.


    Raymond
    Raymond Hettinger, Apr 22, 2010
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Peter Bencsik
    Replies:
    2
    Views:
    808
  2. Chris Rebert

    Re: faster than list.extend()

    Chris Rebert, Nov 16, 2009, in forum: Python
    Replies:
    2
    Views:
    283
    Jason Sewall
    Nov 17, 2009
  3. J
    Replies:
    1
    Views:
    180
    Vinay Sajip
    Apr 21, 2010
  4. Replies:
    0
    Views:
    102
  5. Replies:
    2
    Views:
    42
    Mark H Harris
    May 13, 2014
Loading...

Share This Page