a.extend(b) better than a+=b ?

[candide]

Suppose a and b are lists.

What is more efficient in order to extend the list a by appending all
the items in the list b ?


I imagine a.extend(b)to be more efficient for only appendinding the
items from b while a+=b creates a copy of a before appending, right ?

 

[Alf P. Steinbach]

* candide:

Suppose a and b are lists.

What is more efficient in order to extend the list a by appending all
the items in the list b ?


I imagine a.extend(b)to be more efficient for only appendinding the
items from b while a+=b creates a copy of a before appending, right ?

No.

But in general, if you're concerned about efficiency, *measure*.

And keep in mind that measurement results may only be applicable to a given
Python implementation, for given data, in a given environment...

In most cases you'll do better by focusing on algorithmic efficiency rather than
low level operational efficiency.

Also keep in mind that when or if operational efficiency becomes an issue, then
for Python the answer is in general to use functionality (modules) implemented
in other languages. Python is not designed with operational efficiency as a
goal. It's designed as an easy language.


Cheers & hth.,

- Alf

 

[Stefan Behnel]

candide, 22.04.2010 09:10:

Suppose a and b are lists.

What is more efficient in order to extend the list a by appending all
the items in the list b ?

I imagine a.extend(b)to be more efficient for only appendinding the
items from b while a+=b creates a copy of a before appending, right ?

Wrong.

Try it out yourself:

$ python2.6 -m timeit -s "l=range(1000)" "a=l[:]; a+=l"
100000 loops, best of 3: 9.16 usec per loop
$ python2.6 -m timeit -s "l=range(1000)" "a=l[:]; a.extend(l)"
100000 loops, best of 3: 9.24 usec per loop

$ python2.6 -m timeit -s "l=range(10000)" "a=l[:]; a.extend(l)"
10000 loops, best of 3: 96 usec per loop
$ python2.6 -m timeit -s "l=range(10000)" "a=l[:]; a+=l"
10000 loops, best of 3: 96.7 usec per loop

$ python2.6 -m timeit -s "l=range(10000)" "a=l[:]; a+=l; a+=l"
10000 loops, best of 3: 151 usec per loop
$ python2.6 -m timeit -s "l=range(10000)" \
"a=l[:]; a.extend(l); a.extend(l)"
1000 loops, best of 3: 164 usec per loop

Stefan

 

[Peter Otten]

Suppose a and b are lists.

What is more efficient in order to extend the list a by appending all
the items in the list b ?


I imagine a.extend(b)to be more efficient for only appendinding the
items from b while a+=b creates a copy of a before appending, right ?

No. Both append items to the original list a:

a = original_a = [1,2,3]
a.extend([4,5,6])
a is original_a # you knew that True
a += [7,8,9] # could rebind a
a is original_a # but effectively doesn't for lists

True

(Strictly speaking a += [...] rebinds a to the same value, like a = a)

It is mostly a matter of personal preference which form you use.
I prefer the extend() method because I think it's clearer; you don't run the
risk of mistaking it for an arithmetic operation.

Peter

PS: an example where += does rebind:False

 

[candide]

Alf P. Steinbach a écrit :

* candide:

No.

But in general, if you're concerned about efficiency, *measure*.

But my question refers to memory management rather than to time
execution. Imagine foo is a big size list and bar is a small one. It
would be a waste of memory to copy list foo somewhere in memory before
appending the items in bar.

 

[Steven D'Aprano]

Alf P. Steinbach a écrit :

But my question refers to memory management rather than to time
execution. Imagine foo is a big size list and bar is a small one. It
would be a waste of memory to copy list foo somewhere in memory before
appending the items in bar.

Yes, unfortunately measuring Python's memory use from within Python is
not easy... however the fact that += and extend take approximately the
same time suggests that they are doing approximately the same amount of
work, and therefore it's unlikely that one is radically more wasteful of
memory than the other.

Keep in mind that since Python uses duck-typing, you may not necessarily
be dealing with actual built-in lists, but some other list-like object.
If you're programming a library, and don't control your input, any
conclusion you draw about lists may not apply in practice. If somebody
provides you a sequence like this, what are you going to do?

class StupidList(list):
def extend(self, other):
new = []
for item in self:
new = new[:] + item
for item in other:
new = new[:] + item
return new
def __iadd__(self, other):
self.extend(other)
return self

You can't defend against the caller doing something stupid, so it's not
worth spending too much effort trying.

In general the warnings against premature optimization apply just as
strongly against memory optimizations as against speed optimizations.

 

[Raymond Hettinger]

Suppose a and b are lists.

What is more efficient in order to extend the list a by appending all
the items in the list b ?

I imagine a.extend(b)to be more efficient for only appendinding the
items from b while a+=b creates a copy of a before appending, right ?

The a+=b form invokes list.__iadd__() which is implemented using
list.extend(), so the two are basically the same. Looking at the
source in http://svn.python.org/view/python/trunk/Objects/listobject.c?revision=78522&view=markup
we see:

static PyObject *
list_inplace_concat(PyListObject *self, PyObject *other)
{
PyObject *result;

result = listextend(self, other);
if (result == NULL)
return result;
Py_DECREF(result);
Py_INCREF(self);
return (PyObject *)self;
}

There is a slight and constant difference is the overhead for making
the call. The a.extend(b) does a dictionary lookup for the "extend"
method and creates a bound method. Using a+=b is a little more
direct.


Raymond