how to remove multiple occurrences of a string within a list?

[mik3l3374]

target = "0024"
l = ["0024", "haha", "0024"]

for index, val in enumerate(l):
if val==target:
del l[index]

print l

This latter suggestion (with the for loop) seems to be buggy: if there
are multiple items in the list "l" equal to "target", then only the
first one will be removed!

Thanks anyways.

Prove it.

here's something:

l = ["0024", "haha", "0024", "0024", "sfs"]
target = "0024"
for index, val in enumerate(l):

.... print "index: " , index , "value: " , val
.... del l[index]
.... print "after del: " , l
....
index: 0 value: 0024
after del: ['haha', '0024', '0024', 'sfs']
index: 1 value: 0024
after del: ['haha', '0024', 'sfs']
index: 2 value: sfs
after del: ['haha', '0024']['haha', '0024']

 

[mik3l3374]

Hi,

I have a list like ['0024', 'haha', '0024']
and as output I want ['haha']

If I
myList.remove('0024')

then only the first instance of '0024' is removed.

It seems like regular expressions is the rescue, but I couldn't find
the right tool.

Thanks!
bahoo

how about this:

target = "0024"
l = ["0024", "haha", "0024", "0024", "sfs"]
result = [ item for item in l if item != target]
result

['haha', 'sfs']

 

[=?ISO-8859-1?Q?Ma=EBl_Benjamin_Mettler?=]

How about:

list(frozenset(['0024', 'haha', '0024']))

Hi,

I have a list like ['0024', 'haha', '0024']
and as output I want ['haha']

If I
myList.remove('0024')

then only the first instance of '0024' is removed.

It seems like regular expressions is the rescue, but I couldn't find
the right tool.

Thanks!
bahoo

how about this:

target = "0024"
l = ["0024", "haha", "0024", "0024", "sfs"]
result = [ item for item in l if item != target]
result

['haha', 'sfs']

 

[Bruno Desthuilliers]

bahoo a écrit :

Hi,

I have a list like ['0024', 'haha', '0024']
and as output I want ['haha']

If I
myList.remove('0024')

then only the first instance of '0024' is removed.

It seems like regular expressions is the rescue,

Nope. re wouldn't help here - it works on strings, not on lists (you
need to understand that in Python, strings are *not* 'lists of chars').

Use list comps:
mylist = [item for item in mylist if item != '0024']

or filter() :
mylist = filter(lambda item: item != '0024', mylist)

but I couldn't find
the right tool.

May I second Grant Edward's suggestion in previous thread ? You'd really
do yourself a favor by following a couple Python tutorials. I'd say the
one in the official doc, and Dive into Python.

 

[Hendrik van Rooyen]

On Apr 3, 3:53 pm, "bahoo" <[email protected]> wrote:

My first submission handles duplicates, but not triplicates and more.
Here is one that seems a bit more bulletproof:

duplist = [1, 2, 3, 4, 'haha', 1, 2, 3, 4, 5, 1,2,3,4,6,7,7,7,7,7]
copylist = duplist[:]
fullset = set(duplist)
for x in duplist:
del(copylist[copylist.index(x)])
if x in copylist:
if x in fullset:
fullset.remove(x)

print list(fullset)

when it is run, I get:

IDLE 1.1.3 ==== No Subprocess ====

[5, 6, 'haha']

Now how would one do it and preserve the original order?
This apparently simple problem is surprisingly FOS...
But list comprehension to the rescue :

[x for x in duplist if duplist.count(x) == 1] ['haha', 5, 6]

*shakes head* duh... why does it take so long?

: - (

- Hendrik

 

[Goldfish]

I don't think I would use sets at all. They change the semantic
meaning of the original list. What if you have duplicate entries that
you want to keep? Converting to a set and back will strip out the
duplicates of that.

 

[7stud]

target = "0024"
l = ["0024", "haha", "0024"]
for index, val in enumerate(l):
if val==target:
del l[index]
print l
This latter suggestion (with the for loop) seems to be buggy: if there
are multiple items in the list "l" equal to "target", then only the
first one will be removed!
Thanks anyways.

Prove it.

Try replacing l = ["0024", "haha", "0024"]
with

l = ["0024", "0024", "haha"]

and re-running the code.

Actually, the description of the bug isn't quite right. The behaviour of
the for loop isn't defined -- sometimes it will work, sometimes it won't,
depending on how many items there are, and which of them are equal to the
target.

Thank you for the explanation.

 

[Ayaz Ahmed Khan]

"kyosohma" typed:

If you want to get really fancy, you could do a list comprehension
too:

your_list = ["0024","haha","0024"]
new_list = [i for i in your_list if i != '0024']

Or, just:

In [1]: l = ["0024","haha","0024"]
In [2]: filter(lambda x: x != "0024", l)
Out[2]: ['haha']

 

[Steven Bethard]

"kyosohma" typed:

If you want to get really fancy, you could do a list comprehension
too:

your_list = ["0024","haha","0024"]
new_list = [i for i in your_list if i != '0024']

Or, just:

In [1]: l = ["0024","haha","0024"]
In [2]: filter(lambda x: x != "0024", l)
Out[2]: ['haha']

Only if you want to make your code harder to read and slower::

$ python -m timeit -s "L = ['0024', 'haha', '0024']"
"[i for i in L if i != '0024']"
1000000 loops, best of 3: 0.679 usec per loop

$ python -m timeit -s "L = ['0024', 'haha', '0024']"
"filter(lambda i: i != '1024', L)"
1000000 loops, best of 3: 1.38 usec per loop

There really isn't much use for filter() anymore. Even in the one place
I would have expected it to be faster, it's slower::

$ python -m timeit -s "L = ['', 'a', '', 'b']" "filter(None, L)"
1000000 loops, best of 3: 0.789 usec per loop

$ python -m timeit -s "L = ['', 'a', '', 'b']" "[i for i in L if i]"
1000000 loops, best of 3: 0.739 usec per loop

STeVe

 

[Michael J. Fromberger]

Hi,

I have a list like ['0024', 'haha', '0024']
and as output I want ['haha']

If I
myList.remove('0024')

then only the first instance of '0024' is removed.

It seems like regular expressions is the rescue, but I couldn't find
the right tool.

If you know in advance which items are duplicated, then there have been
several simple solutions already proposed. Here's another way to tackle
the problem of removing ANY duplicated item from the list (i.e., any
string that appears > 1 time).

def killdups(lst):
"""Filter duplicated elements from the input list, and return the
remaining (unique) items in their original order.
"""
count = {}
for elt in lst:
count[elt] = count.get(elt, 0) + 1
return [elt for elt in lst if count[elt] == 1]

This solution is not particularly tricky, but it has the nice properties
that:

1. It works on lists of any hashable type, not just strings,
2. It preserves the order of the unfiltered items,
3. It makes only two passes over the input list.

Cheers,
-M

 

[Alex Martelli]

Hi,

I have a list like ['0024', 'haha', '0024']
and as output I want ['haha']

If I
myList.remove('0024')

then only the first instance of '0024' is removed.

To remove all items with multiple occurances in myList:

list(set(myList) - set([x for x in myList if myList.count(x)>1]))

This approach is unfortunately quite inefficient, because each call to
myList.count is O(N), and there are O(N) such calls, so the whole thing
is O(N squared). [also, the inner square brackets are not needed, and
cause more memory to be consumed than omitting them would].


A "we-don't-need-no-stinkin'-one-liners" more relaxed approach:

import collections
d = collections.defaultdict(int)
for x in myList: d[x] += 1
list(x for x in myList if d[x]==1)

yields O(N) performance (give that dict-indexing is about O(1)...).


Collapsing this kind of approach back into a "one-liner" while keeping
the O(N) performance is not easy -- whether this is a fortunate or
unfortunate ocurrence is of course debatable. If we had a "turn
sequence into bag" function somewhere (and it might be worth having it
for other reasons):

def bagit(seq):
import collections
d = collections.defaultdict(int)
for x in seq: d[x] += 1
return d

then one-linerness might be achieved in the "gimme nonduplicated items"
task via a dirty, rotten trick...:

list(x for bag in [bagit(myList)] for x in myList if bag[x] == 1)

....which I would of course never mention in polite company...


Alex

 

[Paul Rubin]

I have a list like ['0024', 'haha', '0024']
and as output I want ['haha']

If I
myList.remove('0024')

then only the first instance of '0024' is removed.

[x for x in myList if x != '0024']

 

[Ayaz Ahmed Khan]

"Steven Bethard" typed:

Or, just:

In [1]: l = ["0024","haha","0024"]
In [2]: filter(lambda x: x != "0024", l)
Out[2]: ['haha']

Only if you want to make your code harder to read and slower::

Slower, I can see. But harder to read?

There really isn't much use for filter() anymore. Even in the one place
I would have expected it to be faster, it's slower::

$ python -m timeit -s "L = ['', 'a', '', 'b']" "filter(None, L)"
1000000 loops, best of 3: 0.789 usec per loop

$ python -m timeit -s "L = ['', 'a', '', 'b']" "[i for i in L if i]"
1000000 loops, best of 3: 0.739 usec per loop

I am getting varying results on my system on repeated runs. What about
itertools.ifilter()?

$ python -m timeit -s "L = ['0024', 'haha', '0024']; import itertools"
"itertools.ifilter(lambda i: i != '1024', L)"
100000 loops, best of 3: 5.37 usec per loop

$ python -m timeit -s "L = ['0024', 'haha', '0024']"
"[i for i in L if i != '0024']"
100000 loops, best of 3: 5.41 usec per loop

$ python -m timeit -s "L = ['0024', 'haha', '0024']" "[i for i in L if i]"
100000 loops, best of 3: 6.71 usec per loop

$ python -m timeit -s "L = ['0024', 'haha', '0024']; import itertools"
"itertools.ifilter(None, L)"
100000 loops, best of 3: 4.12 usec per loop

 

[attn.steven.kuo]

On Apr 4, 7:43 pm, (e-mail address removed) (Alex Martelli) wrote:

(snipped)

A "we-don't-need-no-stinkin'-one-liners" more relaxed approach:

import collections
d = collections.defaultdict(int)
for x in myList: d[x] += 1
list(x for x in myList if d[x]==1)

yields O(N) performance (give that dict-indexing is about O(1)...).

Collapsing this kind of approach back into a "one-liner" while keeping
the O(N) performance is not easy -- whether this is a fortunate or
unfortunate ocurrence is of course debatable. If we had a "turn
sequence into bag" function somewhere (and it might be worth having it
for other reasons):

def bagit(seq):
import collections
d = collections.defaultdict(int)
for x in seq: d[x] += 1
return d

then one-linerness might be achieved in the "gimme nonduplicated items"
task via a dirty, rotten trick...:

list(x for bag in [bagit(myList)] for x in myList if bag[x] == 1)

...which I would of course never mention in polite company...

Alex

With a "context managed" set one could have, as a one-liner:

with cset() as s: retval = [ (x, s.add(x))[0] for x in myList if x not
in s ]

I've not looked at the underlying implementation of set(), but I
presume that checking set membership is also about O(1).

 

[waylan]

The larger problem is, I have a list of strings that I want to remove
from another list of strings.

If you don't care about the resulting order::

items = ['foo', 'bar', 'baz', 'bar', 'foo', 'frobble']
to_remove = ['foo', 'bar']
set(items) - set(to_remove)

set(['frobble', 'baz'])

I'm surprised no one has mentioned any of the methods of set. For
instance:
'Return the difference of two sets as a new set.\n\n(i.e. all
elements that are in this set but not in the other.)' set(['frobble', 'baz'])

There are a few other cool methods of sets that come in handy for this
sort of thing. If only order could be preserved.

If you do care about the resulting order::

to_remove = set(to_remove)
[item for item in items if item not in to_remove]

['baz', 'frobble']

STeVe

 

[Steven D'Aprano]

Now how would one do it and preserve the original order?
This apparently simple problem is surprisingly FOS...
But list comprehension to the rescue :

[x for x in duplist if duplist.count(x) == 1] ['haha', 5, 6]

*shakes head* duh... why does it take so long?

Because you are using Shlemiel the painter's algorithm:

http://www.joelonsoftware.com/articles/fog0000000319.html

Each time you call duplist.count(), you go back to the beginning
of the list and walk the entire list. You end up walking the list over
and over and over again.

 

[Alex Martelli]

I am getting varying results on my system on repeated runs. What about
itertools.ifilter()?

Calling itertools.ifilter returns an iterator; if you never iterate on
that iterator, that, of course, is going to be very fast (O(1), since it
does not matter how long the list you _don't_ iterate on is), but why
would you care how long it takes to do no useful work at all?

$ python -m timeit -s "L = ['0024', 'haha', '0024']; import itertools"

"itertools.ifilter(lambda i: i != '1024', L)"

100000 loops, best of 3: 5.37 usec per loop

$ python -m timeit -s "L = ['0024', 'haha', '0024']"
"[i for i in L if i != '0024']"
100000 loops, best of 3: 5.41 usec per loop

Here are the numbers I see:

brain:~ alex$ python -m timeit -s "L = ['0024', 'haha', '0024'];

import itertools" "itertools.ifilter(lambda i: i != '1024', L)"

1000000 loops, best of 3: 0.749 usec per loop

This is the "we're not doing any work" timing.

brain:~ alex$ python -m timeit -s "L = ['0024', 'haha',
'0024']" "[i for i in L if i != '1024']"
1000000 loops, best of 3: 1.37 usec per loop

This is "make a list in the obvious way, excluding an item that's never
there" (like in your code's first case, I'm comparing with 1024, which
ain't there, rather than with 0024, which is).

brain:~ alex$ python -m timeit -s "L = ['0024', 'haha',
'0024']; import itertools" "list(itertools.ifilter(lambda i: i !=
'1024', L))"
100000 loops, best of 3: 6.18 usec per loop

This is the "make it the hard way" (excluding a non-present item).
About 5/6 of the overhead comes from the list constructor.

When we exclude the item that IS there twice:

brain:~ alex$ python -m timeit -s "L = ['0024', 'haha',
'0024']" "[i for i in L if i != '0024']"
1000000 loops, best of 3: 0.918 usec per loop

this is the "obvious way to do it",

brain:~ alex$ python -m timeit -s "L = ['0024', 'haha',
'0024']; import itertools" "list(itertools.ifilter(lambda i: i !=
'0024', L))"
100000 loops, best of 3: 6.16 usec per loop

and this is the "hard and contorted way".


If you only want to loop, not to build a list, itertools.ifilter (or a
genexp) may be convenient (if the original list is long); but for making
lists, list comprehensions win hand-down.

Here are a few cases of "just looping" on lists of middling size:

brain:~ alex$ python -m timeit -s "L = 123*['0024', 'haha',
'0024']" "for j in [i for i in L if i != '0024']: pass"
10000 loops, best of 3: 70 usec per loop

brain:~ alex$ python -m timeit -s "L = 123*['0024', 'haha',
'0024']" "for j in (i for i in L if i != '0024'): pass"
10000 loops, best of 3: 70.9 usec per loop

brain:~ alex$ python -m timeit -s "L = 123*['0024', 'haha',
'0024']; import itertools" "for j in itertools.ifilter(lambda i: i !=
'0024', L
): pass"
10000 loops, best of 3: 151 usec per loop

Here, the overhead of itertools.ifilter is only about twice that of a
genexp (or list comprehension; but the LC should cost relatively more as
L's length keeps growing, due to memory-allocation issues).

BTW, sometimes simplest is still best:

brain:~ alex$ python -m timeit -s "L = 123*['0024', 'haha',
'0024']" "for i in L:

if i != '0024':
pass"

10000 loops, best of 3: 52.5 usec per loop

I.e., when you're just looping... just loop!-)


Alex

 

[Paul McGuire]

Beware that converting a list to set and then back to list won't
preserve the order of the items, because the set-type loses the order.

Also beware that this conversion will remove duplicates, so that if
'haha' is in the original list multiple times, there will only be one
after converting to a set and then back to a list. Only you can tell
us whether this behavior is desirable or not.

-- Paul