string issue

Discussion in 'Python' started by Jeremy Bowers, Feb 4, 2005.

  1. On Fri, 04 Feb 2005 14:49:43 -0500, rbt wrote:

    > Alan McIntyre wrote:
    >> I think it's because you're modifying the list as you're iterating over

    >
    > In this case then, shouldn't my 'except Exception' raise an error or
    > warning like:
    >
    > "Hey, stupid, you can't iterate and object and change it at the same time!"


    But you can do that, so why should it?

    Certain operations are dangerous, but, well, it's impossible to enumerate
    all the dangerous things and it's better to not give people false
    assurances when they find one of them that slipped by.

    One idiom that does what you want, though it doesn't always work for all
    situations, is to iterate over the list backwards. If you're only removing
    a few items from a short list, that can be faster than building a new one.
    However, removing items gets to be n^2 pretty fast, so it can be better to
    build a new list, which the list comprehensions make easy:

    newList = [x for x in oldList if myCondition(x)]
     
    Jeremy Bowers, Feb 4, 2005
    #1
    1. Advertising

  2. Jeremy Bowers

    rbt Guest

    Either I'm crazy and I'm missing the obvious here or there is something
    wrong with this code. Element 5 of this list says it doesn't contain the
    string 255, when that's *ALL* it contains... why would it think that???

    import time

    ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']

    for ip in ips:
    if '255' in ip:
    try:
    print "Removing", ip
    ips.remove(ip)
    except Exception, e:
    print e

    print ips
    time.sleep(5)

    Someone tell me I'm going crazy ;)
     
    rbt, Feb 4, 2005
    #2
    1. Advertising

  3. Jeremy Bowers

    Steve Holden Guest

    rbt wrote:

    > Either I'm crazy and I'm missing the obvious here or there is something
    > wrong with this code. Element 5 of this list says it doesn't contain the
    > string 255, when that's *ALL* it contains... why would it think that???
    >
    > import time
    >
    > ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    > '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >
    > for ip in ips:
    > if '255' in ip:
    > try:
    > print "Removing", ip
    > ips.remove(ip)
    > except Exception, e:
    > print e
    >
    > print ips
    > time.sleep(5)
    >
    > Someone tell me I'm going crazy ;)


    You are modifying the list as you iterate over it. Instead, iterate over
    a copy by using:

    for ip in ips[:]:
    ...

    regards
    Steve
    --
    Meet the Python developers and your c.l.py favorites March 23-25
    Come to PyCon DC 2005 http://www.pycon.org/
    Steve Holden http://www.holdenweb.com/
     
    Steve Holden, Feb 4, 2005
    #3
  4. I think it's because you're modifying the list as you're iterating over
    it. Try this:

    import time

    ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    ips_new = []
    for ip in ips:
    if '255' not in ip:
    ips_new.append(ip)

    print ips_new



    Or this:

    ips_new = [ip for ip in ips if '255' not in ip]
    print ips_new


    Hope this helps,
    Alan McIntyre
    http://www.esrgtech.com



    rbt wrote:
    > Either I'm crazy and I'm missing the obvious here or there is something
    > wrong with this code. Element 5 of this list says it doesn't contain the
    > string 255, when that's *ALL* it contains... why would it think that???
    >
    > import time
    >
    > ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    > '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >
    > for ip in ips:
    > if '255' in ip:
    > try:
    > print "Removing", ip
    > ips.remove(ip)
    > except Exception, e:
    > print e
    >
    > print ips
    > time.sleep(5)
    >
    > Someone tell me I'm going crazy ;)
     
    Alan McIntyre, Feb 4, 2005
    #4
  5. rbt wrote:
    > Either I'm crazy and I'm missing the obvious here or there is something
    > wrong with this code. Element 5 of this list says it doesn't contain the
    > string 255, when that's *ALL* it contains... why would it think that???
    >
    > import time
    >
    > ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    > '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >
    > for ip in ips:
    > if '255' in ip:
    > try:
    > print "Removing", ip
    > ips.remove(ip)
    > except Exception, e:
    > print e
    >
    > print ips
    > time.sleep(5)
    >
    > Someone tell me I'm going crazy ;)


    You're going crazy. ;)

    py> ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    py> for ip in ips:
    .... # debugging statement:
    .... print "Looking at", ip
    .... if '255' in ip:
    .... try:
    .... print "Removing", ip
    .... ips.remove(ip)
    .... except Exception, e:
    .... print e
    ....
    Looking at 255.255.255.255
    Removing 255.255.255.255
    Looking at 198.82.247.98
    Looking at 127.0.0.1
    Looking at 255.0.0.0
    Removing 255.0.0.0
    Looking at 128.173.255.34
    Removing 128.173.255.34

    Notice how elements of your list are being skipped. The problem is that
    you're modifying a list while you iterate over it.

    Why don't you try a list comprehension:

    py> ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    py> [ip for ip in ips if '255' not in ip]
    ['128.173.120.79', '198.82.247.98', '127.0.0.1']

    Steve
     
    Steven Bethard, Feb 4, 2005
    #5
  6. Jeremy Bowers

    Bill Mill Guest

    On Fri, 04 Feb 2005 14:23:36 -0500, rbt <> wrote:
    > Either I'm crazy and I'm missing the obvious here or there is something
    > wrong with this code. Element 5 of this list says it doesn't contain the
    > string 255, when that's *ALL* it contains... why would it think that???
    >
    > import time
    >
    > ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    > '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >
    > for ip in ips:
    > if '255' in ip:
    > try:
    > print "Removing", ip
    > ips.remove(ip)
    > except Exception, e:
    > print e
    >
    > print ips
    > time.sleep(5)
    >


    You're gong crazy:

    >>> ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',

    .... '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >>> for ip in ips:

    .... if '255' in ip: print ip
    ....
    255.255.255.255
    255.0.0.0
    255
    128.173.255.34

    The problem is that you're operating in-place on an array while it's
    being iterated over. Since the iterator is only created once, you're
    can't change the array while you're iterating over it. Instead, try a
    list comprehension:

    >>> ips = [ip for ip in ips if '255' not in ip]
    >>> ips

    ['128.173.120.79', '198.82.247.98', '127.0.0.1']

    Peace
    Bill Mill
    bill.mill at gmail.com
     
    Bill Mill, Feb 4, 2005
    #6
  7. Jeremy Bowers

    rbt Guest

    Thanks guys... list comprehension it is!

    Bill Mill wrote:
    > On Fri, 04 Feb 2005 14:23:36 -0500, rbt <> wrote:
    >
    >>Either I'm crazy and I'm missing the obvious here or there is something
    >>wrong with this code. Element 5 of this list says it doesn't contain the
    >>string 255, when that's *ALL* it contains... why would it think that???
    >>
    >>import time
    >>
    >>ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    >>'127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >>
    >>for ip in ips:
    >> if '255' in ip:
    >> try:
    >> print "Removing", ip
    >> ips.remove(ip)
    >> except Exception, e:
    >> print e
    >>
    >>print ips
    >>time.sleep(5)
    >>

    >
    >
    > You're gong crazy:
    >
    >
    >>>>ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',

    >
    > ... '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >
    >>>>for ip in ips:

    >
    > ... if '255' in ip: print ip
    > ...
    > 255.255.255.255
    > 255.0.0.0
    > 255
    > 128.173.255.34
    >
    > The problem is that you're operating in-place on an array while it's
    > being iterated over. Since the iterator is only created once, you're
    > can't change the array while you're iterating over it. Instead, try a
    > list comprehension:
    >
    >
    >>>>ips = [ip for ip in ips if '255' not in ip]
    >>>>ips

    >
    > ['128.173.120.79', '198.82.247.98', '127.0.0.1']
    >
    > Peace
    > Bill Mill
    > bill.mill at gmail.com
     
    rbt, Feb 4, 2005
    #7
  8. Wow, that's cool; I'd never seen that before. :) Thanks, Steve..

    Steve Holden wrote:
    > You are modifying the list as you iterate over it. Instead, iterate over
    > a copy by using:
    >
    > for ip in ips[:]:
    > ...
    >
    > regards
    > Steve
     
    Alan McIntyre, Feb 4, 2005
    #8
  9. Jeremy Bowers

    rbt Guest

    Steve Holden wrote:
    > rbt wrote:
    >
    >> Either I'm crazy and I'm missing the obvious here or there is
    >> something wrong with this code. Element 5 of this list says it doesn't
    >> contain the string 255, when that's *ALL* it contains... why would it
    >> think that???
    >>
    >> import time
    >>
    >> ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    >> '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >>
    >> for ip in ips:
    >> if '255' in ip:
    >> try:
    >> print "Removing", ip
    >> ips.remove(ip)
    >> except Exception, e:
    >> print e
    >>
    >> print ips
    >> time.sleep(5)
    >>
    >> Someone tell me I'm going crazy ;)

    >
    >
    > You are modifying the list as you iterate over it. Instead, iterate over
    > a copy by using:
    >
    > for ip in ips[:]:
    > ...
    >
    > regards
    > Steve


    Very neat. That's a trick that everyone should know about. I vote it
    goes in Dr. Dobbs newsletter.
     
    rbt, Feb 4, 2005
    #9
  10. Steve Holden wrote:
    > You are modifying the list as you iterate over it. Instead, iterate over
    > a copy by using:
    >
    > for ip in ips[:]:
    > ...


    Also worth noting, you can write this like:

    for ip in list(ips):
    ...

    if you're afraid of smiley-faces [:] in your code. ;)

    Steve
     
    Steven Bethard, Feb 4, 2005
    #10
  11. Jeremy Bowers

    rbt Guest

    Alan McIntyre wrote:
    > I think it's because you're modifying the list as you're iterating over


    In this case then, shouldn't my 'except Exception' raise an error or
    warning like:

    "Hey, stupid, you can't iterate and object and change it at the same time!"

    Or, perhaps something similar?
     
    rbt, Feb 4, 2005
    #11
  12. "rbt" <> wrote:

    >> You are modifying the list as you iterate over it. Instead, iterate over a copy by using:
    >>
    >> for ip in ips[:]:
    >> ...
    >>
    >> regards
    >> Steve

    >
    > Very neat. That's a trick that everyone should know about.


    I suppose that's why it's included in the Python tutorial:

    http://docs.python.org/tut/node6.html#SECTION006200000000000000000

    It is not safe to modify the sequence being iterated over in the loop
    (this can only happen for mutable sequence types, such as lists). If
    you need to modify the list you are iterating over (for example, to
    duplicate selected items) you must iterate over a copy. The slice
    notation makes this particularly convenient:

    >>> for x in a[:]: # make a slice copy of the entire list

    ... if len(x) > 6: a.insert(0, x)

    (slice notation is explained in an earlier chapter)

    </F>
     
    Fredrik Lundh, Feb 4, 2005
    #12
  13. Jeremy Bowers

    rbt Guest

    Alan McIntyre wrote:
    > I think it's because you're modifying the list as you're iterating over
    > it.


    One last clarification on this. It's OK to modify the elements of a
    list, but not the list itself while iterating over it... is that the
    correct way to think about this?
     
    rbt, Feb 4, 2005
    #13
  14. Jeremy Bowers

    Bill Mill Guest

    On Fri, 04 Feb 2005 14:43:30 -0500, rbt <> wrote:
    > Steve Holden wrote:
    > > rbt wrote:
    > >
    > >> Either I'm crazy and I'm missing the obvious here or there is
    > >> something wrong with this code. Element 5 of this list says it doesn't
    > >> contain the string 255, when that's *ALL* it contains... why would it
    > >> think that???
    > >>
    > >> import time
    > >>
    > >> ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    > >> '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    > >>
    > >> for ip in ips:
    > >> if '255' in ip:
    > >> try:
    > >> print "Removing", ip
    > >> ips.remove(ip)
    > >> except Exception, e:
    > >> print e
    > >>
    > >> print ips
    > >> time.sleep(5)
    > >>
    > >> Someone tell me I'm going crazy ;)

    > >
    > >
    > > You are modifying the list as you iterate over it. Instead, iterate over
    > > a copy by using:
    > >
    > > for ip in ips[:]:
    > > ...
    > >
    > > regards
    > > Steve

    >
    > Very neat. That's a trick that everyone should know about. I vote it
    > goes in Dr. Dobbs newsletter.


    Once you know it, it's neat, and I use it sometimes. However, it's a
    little too "magical" for my tastes; I'd rather be more explicit about
    what's going on.

    Peace
    Bill Mill
    bill.mill at gmail.com

    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    Bill Mill, Feb 4, 2005
    #14
  15. Jeremy Bowers

    John J. Lee Guest

    Steve Holden <> writes:
    [...]
    > You are modifying the list as you iterate over it. Instead, iterate
    > over a copy by using:
    >
    > for ip in ips[:]:
    > ...


    Just to help popularise the alternative idiom, which IMO is
    significantly less cryptic (sane constructors of mutable objects
    almost always make a copy, and list is no exception: it's guaranteed
    to do so):

    for ip in list(ips):
    ...


    Works back to at least Python 1.5.2.


    John
     
    John J. Lee, Feb 4, 2005
    #15
  16. Jeremy Bowers

    rbt Guest

    John J. Lee wrote:
    > Steve Holden <> writes:
    > [...]
    >
    >>You are modifying the list as you iterate over it. Instead, iterate
    >>over a copy by using:
    >>
    >>for ip in ips[:]:
    >> ...

    >
    >
    > Just to help popularise the alternative idiom, which IMO is
    > significantly less cryptic (sane constructors of mutable objects
    > almost always make a copy, and list is no exception: it's guaranteed
    > to do so):
    >
    > for ip in list(ips):
    > ...
    >
    >
    > Works back to at least Python 1.5.2.
    >
    >
    > John


    I don't know that that approach is less cryptic. ips is already a
    list... it looks cryptic to make it a list again, doesn't it? IMO, the
    two are equally cryptic. The epitome of clarity would be copy(ips)...
    now *that* makes sense, of course, ips[:] or list(ips) work equally well
    to the programmer who has learned them.
     
    rbt, Feb 4, 2005
    #16
  17. Jeremy Bowers

    Bill Mill Guest

    On Fri, 04 Feb 2005 15:25:04 -0500, rbt <> wrote:
    > John J. Lee wrote:
    > > Steve Holden <> writes:
    > > [...]
    > >
    > >>You are modifying the list as you iterate over it. Instead, iterate
    > >>over a copy by using:
    > >>
    > >>for ip in ips[:]:
    > >> ...

    > >
    > >
    > > Just to help popularise the alternative idiom, which IMO is
    > > significantly less cryptic (sane constructors of mutable objects
    > > almost always make a copy, and list is no exception: it's guaranteed
    > > to do so):
    > >
    > > for ip in list(ips):
    > > ...
    > >
    > >
    > > Works back to at least Python 1.5.2.
    > >
    > >
    > > John

    >
    > I don't know that that approach is less cryptic. ips is already a
    > list... it looks cryptic to make it a list again, doesn't it? IMO, the
    > two are equally cryptic. The epitome of clarity would be copy(ips)...
    > now *that* makes sense, of course, ips[:] or list(ips) work equally well
    > to the programmer who has learned them.


    Howsabout:

    >>> from copy import copy
    >>> ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',

    .... '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >>> for ip in copy(ips):

    .... if '255' in ip:
    .... ips.remove(ip)
    ....
    >>> ips

    ['128.173.120.79', '198.82.247.98', '127.0.0.1']

    But I still think that the list comprehension is the best.

    Peace
    Bill Mill
    bill.mill at gmail.com

    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
     
    Bill Mill, Feb 4, 2005
    #17
  18. Jeremy Bowers

    rbt Guest

    Bill Mill wrote:
    > On Fri, 04 Feb 2005 15:25:04 -0500, rbt <> wrote:
    >
    >>John J. Lee wrote:
    >>
    >>>Steve Holden <> writes:
    >>>[...]
    >>>
    >>>
    >>>>You are modifying the list as you iterate over it. Instead, iterate
    >>>>over a copy by using:
    >>>>
    >>>>for ip in ips[:]:
    >>>> ...
    >>>
    >>>
    >>>Just to help popularise the alternative idiom, which IMO is
    >>>significantly less cryptic (sane constructors of mutable objects
    >>>almost always make a copy, and list is no exception: it's guaranteed
    >>>to do so):
    >>>
    >>>for ip in list(ips):
    >>> ...
    >>>
    >>>
    >>>Works back to at least Python 1.5.2.
    >>>
    >>>
    >>>John

    >>
    >>I don't know that that approach is less cryptic. ips is already a
    >>list... it looks cryptic to make it a list again, doesn't it? IMO, the
    >>two are equally cryptic. The epitome of clarity would be copy(ips)...
    >>now *that* makes sense, of course, ips[:] or list(ips) work equally well
    >>to the programmer who has learned them.

    >
    >
    > Howsabout:
    >
    >
    >>>>from copy import copy
    >>>>ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',

    >
    > ... '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']
    >
    >>>>for ip in copy(ips):

    >
    > ... if '255' in ip:
    > ... ips.remove(ip)
    > ...
    >
    >>>>ips

    >
    > ['128.173.120.79', '198.82.247.98', '127.0.0.1']
    >
    > But I still think that the list comprehension is the best.
    >
    > Peace
    > Bill Mill
    > bill.mill at gmail.com


    Wow, I did not know that a copy module existed. I made all that up about
    copy being the perfect example here. Great minds think alike ;) I fell
    Guidoish.
     
    rbt, Feb 4, 2005
    #18
  19. Jeremy Bowers

    Terry Reedy Guest

    "Steve Holden" <> wrote in message
    news:36QMd.103084$Jk5.36127@lakeread01...
    > You are modifying the list as you iterate over it. Instead, iterate over
    > > a copy by using:

    > for ip in ips[:]:


    Or if you are only deleting items, you can iterate backwards.

    You can also build a new list with only the items you want. Do this either
    with an explicit loop or even better, use filter or a list comp.

    ips = ['255.255.255.255', '128.173.120.79', '198.82.247.98',
    '127.0.0.1', '255.0.0.0', '255', '128.173.255.34']

    # 2.2 requires single char for 'not in string' test

    ips2 = []
    for ip in ips:
    if ip.find('255') == -1: ips2.append(ip)

    print ips2
    print filter(lambda ip: ip.find('255') == -1, ips)
    print [ip for ip in ips if ip.find('255') == -1]

    # thrice prints
    ['128.173.120.79', '198.82.247.98', '127.0.0.1']

    This seems much more sensible to me than building a new list with
    everything (a copy), including things you don't want, and then deleting the
    things you don't want (an O(n**2) operation).

    Terry J. Reedy
     
    Terry Reedy, Feb 4, 2005
    #19
  20. Jeremy Bowers

    Steve Holden Guest

    Terry Reedy wrote:

    > "Steve Holden" <> wrote in message
    > news:36QMd.103084$Jk5.36127@lakeread01...
    >

    [ ... ]
    >
    > This seems much more sensible to me than building a new list with
    > everything (a copy), including things you don't want, and then deleting the
    > things you don't want (an O(n**2) operation).
    >

    Which is presumably why the OP decided to go with the list
    comprehension. My main intention was to point out how the code in error
    could be fixed while retaining the same structure. You are of course
    correct in saying this is far from optimal in performance terms.

    regards
    Steve
    --
    Meet the Python developers and your c.l.py favorites March 23-25
    Come to PyCon DC 2005 http://www.pycon.org/
    Steve Holden http://www.holdenweb.com/
     
    Steve Holden, Feb 4, 2005
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mladen Adamovic
    Replies:
    0
    Views:
    755
    Mladen Adamovic
    Dec 4, 2003
  2. Mladen Adamovic
    Replies:
    3
    Views:
    14,684
    Mladen Adamovic
    Dec 5, 2003
  3. Matt
    Replies:
    3
    Views:
    531
    Tor Iver Wilhelmsen
    Sep 17, 2004
  4. Bruce Sam
    Replies:
    15
    Views:
    7,987
    John C. Bollinger
    Nov 19, 2004
  5. Srini
    Replies:
    11
    Views:
    985
    Arne Vajhøj
    Jun 1, 2008
Loading...

Share This Page