Scope of variable inside list comprehensions?

Discussion in 'Python' started by Roy Smith, Dec 5, 2011.

  1. Roy Smith

    Roy Smith Guest

    Consider the following django snippet. Song(id) raises DoesNotExist if theid is unknown.

    try:
    songs = [Song(id) for id in song_ids]
    except Song.DoesNotExist:
    print "unknown song id (%d)" % id

    Is id guaranteed to be in scope in the print statement? I found one thread(http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)which says yes, but hints that it might not always be in the future. Now that we're in the future, is that still true? And for Python 3 also?

    The current docs, http://docs.python.org/tutorial/datastructures.html#list-comprehensions, are mute on this point.
     
    Roy Smith, Dec 5, 2011
    #1
    1. Advertising

  2. Roy Smith writes:

    > Consider the following django snippet. Song(id) raises DoesNotExist
    > if the id is unknown.
    >
    > try:
    > songs = [Song(id) for id in song_ids]
    > except Song.DoesNotExist:
    > print "unknown song id (%d)" % id
    >
    > Is id guaranteed to be in scope in the print statement? I found one
    > thread
    > (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)
    > which says yes, but hints that it might not always be in the future.
    > Now that we're in the future, is that still true? And for Python 3
    > also?


    Another id is in scope in this Python3 example (3.1.2, Ubuntu):

    >>> try:

    .... songs = [1/0 for id in [1]]
    .... except Exception:
    .... print('Caught', id)
    ....
    Caught <built-in function id>
     
    Jussi Piitulainen, Dec 5, 2011
    #2
    1. Advertising

  3. Roy Smith

    Peter Otten Guest

    Roy Smith wrote:

    > Consider the following django snippet. Song(id) raises DoesNotExist if
    > the id is unknown.
    >
    > try:
    > songs = [Song(id) for id in song_ids]
    > except Song.DoesNotExist:
    > print "unknown song id (%d)" % id
    >
    > Is id guaranteed to be in scope in the print statement? I found one
    > thread
    > (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html)
    > which says yes, but hints that it might not always be in the future. Now
    > that we're in the future, is that still true? And for Python 3 also?
    >
    > The current docs,
    > http://docs.python.org/tutorial/datastructures.html#list-comprehensions,
    > are mute on this point.


    If you are using a generator expression id will already be out of scope in
    Python 2. In Python 3 list comprehensions have been changed to work the same
    way:

    $ cat song.py
    class DoesNotExist(Exception):
    pass

    class Song:
    def __init__(self, id):
    if id == 2:
    raise DoesNotExist

    ids = [1, 2]
    try:
    songs = [Song(i) for i in ids]
    except DoesNotExist as e:
    print("song #%d does not exist" % i)
    $ python song.py
    song #2 does not exist
    $ python3 song.py
    Traceback (most recent call last):
    File "song.py", line 11, in <module>
    songs = [Song(i) for i in ids]
    File "song.py", line 11, in <listcomp>
    songs = [Song(i) for i in ids]
    File "song.py", line 7, in __init__
    raise DoesNotExist
    __main__.DoesNotExist

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "song.py", line 13, in <module>
    print("song #%d does not exist" % i)
    NameError: name 'i' is not defined
    $


    $ cat song_gen.py
    class DoesNotExist(Exception):
    pass

    class Song:
    def __init__(self, id):
    if id == 2:
    raise DoesNotExist

    ids = [1, 2]
    try:
    songs = list(Song(i) for i in ids)
    except DoesNotExist as e:
    print("song #%d does not exist" % i)
    $ python song_gen.py
    Traceback (most recent call last):
    File "song_gen.py", line 13, in <module>
    print("song #%d does not exist" % i)
    NameError: name 'i' is not defined

    So you'd rather store the id in the exception.
     
    Peter Otten, Dec 5, 2011
    #3
  4. Roy Smith wrote:
    > Consider the following django snippet. Song(id) raises DoesNotExist if the id is unknown.
    >
    > try:
    > songs = [Song(id) for id in song_ids]
    > except Song.DoesNotExist:
    > print "unknown song id (%d)" % id
    >
    > Is id guaranteed to be in scope in the print statement? I found one thread (http://mail.python.org/pipermail/python-bugs-list/2006-April/033235.html) which says yes, but hints that it might not always be in the future. Now that we're in the future, is that still true? And for Python 3 also?
    >
    > The current docs, http://docs.python.org/tutorial/datastructures.html#list-comprehensions, are mute on this point.
    >

    For python 2, id will always be defined *as you meant it*. But you're
    doing something wrong : overiding the 'id' builtin function.
    With python 3 you will probably print the 'id' builtin function
    representation, which is correct but not want you want to achieve.

    The proper way to propagate information with exceptions is using the
    exception itself:

    try:
    songs = [Song(_id) for _id in song_ids]
    except Song.DoesNotExist, exc:
    print exc

    class DoesNotExist(Exception):
    def __init__(self, songId):
    self.songId = songId
    def __str__(self):
    return "Unkown Song Id %s" % self.songId

    class Song:
    def __init__(self, songId):
    if whatever:
    raise DoesNotExist(songId)
    self.id=songId

    JM
     
    Jean-Michel Pichavant, Dec 5, 2011
    #4
  5. Roy Smith

    Roy Smith Guest

    Hmmm, the use of id was just a simplification for the sake of posting. The real code is a bit more complicated and used a different variable name, but that's a good point.

    As far as storing the value in the exception, unfortunately, DoesNotExist is not my exception; it comes from deep within django. I'm just passing it along.
     
    Roy Smith, Dec 5, 2011
    #5
  6. Roy Smith

    Roy Smith Guest

    Hmmm, the use of id was just a simplification for the sake of posting. The real code is a bit more complicated and used a different variable name, but that's a good point.

    As far as storing the value in the exception, unfortunately, DoesNotExist is not my exception; it comes from deep within django. I'm just passing it along.
     
    Roy Smith, Dec 5, 2011
    #6
  7. Roy Smith

    Terry Reedy Guest

    On 12/5/2011 2:15 PM, Roy Smith wrote:
    > Hmmm, the use of id was just a simplification for the sake of
    > posting. The real code is a bit more complicated and used a
    > different variable name, but that's a good point.
    >
    > As far as storing the value in the exception, unfortunately,
    > DoesNotExist is not my exception; it comes from deep within django.
    > I'm just passing it along.


    It is hard to sensibly answer a question when the questioner
    *significantly* changes the problem in the process of 'simplifying' it.
    Both of those are significant changes ;-)

    Changing a name to a built-in name is a complexification, not a
    simplification, because it introduces new issues that were not in the
    original.

    Changing the exception from one you do not control to one you apparently
    do also changes the appropriate answer. If you do not control the
    exception and you want guaranteed access to the loop variable with
    Python 3 (and the upgrade of django to work with Python 3 is more or
    less done), then use an explicit loop. If you want the loop to continue
    after an error, instead of stopping, you can put the try/except within
    the loop, instead of without. This possibility is one advantage of using
    an explicit loop.

    songs = []
    for song_id in song_ids:
    try:
    songs.append(Song(song_id))
    except django.error:
    print("unknown song id {}".format(song_id))

    --
    Terry Jan Reedy
     
    Terry Reedy, Dec 5, 2011
    #7
  8. On Mon, 05 Dec 2011 19:57:15 +0100, Jean-Michel Pichavant wrote:

    > The proper way to propagate information with exceptions is using the
    > exception itself:
    >
    > try:
    > songs = [Song(_id) for _id in song_ids]
    > except Song.DoesNotExist, exc:
    > print exc



    I'm not entirely sure that this is the proper way to propagate the
    exception. I see far to many people catching exceptions to print them, or
    worse, to print a generic, useless message like "an error occurred".

    The problem here is that having caught the exception, songs now does not
    exist, and will surely cause another, unexpected, exception in a moment
    or two when the code attempts to use it. Since the error is (apparently)
    unrecoverable, the right way as far as I can see is:

    songs = [Song(_id) for _id in song_ids]

    allowing any exception to be fatal and the traceback to be printed as
    normal.

    If the error is recoverable, you will likely need to do more to recover
    from it than merely print the exception and continue.



    --
    Steven
     
    Steven D'Aprano, Dec 5, 2011
    #8
  9. Roy Smith

    Roy Smith Guest

    Well, in my defense, I did ask a pretty narrow question, "Is id guaranteed to be in scope in the print statement?". While I will admit that not knowing whether I could alter the exception, or whether id masked a builtin or not does complexify answering some questions, those are questions I didn't ask :)
     
    Roy Smith, Dec 5, 2011
    #9
  10. Roy Smith

    Roy Smith Guest

    Well, in my defense, I did ask a pretty narrow question, "Is id guaranteed to be in scope in the print statement?". While I will admit that not knowing whether I could alter the exception, or whether id masked a builtin or not does complexify answering some questions, those are questions I didn't ask :)
     
    Roy Smith, Dec 5, 2011
    #10
  11. Roy Smith

    Roy Smith Guest

    Sigh. I attempted to reduce this to a minimal example to focus the discussion on the question of list comprehension variable scope. Instead I seem to have gotten people off on other tangents. I suppose I should post more of the real code...

    song_ids = request.POST.getlist('song_id')
    try:
    songs = [Song.get(int(id)) for id in song_ids]
    except Song.DoesNotExist:
    return HttpResponseBadRequest("unknown song id (%d)" % id)

    I may be in the minority here, but it doesn't bother me much that my use of'id' shadows a built-in. Especially in small scopes like this, I use whatever variable names make the the code easiest to read and don't worry aboutshadowing builtins. I don't have an exhaustive list of builtins in my head, so even if I worried about common ones like file or id, I'm sure I'd miss some others. So I don't sweat it.

    If shadowing builtins was really evil, they'd be reserved keywords and thenyou wouldn't be able to do it.
     
    Roy Smith, Dec 5, 2011
    #11
  12. Roy Smith

    Terry Reedy Guest

    On 12/5/2011 5:36 PM, Roy Smith wrote:
    > Well, in my defense, I did ask a pretty narrow question, "Is id
    > guaranteed to be in scope in the print statement?".


    Yes for 2.x, guaranteed no for 3.x.

    If you had simply asked "Is the loop variable of a list comprehension
    guaranteed to be in scope after the list comprehension?", without a
    distracting example, that is the answer you would have received.

    I intend(ed) to inform, not attack, hence no 'defense' needed.

    > While I will
    > admit that not knowing whether I could alter the exception, or
    > whether id masked a builtin or not does complexify answering some
    > questions, those are questions I didn't ask :)


    Except that it bears on the question you did ask because it means that
    the code will run in 3.x but with different results, whereas a random
    name will fail in 3.x with a NameError.

    --
    Terry Jan Reedy
     
    Terry Reedy, Dec 6, 2011
    #12
  13. On Tue, Dec 6, 2011 at 9:57 AM, Roy Smith <> wrote:
    > I may be in the minority here, but it doesn't bother me much that my use of 'id' shadows a built-in.  Especially in small scopes like this, I use whatever variable names make the the code easiest to read and don't worry about shadowing builtins.  I don't have an exhaustive list of builtins in my head, so even if I worried about common ones like file or id, I'm sure I'd miss some others.  So I don't sweat it.
    >
    > If shadowing builtins was really evil, they'd be reserved keywords and then you wouldn't be able to do it.


    Agreed. The name 'id' is one that's shadowed benignly in a lot of
    code. In the context of the original question, it's not an issue -
    there'll be no confusion.

    Python's scoping rules are "do what the programmer probably wants".
    This works most of the time, but occasionally you get edge cases where
    things are a bit weird, and C-style explicit scoping (with infinitely
    nested block scope) begins to look better. But for probably 99% of
    situations, Python's system "just works".

    If you need more flexibility in the exception you throw, it may be
    worth putting together a filter:

    def HttpSongId(id):
    try:
    return Song.get(int(id))
    except Song.DoesNotExist:
    return HttpResponseBadRequest("unknown song id (%d)" % id)

    song_ids = request.POST.getlist('song_id')
    songs = [HttpSongId(id) for id in song_ids]

    This encapsulates things in a somewhat weird way, but if there's any
    other work to be done at the same time, you could probably come up
    with a better name for the exception filter function.

    ChrisA
     
    Chris Angelico, Dec 6, 2011
    #13
  14. Roy Smith

    Rainer Grimm Guest

    Hello,

    > try:
    > songs = [Song(id) for id in song_ids]
    > except Song.DoesNotExist:
    > print "unknown song id (%d)" % id

    that's is a bad programming style. So it will be forbidden with python 3. The reason is that list comprehension is a construct from the functional world. It's only syntactic sugar for the functions map and filter. So functions have to be pure functions. To say it in other words, they have to be side-effect free. But the python construct from above pollutes the namespace with name id.

    Greetings from Rottenburg,
    Rainer
     
    Rainer Grimm, Dec 6, 2011
    #14
  15. On Tuesday, December 6, 2011 2:42:35 PM UTC+8, Rainer Grimm wrote:
    > Hello,
    >
    > > try:
    > > songs = [Song(id) for id in song_ids]
    > > except Song.DoesNotExist:
    > > print "unknown song id (%d)" % id

    > that's is a bad programming style. So it will be forbidden with python 3.The reason is that list comprehension is a construct from the functional world. It's only syntactic sugar for the functions map and filter. So functions have to be pure functions. To say it in other words, they have to be side-effect free. But the python construct from above pollutes the namespace with name id.
    >
    > Greetings from Rottenburg,
    > Rainer


    The list might have to grow in a careless way that might lead to a crash
    in the for inside a list that can't be trapped for errors directly.
     
    88888 Dihedral, Dec 6, 2011
    #15
  16. Steven D'Aprano wrote:
    > On Mon, 05 Dec 2011 19:57:15 +0100, Jean-Michel Pichavant wrote:
    >
    >
    >> The proper way to propagate information with exceptions is using the
    >> exception itself:
    >>
    >> try:
    >> songs = [Song(_id) for _id in song_ids]
    >> except Song.DoesNotExist, exc:
    >> print exc
    >>

    >
    >
    > I'm not entirely sure that this is the proper way to propagate the
    > exception. I see far to many people catching exceptions to print them, or
    > worse, to print a generic, useless message like "an error occurred".
    >

    [snip]

    You misread me, I was referering to passing *information* with exception
    (in other words, use the exception attributes). In the example I gave,
    the exception has the songId value responsible for raising the error.
    I totaly second your opinion on how poor the above handler is (hmm not
    sure about this grammar construct, it sounds like a Yoda sentence).

    JM
     
    Jean-Michel Pichavant, Dec 6, 2011
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. seguso
    Replies:
    9
    Views:
    400
    seguso
    Dec 22, 2004
  2. Dave Kuhlman

    Re: Style in list comprehensions

    Dave Kuhlman, Aug 15, 2003, in forum: Python
    Replies:
    1
    Views:
    341
    Alex Martelli
    Aug 16, 2003
  3. Frank Millman

    Simple db using list comprehensions

    Frank Millman, Apr 5, 2004, in forum: Python
    Replies:
    1
    Views:
    299
    Paddy McCarthy
    Apr 16, 2004
  4. Steven Bethard
    Replies:
    7
    Views:
    408
    Rocco Moretti
    Jan 20, 2006
  5. David Filmer
    Replies:
    19
    Views:
    267
    Kevin Collins
    May 21, 2004
Loading...

Share This Page