loops -> list/generator comprehensions

Discussion in 'Python' started by jamesthiele.usenet@gmail.com, Feb 6, 2005.

  1. Guest

    I wrote this little piece of code to get a list of relative paths of
    all files in or below the current directory (*NIX):

    walkList = [(x[0], x[2]) for x in os.walk(".")]
    filenames = []
    for dir, files in walkList:
    filenames.extend(["/".join([dir, f]) for f in files])

    It works fine, I don't need to change it, but I know there is a one
    liner list/generator comprehension to do this - I'm just not well
    enough versed in comprehensions to figure it out. Can someone please
    show me what it is?

    Even better, is there a generalized way to transform simple loops into
    comprehensions that someone can point me to?

    james
     
    , Feb 6, 2005
    #1
    1. Advertising

  2. wrote:
    > I wrote this little piece of code to get a list of relative paths of
    > all files in or below the current directory (*NIX):
    >
    > walkList = [(x[0], x[2]) for x in os.walk(".")]
    > filenames = []
    > for dir, files in walkList:
    > filenames.extend(["/".join([dir, f]) for f in files])
    >
    > It works fine, I don't need to change it, but I know there is a one
    > liner list/generator comprehension to do this - I'm just not well
    > enough versed in comprehensions to figure it out. Can someone please
    > show me what it is?


    I've used os.path.join instead of "/".join since it's more general, but
    other than that, this should be eqivalent:

    filenames = [os.path.join(dirpath, filename)
    for dirpath, _, filenames in os.walk('.')
    for filename in filenames]


    > Even better, is there a generalized way to transform simple loops into
    > comprehensions that someone can point me to?


    Well, generally, you need to write your loops to use an append, and then
    the translation to LC is simpler.

    filenames = []
    for dirpath, _, filenames in os.walk('.'):
    for filename in filenames:
    filenames.append(os.path.join(dirpath, filename))

    Now that you know what it looks like with an append, you simply move the
    expression in the append to the top, and leave the fors in the same order:

    filenames = [os.path.join(dirpath, filename)
    for dirpath, _, filenames in os.walk('.')
    for filename in filenames]

    HTH,

    STeVe
     
    Steven Bethard, Feb 6, 2005
    #2
    1. Advertising

  3. Guest

    > HTH,

    It does. Thanks.
     
    , Feb 6, 2005
    #3
  4. On 6 Feb 2005 11:28:37 -0800, <> wrote:
    >
    > walkList = [(x[0], x[2]) for x in os.walk(".")]
    > filenames = []
    > for dir, files in walkList:
    > filenames.extend(["/".join([dir, f]) for f in files])


    Caleb Hattingh top-posted:
    > I would be interested to see an example of code that is more concise
    > but yet as *clear* as the one you already have.


    Out of curiosity, do you find:

    filenames = [os.path.join(dirpath, filename)
    for dirpath, _, filenames in os.walk('.')
    for filename in filenames]

    harder to read?

    Steve
     
    Steven Bethard, Feb 6, 2005
    #4
  5. Caleb Hattingh wrote:
    >> filenames = [os.path.join(dirpath, filename)

    > # This is cool
    > for dirpath, _, filenames in os.walk('.')
    > # This is getting tricky, whats the '_' for?


    Nothing to do with the list comprehension really. '_' is a commonly
    used variable name for an object that we don't care about. If you look
    at the OP's original code, the line:

    [(x[0], x[2]) for x in os.walk(".")]

    is the equivalent of:

    [dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]

    I prefer to give names to the values produced by os.walk -- I think it
    makes the usage much clearer. However, since I don't use 'dirnames', I
    use '_' to indicate this:

    [dirpath, filenames for dirpath, _, filenames in os.walk('.')]

    Would

    filenames = [os.path.join(dirpath, filename)
    for dirpath, dirnames, filenames in os.walk('.')
    for filename in filenames]

    have been clearer for you? Then all you have to do is remember the
    order of the for-loop execution:

    > # Which thing goes where again in a comprehension?
    > for filename in filenames]


    As mentioned in my other post, the order is identical to that of
    for-loops, e.g.

    result = []
    for dirpath, _, filenames in os.walk('.'):
    for filename in filenames:
    result.append(os.path.join(dirpath, filename))

    is basically equivalent to:

    [os.path.join(dirpath, filename)
    for dirpath, _, filenames in os.walk('.')
    for filename in filenames]

    Does that help?

    Steve
     
    Steven Bethard, Feb 6, 2005
    #5
  6. Steven Bethard <> wrote:

    > at the OP's original code, the line:
    >
    > [(x[0], x[2]) for x in os.walk(".")]
    >
    > is the equivalent of:
    >
    > [dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]


    Just a nit: you need parentheses in your second LC too, i.e.:

    [(dirpath, filenames) for dirpath, dirnames, filenames in os.walk('.')]

    > I prefer to give names to the values produced by os.walk -- I think it
    > makes the usage much clearer.


    Full agreement here -- those numeric indices are nasty.


    Alex
     
    Alex Martelli, Feb 6, 2005
    #6
  7. Alex Martelli wrote:
    > Steven Bethard <> wrote:
    >
    >
    >>at the OP's original code, the line:
    >>
    >> [(x[0], x[2]) for x in os.walk(".")]
    >>
    >>is the equivalent of:
    >>
    >> [dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]

    >
    >
    > Just a nit: you need parentheses in your second LC too, i.e.:
    >
    > [(dirpath, filenames) for dirpath, dirnames, filenames in os.walk('.')]


    Yup, you're right (of course) ;) Thanks for the catch!

    Steve
     
    Steven Bethard, Feb 6, 2005
    #7
  8. I would be interested to see an example of code that is more concise but
    yet as *clear* as the one you already have. I can actually read and
    understand what you've got there. That's cool :)


    On 6 Feb 2005 11:28:37 -0800, <> wrote:

    > I wrote this little piece of code to get a list of relative paths of
    > all files in or below the current directory (*NIX):
    >
    > walkList = [(x[0], x[2]) for x in os.walk(".")]
    > filenames = []
    > for dir, files in walkList:
    > filenames.extend(["/".join([dir, f]) for f in files])
    >
    > It works fine, I don't need to change it, but I know there is a one
    > liner list/generator comprehension to do this - I'm just not well
    > enough versed in comprehensions to figure it out. Can someone please
    > show me what it is?
    >
    > Even better, is there a generalized way to transform simple loops into
    > comprehensions that someone can point me to?
    >
    > james
    >
     
    Caleb Hattingh, Feb 7, 2005
    #8
  9. Sure Steve

    Lemme see ... (indentation changed so comments associate with correct bits)

    > Out of curiosity, do you find:
    >
    > filenames = [os.path.join(dirpath, filename)

    # This is cool
    for dirpath, _, filenames in os.walk('.')
    # This is getting tricky, whats the '_' for? Which thing
    goes where again in a comprehension?
    for filename in filenames]
    # The hierarchy has nailed me by this point. I
    will have to start over...

    Yes, yes I do find it more difficult to read. Maybe I just don't know
    python as well as I should (certainly not as well as others here!). I
    guess it is just familiarity. A single comprehension is ok, nesting them
    gets tricky, and 3 times is a strike for me. I will practise :)

    keep well
    Caleb
     
    Caleb Hattingh, Feb 7, 2005
    #9
  10. Caleb Hattingh wrote:
    >> Would
    >>
    >> filenames = [os.path.join(dirpath, filename)
    >> for dirpath, dirnames, filenames in os.walk('.')
    >> for filename in filenames]
    >>
    >> have been clearer for you? Then all you have to do is remember the
    >> order of the for-loop execution:

    >
    > Bizarre as this may sound, it was the '_' that was throwing me off the
    > whole thing (at the 'grok' level I generally read the newsgroup,
    > anyway). For some weird reason, I can read *this* comprehension
    > pretty easily! Does that make sense at all? I figure a little bit of
    > uncertainty along the way probably derails understanding of the whole
    > thing a little bit


    Yup, actually, that's what I kinda suspected happened. But if it
    happened for you, it probably happened for a dozen other folks who read
    the same thing, so I'm glad to hear rewriting it and explaining the '_'
    was helpful.

    STeVe
     
    Steven Bethard, Feb 7, 2005
    #10
  11. Wow, Steve, thanks, you went to some effort here.

    > I prefer to give names to the values produced by os.walk -- I think it
    > makes the usage much clearer. However, since I don't use 'dirnames', I
    > use '_' to indicate this:


    Actually, I feel silly for not recognising this - I read about the Python3
    suggestion for adding a "with" syntax, and the suggestion was rather to
    use something like

    _ = instname
    _.a = 1
    _.b = 2

    So I actually have seen this "_" placeholder before :) Sorry bout that.

    > Would
    >
    > filenames = [os.path.join(dirpath, filename)
    > for dirpath, dirnames, filenames in os.walk('.')
    > for filename in filenames]
    >
    > have been clearer for you? Then all you have to do is remember the
    > order of the for-loop execution:


    Bizarre as this may sound, it was the '_' that was throwing me off the
    whole thing (at the 'grok' level I generally read the newsgroup,
    anyway). For some weird reason, I can read *this* comprehension pretty
    easily! Does that make sense at all? I figure a little bit of
    uncertainty along the way probably derails understanding of the whole
    thing a little bit - and (mental note) I *must* remember this when I
    explain stuff to people at work, having now experienced it first hand.

    Thanks again
    Caleb
     
    Caleb Hattingh, Feb 8, 2005
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jeff Epler
    Replies:
    2
    Views:
    337
    Jeff Epler
    Aug 27, 2003
  2. Mahesh Padmanabhan

    Generator expressions v/s list comprehensions

    Mahesh Padmanabhan, Aug 28, 2004, in forum: Python
    Replies:
    24
    Views:
    697
    Raymond Hettinger
    Sep 1, 2004
  3. Mike Meyer
    Replies:
    23
    Views:
    687
    Bengt Richter
    Apr 27, 2005
  4. Steven Bethard
    Replies:
    7
    Views:
    410
    Rocco Moretti
    Jan 20, 2006
  5. Ian Kelly
    Replies:
    7
    Views:
    376
    Terry Reedy
    May 25, 2010
Loading...

Share This Page