loops -> list/generator comprehensions

[jamesthiele.usenet]

I wrote this little piece of code to get a list of relative paths of
all files in or below the current directory (*NIX):

walkList = [(x[0], x[2]) for x in os.walk(".")]
filenames = []
for dir, files in walkList:
filenames.extend(["/".join([dir, f]) for f in files])

It works fine, I don't need to change it, but I know there is a one
liner list/generator comprehension to do this - I'm just not well
enough versed in comprehensions to figure it out. Can someone please
show me what it is?

Even better, is there a generalized way to transform simple loops into
comprehensions that someone can point me to?

james

 

[Steven Bethard]

I wrote this little piece of code to get a list of relative paths of
all files in or below the current directory (*NIX):

walkList = [(x[0], x[2]) for x in os.walk(".")]
filenames = []
for dir, files in walkList:
filenames.extend(["/".join([dir, f]) for f in files])

It works fine, I don't need to change it, but I know there is a one
liner list/generator comprehension to do this - I'm just not well
enough versed in comprehensions to figure it out. Can someone please
show me what it is?

I've used os.path.join instead of "/".join since it's more general, but
other than that, this should be eqivalent:

filenames = [os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]

Even better, is there a generalized way to transform simple loops into
comprehensions that someone can point me to?

Well, generally, you need to write your loops to use an append, and then
the translation to LC is simpler.

filenames = []
for dirpath, _, filenames in os.walk('.'):
for filename in filenames:
filenames.append(os.path.join(dirpath, filename))

Now that you know what it looks like with an append, you simply move the
expression in the append to the top, and leave the fors in the same order:

filenames = [os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]

HTH,

STeVe

 

[jamesthiele.usenet]

HTH,

It does. Thanks.

 

[Steven Bethard]

>
> walkList = [(x[0], x[2]) for x in os.walk(".")]
> filenames = []
> for dir, files in walkList:
> filenames.extend(["/".join([dir, f]) for f in files])

Caleb Hattingh top-posted:

I would be interested to see an example of code that is more concise
but yet as *clear* as the one you already have.

Out of curiosity, do you find:

filenames = [os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]

harder to read?

Steve

 

[Steven Bethard]

filenames = [os.path.join(dirpath, filename)

# This is cool
for dirpath, _, filenames in os.walk('.')
# This is getting tricky, whats the '_' for?

Nothing to do with the list comprehension really. '_' is a commonly
used variable name for an object that we don't care about. If you look
at the OP's original code, the line:

[(x[0], x[2]) for x in os.walk(".")]

is the equivalent of:

[dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]

I prefer to give names to the values produced by os.walk -- I think it
makes the usage much clearer. However, since I don't use 'dirnames', I
use '_' to indicate this:

[dirpath, filenames for dirpath, _, filenames in os.walk('.')]

Would

filenames = [os.path.join(dirpath, filename)
for dirpath, dirnames, filenames in os.walk('.')
for filename in filenames]

have been clearer for you? Then all you have to do is remember the
order of the for-loop execution:

# Which thing goes where again in a comprehension?
for filename in filenames]

As mentioned in my other post, the order is identical to that of
for-loops, e.g.

result = []
for dirpath, _, filenames in os.walk('.'):
for filename in filenames:
result.append(os.path.join(dirpath, filename))

is basically equivalent to:

[os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]

Does that help?

Steve

 

[Alex Martelli]

at the OP's original code, the line:

[(x[0], x[2]) for x in os.walk(".")]

is the equivalent of:

[dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]

Just a nit: you need parentheses in your second LC too, i.e.:

[(dirpath, filenames) for dirpath, dirnames, filenames in os.walk('.')]

I prefer to give names to the values produced by os.walk -- I think it
makes the usage much clearer.

Full agreement here -- those numeric indices are nasty.


Alex

 

[Steven Bethard]

at the OP's original code, the line:

[(x[0], x[2]) for x in os.walk(".")]

is the equivalent of:

[dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]

Just a nit: you need parentheses in your second LC too, i.e.:

[(dirpath, filenames) for dirpath, dirnames, filenames in os.walk('.')]

Yup, you're right (of course) Thanks for the catch!

Steve

 

[Caleb Hattingh]

I would be interested to see an example of code that is more concise but
yet as *clear* as the one you already have. I can actually read and
understand what you've got there. That's cool

 

[Caleb Hattingh]

Sure Steve

Lemme see ... (indentation changed so comments associate with correct bits)

Out of curiosity, do you find:

filenames = [os.path.join(dirpath, filename)

# This is cool
for dirpath, _, filenames in os.walk('.')
# This is getting tricky, whats the '_' for? Which thing
goes where again in a comprehension?
for filename in filenames]
# The hierarchy has nailed me by this point. I
will have to start over...

Yes, yes I do find it more difficult to read. Maybe I just don't know
python as well as I should (certainly not as well as others here!). I
guess it is just familiarity. A single comprehension is ok, nesting them
gets tricky, and 3 times is a strike for me. I will practise

keep well
Caleb

 

[Steven Bethard]

Would

filenames = [os.path.join(dirpath, filename)
for dirpath, dirnames, filenames in os.walk('.')
for filename in filenames]

have been clearer for you? Then all you have to do is remember the
order of the for-loop execution:

Bizarre as this may sound, it was the '_' that was throwing me off the
whole thing (at the 'grok' level I generally read the newsgroup,
anyway). For some weird reason, I can read *this* comprehension
pretty easily! Does that make sense at all? I figure a little bit of
uncertainty along the way probably derails understanding of the whole
thing a little bit

Yup, actually, that's what I kinda suspected happened. But if it
happened for you, it probably happened for a dozen other folks who read
the same thing, so I'm glad to hear rewriting it and explaining the '_'
was helpful.

STeVe

 

[Caleb Hattingh]

Wow, Steve, thanks, you went to some effort here.

I prefer to give names to the values produced by os.walk -- I think it
makes the usage much clearer. However, since I don't use 'dirnames', I
use '_' to indicate this:

Actually, I feel silly for not recognising this - I read about the Python3
suggestion for adding a "with" syntax, and the suggestion was rather to
use something like

_ = instname
_.a = 1
_.b = 2

So I actually have seen this "_" placeholder before Sorry bout that.

Would

filenames = [os.path.join(dirpath, filename)
for dirpath, dirnames, filenames in os.walk('.')
for filename in filenames]

have been clearer for you? Then all you have to do is remember the
order of the for-loop execution:

Bizarre as this may sound, it was the '_' that was throwing me off the
whole thing (at the 'grok' level I generally read the newsgroup,
anyway). For some weird reason, I can read *this* comprehension pretty
easily! Does that make sense at all? I figure a little bit of
uncertainty along the way probably derails understanding of the whole
thing a little bit - and (mental note) I *must* remember this when I
explain stuff to people at work, having now experienced it first hand.

Thanks again
Caleb