PEP ? os.listdir enhancement

[Riccardo Galli]

Hi,
I noticed that when I use os.listdir I need to work with absolute paths
90% of times.
While I can use a for cycle, I'd prefere to use a list comprehension,
but it becomes too long.

I propose to add an 'abs' keyword which would make os.listdir return the
absolute path of files instead of a relative path.
This would bring only advantages, I think.

I show two examples

from os import listdir
from os.path import isdir,join,getsize,isfile

### e.g. 1 part 1 - getting a list of directories ###
dirs=[]
for i in os.listdir(path):
tmp_path=os.path.join(path,i)
if os.path.isdir(tmp_path):
dirs.append(tmp_path)

### e.g. 1 part 2 ###
dirs=[join(path,x) for x in listdir(path) if isdir(join(path,x))]

here the list comprehension is still clear, but only because we have
direct references to join and friends. moreover whe need to use join twice
for each directory.

### e.g. 2 part 1 - getting a list of (file,size) tuples ###
path_size=[]
for i in os.listdir(path):
tmp_path=os.path.join(path,i)
if os.path.isfile(tmp_path):
path_size.append((tmp_path,getsize(tmp_path))

### e.g. 2 part 2 ###
dirs=[(join(path,x),getsize(join(path,x)) for x in listdir(path) if
isfile(join(path,x))]


now list comprehension is way too long, and in the worst case we must use
join 3 times for each iteration.

adding an 'abs' keyword to os.listdir would give benefits both to for
cycle and list comprehensions.
for cycle would lose the tmp_path assignment and list comprehensions ...


### e.g. 1 part 2 bis ###
dirs=[x for x in listdir(path,abs=True) if isdir(x)]

here we gain clearness and speed.

### e.g. 2 part 2 bis ###
dirs=[(x,getsize(x)) for x in listdir(path,abs=True) if isfile(x)]

and here we gain clearness, speed and a truely _usable_ list comprehension

What do you think about this ?

Thanks for reading,
Riccardo

 

[Jeff Epler]

Why not just define the function yourself? Not every 3-line function
needs to be built in.

def listdir_joined(path):
return [os.path.join(path, entry) for entry in os.listdir(path)]

dirs = [x for x in listdir_joined(path) if os.path.isdir(x)]

path_size = [(x, getsize(x)) for x in listdir_joined(path) if os.path.isfile(x)]


Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFCuZFYJd01MZaTXX0RAqz2AKCozk1iJjrTwFRv4up5NdeVXqUNUQCeKniG
Q4Sl42Sp6dMnMFL0u607gwI=
=QEy2
-----END PGP SIGNATURE-----

 

[Peter Hansen]

I noticed that when I use os.listdir I need to work with absolute paths
90% of times.
While I can use a for cycle, I'd prefere to use a list comprehension,
but it becomes too long.

### e.g. 1 part 1 - getting a list of directories ###
dirs=[]
for i in os.listdir(path):
tmp_path=os.path.join(path,i)
if os.path.isdir(tmp_path):
dirs.append(tmp_path)

### e.g. 1 part 2 ###
dirs=[join(path,x) for x in listdir(path) if isdir(join(path,x))]

Using Jason Orendorff's "path" module, all this code basically collapses
down to this beauty (with your variable "path" renamed to myPath to
avoid a name collision):

from path import path
dirs = path(myPath).abspath().dirs()

-Peter

 

[Michael Hoffman]

Using Jason Orendorff's "path" module, all this code basically collapses
down to this beauty (with your variable "path" renamed to myPath to
avoid a name collision):

This has to be the non-stdlib library I use the most. It's a great module.

 

[Riccardo Galli]

Why not just define the function yourself? Not every 3-line function
needs to be built in.

Of course I can code such a function, and I agree with the second
sentence, but I think that obtaining absolutes path is a task so commonly
needed that adding a keyword to an existing function would give a plus to
the library without adding complexity (speaking of number of modules).

Usually when you use os.listdir do you end using os.path.join to obtain
absolutes path? I'm interested to know if the task is so common as I
think, or if I'm wrong.

Thank you,
Riccardo

 

[Andreas Kostyrka]

What's wrong with

(os.path.join(d, x) for x in os.listdir(d))

It's short, and easier to understand then some obscure option

Andreas

 

[Konstantin Veretennicov]

I propose to add an 'abs' keyword which would make os.listdir return the
absolute path of files instead of a relative path.

What about os.listdir(dir='relative/path', abs=True)? Should listdir
call abspath on results? Should we add another keyword rel? Would it
complicate listdir unnecessarily?

- kv

 

[Riccardo Galli]

What's wrong with

(os.path.join(d, x) for x in os.listdir(d))

It's short, and easier to understand then some obscure option

Andreas

how does it help in using list comprehension, as the ones in the first
post?

 

[Riccardo Galli]

What about os.listdir(dir='relative/path', abs=True)? Should listdir call
abspath on results? Should we add another keyword rel? Would it complicate
listdir unnecessarily?

- kv

keyword dir not exists (don't know if you added as example or not) and
abs set to true would return abspath on result. What else could it do ?

 

[Daniel Dittmar]

how does it help in using list comprehension, as the ones in the first
post?

You can nest list comprehension
[ e
for e in (os.path.join(d, x) for x in os.listdir(d))
if os.path.isdir (e)]

You might also want to look at module itertools, which has better
support for transforming and filtering in multiple steps.

Daniel

 

[Daniel Dittmar]

keyword dir not exists (don't know if you added as example or not) and
abs set to true would return abspath on result. What else could it do ?

He probably meant that a 'join' option would be more natural than an
'abs' option. After all, your examples use os.path.join to create a
valid path that can be used as the argument to other module os
functions. Whether the results are absolute or relative should depend on
the initial argument to os.listdir.

Daniel

 

[Peter Otten]

What about os.listdir(dir='relative/path', abs=True)? Should listdir
call abspath on results? Should we add another keyword rel? Would it
complicate listdir unnecessarily?

- kv

I'm in favour of Riccardo's suggestion, but I think he's got the name of the
keyword wrong. The signature should be

listdir(path, with_path=False)

and cater absolute and relative paths alike.

The need for such an enhancement is not particularly pressing, as
workarounds abound. Here's another one:
['/usr/lib/games/../games/schwarzerpeter']


Peter

 

[Riccardo Galli]

He probably meant that a 'join' option would be more natural than an 'abs'
option. After all, your examples use os.path.join to create a valid path
that can be used as the argument to other module os functions. Whether the
results are absolute or relative should depend on the initial argument to
os.listdir.

Daniel

I got the point, and you're right. I didn't though about that and 'abs' as
keyword becomes nonsense. Needing a more general kewyword, as pointed out
by Peter Otten

 

[Thomas Guettler]

Am Wed, 22 Jun 2005 17:57:14 +0200 schrieb Riccardo Galli:

Hi,
I noticed that when I use os.listdir I need to work with absolute paths
90% of times.
While I can use a for cycle, I'd prefere to use a list comprehension,
but it becomes too long.

Hi,

I like it. But as you noticed, too, "join" would be better than "abs".

Example:

# mylistdir.py
import os
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
yield os.path.join(dir, file)


print list(mylistdir(sys.argv[1]))


Thomas

 

[Ivan Van Laningham]

Hi All--

I like it. But as you noticed, too, "join" would be better than "abs".

Example:

# mylistdir.py
import os
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
yield os.path.join(dir, file)

print list(mylistdir(sys.argv[1]))

Mmmm, how about:

# mylistdir.py
import os, os.path
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
if join:
yield join(dir, file)
else:
yield file

print list(mylistdir(sys.argv[1]))

or

print list(mylistdir(sys.argv[1],os.path.join))

That way I could def my own join and call it as

print list(mylistdir(sys.argv[1],myjoin))

(Note that in your version the join argument isn't used at all.)

Metta,
Ivan
----------------------------------------------
Ivan Van Laningham
God N Locomotive Works
http://www.andi-holmes.com/
http://www.foretec.com/python/workshops/1998-11/proceedings.html
Army Signal Corps: Cu Chi, Class of '70
Author: Teach Yourself Python in 24 Hours

 

[Riccardo Galli]

Mmmm, how about:

# mylistdir.py
import os, os.path
import sys

def mylistdir(dir, join=False):
for file in os.listdir(dir):
if join:
yield join(dir, file)
else:
yield file

print list(mylistdir(sys.argv[1]))

or

print list(mylistdir(sys.argv[1],os.path.join))

That way I could def my own join and call it as

print list(mylistdir(sys.argv[1],myjoin))

I think that the implementation of listdir is done in C, so the
functionality would be added in C too.
by the way, I think 'join' (cute keyword) should be a boolean and not a
callable: the obvious way is that we join path using os.path.join, and not
some sophisticated function. With a boolean keyword code is smaller and if
we want to use our special join function we can do it simply.

e.g
def func(dir,join=False):
return (join and join(dir,x) or x for x in os.listdir(dir))

os.listdir is actually supposed not to be a generator, like you suggested.
Are there known future changes ?

Bye,
Riccardo