PEP on path module for standard library

[Michael Hoffman]

Many of you are familiar with Jason Orendorff's path module
<http://www.jorendorff.com/articles/python/path/>, which is frequently
recommended here on c.l.p. I submitted an RFE to add it to the Python
standard library, and Reinhold Birkenfeld started a discussion on it in
python-dev
<http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.

The upshot of the discussion was that many python-dev'ers wanted path
added to the stdlib, but Guido was not convinced and said it must have a
PEP. So Reinhold and I are going to work on one. Reinhold has already
made some changes to the module to fit the python-dev discussion and put
it in CPython CVS at nondist/sandbox/path.

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.

I use path in more of my modules and scripts than any other third-party
module, and I know it will be very helpful when I no longer have to
worry about deploying it.

Thanks in advance,

 

[Peter Hansen]

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.

I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.

"Easier" or "more difficult" is a subjective thing, of course, but one
can't argue with the fact that path can sometimes do through a single
object what would otherwise require several imports and a bunch of calls
into things like open(), os.path, grep, and shutil.

Examples showing effective uses of path that simplify those cases would
probably merit the label "easier" even in Guido's mind, though
unfortunately that's not certain. "Easier" in some minds might simply
translate to "many lines less code", and while path can sometimes do
that, aside from the ease of splitting and joining stuff without
multiple calls to os.path.this-and-that, it really doesn't often reduce
code size _that_ much, in my experience. (Postings to c.l.p showing a
50% reduction in code size for contrived examples notwithstanding.)

A related thoughts: since paths are objects, they have attributes or
properties, and having things like ".basename" and ".parent" readily
available without having to do obscure things like
os.path.split(somepath)[0] makes things much easier to read (therefore
more maintainable). In fact, I'd propose that as another strong
argument in path's favour: it makes code much more readable, even if not
"easier" to write.

Hmm... does "easier" or "more difficult" apply to the writing of the
code or the reading of it? I find it self-evident that code written
using "path" is much easier to read, not necessarily much easier to
write (for non-newbies).

I'd summarize this by saying that the integration of "path" in the
stdlib would make it easier for newbies to write code (that might not be
obvious to a non-newbie... shall we ask some to help?), and easier for
everyone to read code (self-evident, no?), and if that's not a
sufficient condition for inclusion I don't know what is.

-Peter

 

[Fernando Perez]

I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.

+10

One of the few things that annoys me about the stdlib is what one could call
performing 'shell-scripting-like' tasks, and precisely because of the problem
you point out. A number of conceptually related and common tasks are
scattered all over, and every time I need to write this kind of code, I find
myself paging over the docs for multiple modules, with no real intuition as to
where I could even guess where to find things. This is very unusual for
python, where in most cases things are so well organized, that blind guessing
tends to work remarkably well.

Personally I like the path module _a lot_, though I'm sure a thorough once-over
from c.l.py and python-dev, via a PEP, can only make it better and smooth out
hidden rough edges and corner cases. But I'll be very happy if it does go
into the stdlib in the future.

Just my .02.

Best,

f

 

[mkent]

I really love Jason's 'path' module. Sadly, I've encountered a serious
problem with using it. When you try to 'freeze' an application module,
and Jason's 'path' module is present in any of the directories that are
looked at by freeze's module finder (your app doesn't have to import
it), freeze goes into an infinite loop of imports, eventually getting a
'maximum recursion depth' exception. This seems to be related to
freeze getting confused between 'os.path' and Jason's 'path'.

I encountered this using Jason's latest 'path' module and Python 2.3.2.
I was able to solve it for my use by renaming path.py to newpath.py
and using 'from newpath import path' in my modules.

I've just notified Jason about this. I presume a solution like mine
will be used, and look forward to seeing Jason's module in stdlib.

 

[Reinhold Birkenfeld]

FYI: I modified the path module a bit so that it fits many of the suggestions
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.

Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.

I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.

"Easier" or "more difficult" is a subjective thing, of course, but one
can't argue with the fact that path can sometimes do through a single
object what would otherwise require several imports and a bunch of calls
into things like open(), os.path, grep, and shutil.
Correct.

Examples showing effective uses of path that simplify those cases would
probably merit the label "easier" even in Guido's mind, though
unfortunately that's not certain. "Easier" in some minds might simply
translate to "many lines less code", and while path can sometimes do
that, aside from the ease of splitting and joining stuff without
multiple calls to os.path.this-and-that, it really doesn't often reduce
code size _that_ much, in my experience. (Postings to c.l.p showing a
50% reduction in code size for contrived examples notwithstanding.)

Well, these examples are the ones we'd like to see here. So, people:
If you posted examples to c.l.py in the past, please try to collect
them here!

A related thoughts: since paths are objects, they have attributes or
properties, and having things like ".basename" and ".parent" readily
available without having to do obscure things like
os.path.split(somepath)[0] makes things much easier to read (therefore
more maintainable). In fact, I'd propose that as another strong
argument in path's favour: it makes code much more readable, even if not
"easier" to write.

Hmm... does "easier" or "more difficult" apply to the writing of the
code or the reading of it? I find it self-evident that code written
using "path" is much easier to read, not necessarily much easier to
write (for non-newbies).

And it is much more "Pythonic" in my eyes. Though that word may be inaccurate
when it comes from someone else that Guido, I feel that endless chains of
'[os.path.join(os.path.join(z, "a"), x) for x in os.path.listdir(os.path.join(z, "a") if os.path.isfile(os.path.join(....
are not qualified as being Pythonic.

I'd summarize this by saying that the integration of "path" in the
stdlib would make it easier for newbies to write code (that might not be
obvious to a non-newbie... shall we ask some to help?), and easier for
everyone to read code (self-evident, no?), and if that's not a
sufficient condition for inclusion I don't know what is.

Reinhold

 

[Terry Reedy]

Most prominent change is that it doesn't inherit from str/unicode
anymore.
I found this distinction important, because as a str subclass the Path
object
has many methods that don't make sense for it.

While I am - on adding more to learn, I am + on collecting scattered
filesystem functions into methods of coherent classes for eventually
replacement of the former. And I understand that this will mean a period
of duplication. But it is important to get the replacement right.

My only knowledge of the path module is what has been posted. However, it
seems to me that a path is conceptually a sequence of strings, rather than
the single joined-string representation thereof. If so, then of course it
is not and should not be a subclass of single strings. But that Path was
so defined makes me wonder, in my ignorance, whether the current
implementation is the best we can do for the future.

One advantage, for instance, of a split-up list implementation is that a
set of paths with a common prefix could be represented by replacing the
last string with a set of strings.

A more OO-friendly OS than the dominant ones today would accept a path as a
list (sequence) instead of requiring that the list be joined (by an
artifactual character) just to be split back into a list again.

My thoughts anyway.

Terry J. Reedy

 

[Michael Hoffman]

I really love Jason's 'path' module. Sadly, I've encountered a serious
problem with using it. When you try to 'freeze' an application module,
and Jason's 'path' module is present in any of the directories that are
looked at by freeze's module finder (your app doesn't have to import
it), freeze goes into an infinite loop of imports, eventually getting a
'maximum recursion depth' exception. This seems to be related to
freeze getting confused between 'os.path' and Jason's 'path'.

I encountered this using Jason's latest 'path' module and Python 2.3.2.
I was able to solve it for my use by renaming path.py to newpath.py
and using 'from newpath import path' in my modules.

I've just notified Jason about this. I presume a solution like mine
will be used, and look forward to seeing Jason's module in stdlib.

This sounds like a bug in "freeze" rather than something that should be
worked around in the standard library. Although there have already been
people opposed to naming it path because the duplication with os.path
might confuse humans.

 

[Michael Hoffman]

FYI: I modified the path module a bit so that it fits many of the suggestions
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.

Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.

Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.

What do you gain from removing these methods? A smaller dir()?

[1] Probably str in actuality.

 

[Reinhold Birkenfeld]

FYI: I modified the path module a bit so that it fits many of the suggestions
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.

Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.

Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.

Where do you see that pattern? IIRC it's not in the stdlib.

What do you gain from removing these methods? A smaller dir()?

It made sense to me at the time I changed this, although at the moment
I can't exactly recall the reasons.

Probably as Terry said: a path is both a list and a string.

Reinhold

 

[Reinhold Birkenfeld]

This sounds like a bug in "freeze" rather than something that should be
worked around in the standard library. Although there have already been
people opposed to naming it path because the duplication with os.path
might confuse humans.

As the most likely placement will be a class named "Path" inside the "os.path"
module, that bug with freeze won't apply to the "stdlib version" of Path.

Reinhold

 

[John Roth]

Many of you are familiar with Jason Orendorff's path module
<http://www.jorendorff.com/articles/python/path/>, which is frequently
recommended here on c.l.p. I submitted an RFE to add it to the Python
standard library, and Reinhold Birkenfeld started a discussion on it in
python-dev
<http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.

The upshot of the discussion was that many python-dev'ers wanted path
added to the stdlib, but Guido was not convinced and said it must have a
PEP.

Why did Guido want a PEP? Is it because he likes the idea but
feels the feature set needs to be examined a bit more by the wider
community, or is it some other reason?

I'm all in favor of having something that gives an alternative to
the kludge of functions that are "just a thin wrapper on the C
standard library." Considering the known problems with the
C standard library and the fact that it's strictly procedural, that
statement doesn't fill me with confidence. Rather it creates a
mild sense of dread: nobody has thought out how to do those
functions in a useful oo manner.

Path looks useable to me. Do I think it's going to be the
last word? I sincerely hope not! The only way we're going
to find out where it really needs to go from here, though, is
to put it out and find out how the wider community uses
and abuses it.

John Roth

 

[Reinhold Birkenfeld]

Why did Guido want a PEP? Is it because he likes the idea but
feels the feature set needs to be examined a bit more by the wider
community, or is it some other reason?

He said,

"""
Whoa! Do we really need a completely different mechanism for doing the
same stuff we can already do? The path module seems mostly useful for
folks coming from Java who are used to the Java Path class. With the
massive duplication of functionality we should also consider what to
recommend for the future: will the old os.path module be deprecated,
or are we going to maintain both alternatives forever? (And what about
all the duplication with the os module itself, like the cwd()
constructor?) Remember TOOWTDI.
"""

Reinhold

 

[Michael Hoffman]

Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.

Where do you see that pattern? IIRC it's not in the stdlib.

I do not think it is a *good* pattern, but it is used in Biopython. Of
course, there they ARE using things like type("") so on a unicode
filesystem it would already break. I seem to recall seeing it elsewhere,
but I can't remember where.

If you remove the basestring superclass, then you remove the ability to
use path objects as a drop-in replacement for any path string right now.
You will either have to use str(pathobj) or carefully check that the
function/framework you are passing the path to does not use isinstance()
or any of the string methods that are now gone.

It made sense to me at the time I changed this, although at the moment
I can't exactly recall the reasons.

Probably as Terry said: a path is both a list and a string.

I can see the case for thinking of it in both of those ways. In the end
a path is a sequence object. But a sequence of what?

I have a path that looks like this:

r"c:\windows\system32:altstream\test.dir\myfile.txt.zip:altstream"

One way to divide this is solely based on path separators:

['c:', 'windows', 'system32:altstream', 'test.dir',
'myfile.txt.zip:altstream']

But then some of the elements of this sequence have more meaning than
just being strings. "c:" is certainly something different from
"windows." The file name and alternate data stream name of each element
could be represented as a tuple.

The extensions can also be dealt with as a sequence. I have dealt with
things like filename = "filename.x.y.z" and wanted to get "filename.x"
before. The current stdlib solution,
os.path.splitext(os.path.splitext(filename)[0])[0] is extremely clunky,
and I have long desired something better. (OK, using
filename.split(os.extsep) works a little better, but you get the idea.)

So if you start breaking the path into a sequence of bigger items than
single character, where does it stop? What is a good design for this?

 

[Michael Hoffman]

He said,

"""
Whoa! Do we really need a completely different mechanism for doing the
same stuff we can already do? The path module seems mostly useful for
folks coming from Java who are used to the Java Path class.
> """

What is this Java Path class? I have been STFWing and have found nothing
on it in the. Indeed if you search for "Java Path class" (with quotes)
almost half of the pages are this message from Guido.

Any Java hackers here want to tell us of the wonders of the Java Path
class? I would be interested in seeing how other OO languages deal with
paths.

 

[John Roth]

He said,

"""
Whoa! Do we really need a completely different mechanism for doing the
same stuff we can already do? The path module seems mostly useful for
folks coming from Java who are used to the Java Path class. With the
massive duplication of functionality we should also consider what to
recommend for the future: will the old os.path module be deprecated,
or are we going to maintain both alternatives forever? (And what about
all the duplication with the os module itself, like the cwd()
constructor?) Remember TOOWTDI.
"""

Read literally, this says (at least to me) "I don't want to fix it because
I don't think it's broke."

As far as the Java remark is concerned, I suspect that it's because
in Java there is no such thing as a function; everything is either a
method on an object or a static method on a class.

And as far as I'm concerned, it's broke. I could see getting rid of the
bits and pieces of procedural code in 3.0, but not sooner.

John Roth

 

[George Sakkis]

He said,

"""
Whoa! Do we really need a completely different mechanism for doing the
same stuff we can already do? The path module seems mostly useful for
folks coming from Java who are used to the Java Path class. With the
massive duplication of functionality we should also consider what to
recommend for the future: will the old os.path module be deprecated,
or are we going to maintain both alternatives forever? (And what about
all the duplication with the os module itself, like the cwd()
constructor?) Remember TOOWTDI.
"""

Duplication is a valid point for debate, so the PEP should definitely address it. IMO os.path and
most (if not all) other equivalent modules and functions should be deprecated, though still working
until 2.9 for backwards compatibility, and dropped for python 3K.

George

 

[George Sakkis]

Read literally, this says (at least to me) "I don't want to fix it because
I don't think it's broke."

Or rather "I prefer a single existing mediocre solution than two solutions (even if the second was
better)".

George

 

[Neil Hodgson]

Reinhold Birkenfeld:

And it is much more "Pythonic" in my eyes. Though that word may be inaccurate
when it comes from someone else that Guido, I feel that endless chains of
'[os.path.join(os.path.join(z, "a"), x) for x in os.path.listdir(os.path.join(z, "a") if os.path.isfile(os.path.join(....
are not qualified as being Pythonic.

I like Path but the above is trying too hard to be poor code.
os.path.join takes more than 2 arguments, so that should be
[os.path.join(z, "a", x) for x in os.path.listdir(os.path.join(z, "a")
if os.path.isfile(os.path.join(....

Neil

 

[Andrew Dalke]

Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.

Where do you see that pattern? IIRC it's not in the stdlib.

Here's the first place that comes to mind for me

xml.sax.saxutils

def prepare_input_source(source, base = ""):
"""This function takes an InputSource and an optional base URL and
returns a fully resolved InputSource object ready for reading."""

if type(source) in _StringTypes:
source = xmlreader.InputSource(source)
elif hasattr(source, "read"):
f = source
source = xmlreader.InputSource()
source.setByteStream(f)
if hasattr(f, "name"):
source.setSystemId(f.name)


and xml.dom.pulldom

def parse(stream_or_string, parser=None, bufsize=None):
if bufsize is None:
bufsize = default_bufsize
if type(stream_or_string) in _StringTypes:
stream = open(stream_or_string)
else:
stream = stream_or_string
if not parser:
parser = xml.sax.make_parser()
return DOMEventStream(stream, parser, bufsize)

Using the power of grep

aifc.py
def __init__(self, f):
if type(f) == type(''):
f = __builtin__.open(f, 'rb')
# else, assume it is an open file object already
self.initfp(f)

binhex.py
class HexBin:
def __init__(self, ifp):
if type(ifp) == type(''):
ifp = open(ifp)

imghdr.py
if type(file) == type(''):
f = open(file, 'rb')
h = f.read(32)
else:
location = file.tell()
h = file.read(32)
file.seek(location)
f = None

mimify.py
if type(infile) == type(''):
ifile = open(infile)
if type(outfile) == type('') and infile == outfile:
import os
d, f = os.path.split(infile)
os.rename(infile, os.path.join(d, ',' + f))
else:
ifile = infile

wave.py
def __init__(self, f):
self._i_opened_the_file = None
if type(f) == type(''):
f = __builtin__.open(f, 'rb')
self._i_opened_the_file = f
# else, assume it is an open file object already
self.initfp(f)


compiler/transformer.py:

if type(file) == type(''):
file = open(file)
return self.parsesuite(file.read())

plat-mac/applesingle.py
if type(input) == type(''):
input = open(input, 'rb')
# Should we also test for FSSpecs or FSRefs?
header = input.read(AS_HEADER_LENGTH)

site-packages/ZODB/ExportImport.py
if file is None: file=TemporaryFile()
elif type(file) is StringType: file=open(file,'w+b')


site-packages/numarray/ndarray.py
if type(file) == type(""):
name = 1
file = open(file, 'wb')


site-packages/kiva/imaging/GdImageFile.py
if type(fp) == type(""):
import __builtin__
filename = fp
fp = __builtin__.open(fp, "rb")
else:
filename = ""

site-packages/reportlab/graphics/renderPM.py
if type(image.path) is type(''):
im = _getImage().open(image.path).convert('RGB')
else:
im = image.path.convert('RGB')


site-packages/twisted/protocols/irc.py
def __init__(self, file):
if type(file) is types.StringType:
self.file = open(file, 'r')

(hmm, that last one looks buggy. It should
have a "else: self.file = file" afterwards.)


Used in the std. lib and used by many different
people. (I excluded the Biopython libraries
in this list, btw, because I may have influenced
the use of this sort of type check.)

Andrew
(e-mail address removed)

 

[Stefan Rank]

What is this Java Path class? I have been STFWing and have found nothing
on it in the. Indeed if you search for "Java Path class" (with quotes)
almost half of the pages are this message from Guido.

Any Java hackers here want to tell us of the wonders of the Java Path
class?

no such thing exists.

there is only the `File` class that incorporates a little bit of the
`path` functionality and some of the python built-in `file` functionality.

my little self would actually propose to make *path* a built-in type as
an intermediate between file and basestring/str/unicode (the latter is
probably needed).

*ducks*