Can This Code Be Made Faster?

[John Abel]

I have a problem, in that the following code is sometimes failing. I am
stat'ing a list of files, but, sometimes, one of the files may be moved
by an external process, after the os.path.exists and before the lstat,
causing the script to fail. My question, is there anything I can do to
speed it up, or put in a trap, to stop the stat from failing?

fileTimes = [ ( os.lstat( os.path.join( rootPath, fileName )
)[stat.ST_MTIME], fileName) for fileName in fileList \

if os.path.exists( os.path.join( rootPath, fileName ) ) ]

I'd appreciate any pointers.

John

 

[Ganesan R]

I have a problem, in that the following code is sometimes failing. I
am stat'ing a list of files, but, sometimes, one of the files may be
moved by an external process, after the os.path.exists and before the
lstat, causing the script to fail. My question, is there anything I
can do to speed it up, or put in a trap, to stop the stat from failing?
fileTimes = [ ( os.lstat( os.path.join( rootPath, fileName )
)[stat.ST_MTIME], fileName) for fileName in fileList \
if
os.path.exists( os.path.join( rootPath, fileName ) ) ]

Then there's no point in doing os.path.exists(), you might as well call
os.lstat directly. Some like this should work:

======
fileTimes = []
for fileName in fileList:
try:
fileTimes.append(
(os.lstat(os.path.join(rootPath, fileName)).st_mtime, fileName)
)
except:
pass

======

Ganesan

 

[Michael Peuser]

I have a problem, in that the following code is sometimes failing. I am
stat'ing a list of files, but, sometimes, one of the files may be moved
by an external process, after the os.path.exists and before the lstat,
causing the script to fail. My question, is there anything I can do to
speed it up, or put in a trap, to stop the stat from failing?

I think best help is with try - except (Ganesan already posted...)
But by all means NEVER use except: without any explicit exception; in this
case
except OSError:
would be appropriate.

If you by chance are using Windows only, a much better (and much faster)
suport will be given by
win32api.FindFiles
Of course this is not portable....

Kindly
Michael P

 

[Sean Ross]

[snip]
======
fileTimes = []
for fileName in fileList:
try:
fileTimes.append(
(os.lstat(os.path.join(rootPath, fileName)).st_mtime, fileName)
)
except:
pass

======

def getFileTimes(fileList, rootPath):
# using a function can speed things up (I seem to recall)
fileTimes = []
# use aliases to save some look-up time inside loop
append = fileTimes.append; lstat = os.lstat; joinPath = os.path.join;
for fileName in fileList:
try:
append((lstat(joinPath(rootPath, fileName)).st_mtime, fileName))
except OSError:
pass
return fileTimes


I don't know if function composition would help speed (you'd have to
timeit.py),
but it might help readability:

# from "Python Cookbook", pg. 467
def compose(f, g, *orig_args, **orig_kwds):
def nested_function(*more_args, **more_kwds):
return f(g(*more_args, **more_kwds), *orig_args, **orig_kwds)
return nested_function

def getFileTimes2(fileList, rootPath):
fileTimes = []
append = fileTimes.append
lstat = compose(os.lstat, os.path.join)
for fileName in fileList:
try:
append( (lstat(rootPath, fileName).st_mtime, fileName) )
except OSError:
pass
return fileTimes


HTH
Sean