copyfile avoiding overwrites and race conditions

[manuelg]

Here is a code fragment, where I am trying to copy a file, avoiding
overwrites and race conditions. The filename gets a '02','03','04' etc
appended to the end if a file with that name already exists.

I know the writing of the single space is overkill, but I am surprised
I cannot find an example of this floating around the web and newsgroups

my understanding of 'os.open' and 'os.fdopen' is minimal

## start fragment

i = 0
while True:

if i >= 100: raise RanOutFileNamesError(fullpath)
i += 1

if i > 1:
name0 = '%s_%02i%s' % (suggested_name, i, file_extension)
else:
assert i == 1
name0 = '%s%s' % (suggested_name, file_extension)
fullpath = os.path.join(path0, name0)

# create dummy file, force overwrite

try:
fd = os.open(
fullpath,
os.O_CREAT | os.O_EXCL | os.O_WRONLY)
except OSError:
continue

# is this really necessary?
file = os.fdopen(fd, "w")
file.write(' ')
file.close()

shutil.copyfile(original_filepath, fullpath)

return fullpath

## end fragment

 

[Larry Bates]

Here is a code fragment, where I am trying to copy a file, avoiding
overwrites and race conditions. The filename gets a '02','03','04' etc
appended to the end if a file with that name already exists.

I know the writing of the single space is overkill, but I am surprised
I cannot find an example of this floating around the web and newsgroups

my understanding of 'os.open' and 'os.fdopen' is minimal

## start fragment

i = 0
while True:

if i >= 100: raise RanOutFileNamesError(fullpath)
i += 1

if i > 1:
name0 = '%s_%02i%s' % (suggested_name, i, file_extension)
else:
assert i == 1
name0 = '%s%s' % (suggested_name, file_extension)
fullpath = os.path.join(path0, name0)

# create dummy file, force overwrite

try:
fd = os.open(
fullpath,
os.O_CREAT | os.O_EXCL | os.O_WRONLY)
except OSError:
continue

# is this really necessary?
file = os.fdopen(fd, "w")
file.write(' ')
file.close()

shutil.copyfile(original_filepath, fullpath)

return fullpath

## end fragment

I guess my approach would be different. To eliminate any race
conditions, I would keep a small text file that always contained
the next filename that is to be written. Something like:

nextfiletowrite=/path/filename006.dat

I would try to get a lock on this file, read it, extract next
filename, increment the counter portion of the filename,
write it back out and unlock it. Now I have the name of the
file to write that is unique to my instance and I can write it
without worrying about other processes.

Hope this helps.

-Larry Bates

 

[manuelg]

I guess my approach would be different. To eliminate any race
conditions, I would keep a small text file that always contained
the next filename that is to be written. Something like:

nextfiletowrite=/path/filename006.dat

I would try to get a lock on this file, read it, extract next
filename, increment the counter portion of the filename,
write it back out and unlock it. Now I have the name of the
file to write that is unique to my instance and I can write it
without worrying about other processes.

Yes, that would work as well

but I get the feeling that

1) most people don't write to an automatically incrementing filename
2) the few who do don't sweat the race condition

Thanks for your reply

Manuel