Safely renaming a file without overwriting

[Steven D'Aprano]

I want to rename a file, but only if the destination file name doesn't
already exist.

I can do this:

if os.path.exists(dest):
# skip file, raise an exception, make a backup...
do_something_else()
else:
os.rename(src, dest)


But on a multi-user system, it is possible that dest is created in the
time period between checking if it exists and attempting the rename.

Is there any way to prevent this? Or do I just try to keep the check and
the rename as close together as possible, minimizing the chances and
hoping for the best?

 

[Wolfgang Draxinger]

I want to rename a file, but only if the destination file name
doesn't already exist.

I can do this:

if os.path.exists(dest):
# skip file, raise an exception, make a backup...
do_something_else()
else:
os.rename(src, dest)


But on a multi-user system, it is possible that dest is created
in the time period between checking if it exists and attempting
the rename.

Is there any way to prevent this? Or do I just try to keep the
check and the rename as close together as possible, minimizing
the chances and hoping for the best?

1: Open the file with os.open

2: Lock the file exclusively -> no other process can now access
it.

3: Use rename to rename the file; this causes a file system level
implicit unlink of the old file (it dissappears from the file
system) but the opening process can still access it.

4: close the file -> the lock is removed and the rename
finalized.

Wolfgang Draxinger

 

[Diez B. Roggisch]

1: Open the file with os.open

2: Lock the file exclusively -> no other process can now access
it.

3: Use rename to rename the file; this causes a file system level
implicit unlink of the old file (it dissappears from the file
system) but the opening process can still access it.

4: close the file -> the lock is removed and the rename
finalized.

Where does that help for new files? The OP was right in assuming that a
race condition could occur when he tests for a file & then tries to
create it, as in the meantime it could have been created.

Diez

 

[Wolfgang Draxinger]

The open is to happen on the new file name with O_CREAT | O_EXCL
flags. Sorry, I forgot that to mention explicitly. However I
have not tried it yet, but it should work.

Wolfgang Draxinger

 

[Chetan]

I want to rename a file, but only if the destination file name doesn't
already exist.

I can do this:

if os.path.exists(dest):
# skip file, raise an exception, make a backup...
do_something_else()
else:
os.rename(src, dest)


But on a multi-user system, it is possible that dest is created in the
time period between checking if it exists and attempting the rename.

Is there any way to prevent this? Or do I just try to keep the check and
the rename as close together as possible, minimizing the chances and
hoping for the best?

The answer, unfortunately, depends on the platform. I haven't tried, but it
looks like rename() will fail on Win32 if the file already exists. On Unix, you
can use link to rename the file - which will not overwrite the file if it
exists. Then use unlink to remove the src file.

Chetan

 

[Steven D'Aprano]

1: Open the file with os.open

Open "the" file? There are potentially two files -- the source and
destination. I only want to do the rename if the destination
*doesn't* exist, so there is no destination file to open. How will it help
me to lock the source file? Have I misunderstood?

 

[Steven D'Aprano]

Where does that help for new files? The OP was right in assuming that a
race condition could occur when he tests for a file & then tries to
create it, as in the meantime it could have been created.

Ah! "Race condition" -- that was the term I was looking for ... now maybe
I'll have some better results with Google.

 

[Wolfgang Draxinger]

Open "the" file? There are potentially two files -- the source
and destination. I only want to do the rename if the
destination *doesn't* exist, so there is no destination file to
open. How will it help me to lock the source file? Have I
misunderstood?

I forgot to say, to open the source file with O_CREAT | O_EXCL.
The open will fail if the file already exists. By locking the
file no other process will be able to access it, but the process
that holds the lock can do with it anything. This includes to
rename an existing file to the name of the opened one - the
previously created placeholder file will get unlinked before,
but there is _probably_ no way that any process can intercept
this. It is an interesting thing, that files that are opened
remain fully usable and accessible if you unlink them as long
you don't close them. You have to close it, to remove all
remains of it.

Wolfgang Draxinger

 

[Steven D'Aprano]

I forgot to say, to open the source file with O_CREAT | O_EXCL.
The open will fail if the file already exists.

But the source file always exists, otherwise there is nothing to rename!
Do you mean, open the destination filename?

 

[Wolfgang Draxinger]

But the source file always exists, otherwise there is nothing
to rename! Do you mean, open the destination filename?

Of course I meant the destination file. Someone please spill some
ice chilled water over me to get me awake again. Time to go to
bed before I make more dumb mistakes/typos...

Wolfgang Draxinger

 

[Diez B. Roggisch]

Of course I meant the destination file. Someone please spill some
ice chilled water over me to get me awake again. Time to go to
bed before I make more dumb mistakes/typos...

But that doesn't help. Opening the destination file that way will only
make a difference if _both_ processes that fight over that filename work
that way. Which might be possible in the actual case, but not as a
general recipe.

The link/unlink trick of Chetan sounds reasonable, though.

 

[Wolfgang Draxinger]

The link/unlink trick of Chetan sounds reasonable, though.

It will work only if both source and destination are on the same
file system, which means you can't move a file between mount
points - at least that's the way how it's defined by POSIX; NTFS
supports hard links and soft links, too but I don't know if
those got the same constraints.

Wolfgang Draxinger