Can't get exclusive file lock when safely renaming a file

[Steven D'Aprano]

I'm trying to safely rename a file without over-writing any existing
files, and I've run into a problem with file locks. Here's a naive way of
renaming without over-writing:

import os
oldname = "ham.txt"
newname = "spam.txt"
if not os.path.exists(newname):
os.rename(oldname, newname)

It's naive because there's a race-condition: if file newname is created
by another process after the call to exists(), but before the call to
rename(), then it will be over-written.

Here's my current solution, based on advice given by people on this
thread:
http://mail.python.org/pipermail/python-list/2006-October/411432.html

and this recipe:
http://code.activestate.com/recipes/65203/

but it isn't working for me.


import os, fcntl
oldname = "ham.txt"
newname = "spam.txt"

def lock_destination(name):
fileno = os.open(name, os.O_CREAT | os.O_EXCL)
fcntl.flock(fileno, fcntl.LOCK_EX) # POSIX systems only
return fileno

# Create a test file to be renamed.
f = open(oldname, 'w')
f.write('this is my file\n')
f.close()
fileno = lock_destination(newname)

# At this point, I can see "ham.txt" plus an empty file
# "spam.txt" in my file browser

os.rename(oldname, newname)



The rename works, but here is my problem: after getting what I thought
was an exclusive lock on the new file, but before calling os.rename(), I
can still over-write it from another process:

$ echo "this comes from another process" > spam.txt
$ cat spam.txt
this comes from another process


This is despite running lock_destination("spam.txt"). What am I doing
wrong?

Before anyone asks, yes I have checked that the Python process and the
shell process are in the same working directory, and therefore are
writing to the same file:
'this comes from another process\n'


I'm using Linux (Fedora).

 

[Martin P. Hellwig]

Steven D'Aprano wrote:

import os, fcntl
oldname = "ham.txt"
newname = "spam.txt"

def lock_destination(name):
fileno = os.open(name, os.O_CREAT | os.O_EXCL)
fcntl.flock(fileno, fcntl.LOCK_EX) # POSIX systems only
return fileno

# Create a test file to be renamed.
f = open(oldname, 'w')
f.write('this is my file\n')
f.close()
fileno = lock_destination(newname)

# At this point, I can see "ham.txt" plus an empty file
# "spam.txt" in my file browser

os.rename(oldname, newname)

The rename works, but here is my problem: after getting what I thought
was an exclusive lock on the new file, but before calling os.rename(), I
can still over-write it from another process:

$ echo "this comes from another process" > spam.txt
$ cat spam.txt
this comes from another process

<cut>
This is weird, you would indeed expect it to be locked, I tried it on
FreeBSD and it behaves the same.
I guess the behavior of os.rename should be changed so it raises an
exception when the source destination exists independent on the below
platform.

I tried searching for another way to do it but the closest I came was to
popen mv and raise an exception if it asks to overwrite the existing file.

Sorry I can't be of more help.

 

[Steve M]

The rename works, but here is my problem: after getting what I thought
was an exclusive lock on the new file, but before calling os.rename(), I
can still over-write it from another process:

$ echo "this comes from another process" > spam.txt
$ cat spam.txt
this comes from another process

What happens if you try to delete spam.txt at this point instead of
over-write it?

 

[Jeffrey Straszheim]

I'm trying to safely rename a file without over-writing any existing
files, and I've run into a problem with file locks. Here's a naive way of
renaming without over-writing

By default on a Linux filesystem, flock() gives you an _advisory_ lock.
Other processes can touch the file unless they explicitly check the lock.

Try

man mount (see under the -o mand)

and

man fcntl (see under file locking)

Jeffrey Straszheim