Can't get exclusive file lock when safely renaming a file

Discussion in 'Python' started by Steven D'Aprano, Dec 6, 2008.

  1. I'm trying to safely rename a file without over-writing any existing
    files, and I've run into a problem with file locks. Here's a naive way of
    renaming without over-writing:

    import os
    oldname = "ham.txt"
    newname = "spam.txt"
    if not os.path.exists(newname):
    os.rename(oldname, newname)

    It's naive because there's a race-condition: if file newname is created
    by another process after the call to exists(), but before the call to
    rename(), then it will be over-written.

    Here's my current solution, based on advice given by people on this
    thread:
    http://mail.python.org/pipermail/python-list/2006-October/411432.html

    and this recipe:
    http://code.activestate.com/recipes/65203/

    but it isn't working for me.


    import os, fcntl
    oldname = "ham.txt"
    newname = "spam.txt"

    def lock_destination(name):
    fileno = os.open(name, os.O_CREAT | os.O_EXCL)
    fcntl.flock(fileno, fcntl.LOCK_EX) # POSIX systems only
    return fileno

    # Create a test file to be renamed.
    f = open(oldname, 'w')
    f.write('this is my file\n')
    f.close()
    fileno = lock_destination(newname)

    # At this point, I can see "ham.txt" plus an empty file
    # "spam.txt" in my file browser

    os.rename(oldname, newname)



    The rename works, but here is my problem: after getting what I thought
    was an exclusive lock on the new file, but before calling os.rename(), I
    can still over-write it from another process:

    $ echo "this comes from another process" > spam.txt
    $ cat spam.txt
    this comes from another process


    This is despite running lock_destination("spam.txt"). What am I doing
    wrong?

    Before anyone asks, yes I have checked that the Python process and the
    shell process are in the same working directory, and therefore are
    writing to the same file:

    >>> os.read(fileno, 200)

    'this comes from another process\n'


    I'm using Linux (Fedora).



    --
    Steven
    Steven D'Aprano, Dec 6, 2008
    #1
    1. Advertising

  2. Steven D'Aprano wrote:
    <cut>
    > import os, fcntl
    > oldname = "ham.txt"
    > newname = "spam.txt"
    >
    > def lock_destination(name):
    > fileno = os.open(name, os.O_CREAT | os.O_EXCL)
    > fcntl.flock(fileno, fcntl.LOCK_EX) # POSIX systems only
    > return fileno
    >
    > # Create a test file to be renamed.
    > f = open(oldname, 'w')
    > f.write('this is my file\n')
    > f.close()
    > fileno = lock_destination(newname)
    >
    > # At this point, I can see "ham.txt" plus an empty file
    > # "spam.txt" in my file browser
    >
    > os.rename(oldname, newname)
    >
    > The rename works, but here is my problem: after getting what I thought
    > was an exclusive lock on the new file, but before calling os.rename(), I
    > can still over-write it from another process:
    >
    > $ echo "this comes from another process" > spam.txt
    > $ cat spam.txt
    > this comes from another process
    >

    <cut>
    This is weird, you would indeed expect it to be locked, I tried it on
    FreeBSD and it behaves the same.
    I guess the behavior of os.rename should be changed so it raises an
    exception when the source destination exists independent on the below
    platform.

    I tried searching for another way to do it but the closest I came was to
    popen mv and raise an exception if it asks to overwrite the existing file.

    Sorry I can't be of more help.
    --
    mph
    Martin P. Hellwig, Dec 6, 2008
    #2
    1. Advertising

  3. Steven D'Aprano

    Steve M Guest

    On Dec 6, 12:25 am, Steven D'Aprano <st...@REMOVE-THIS-
    cybersource.com.au> wrote:

    > The rename works, but here is my problem: after getting what I thought
    > was an exclusive lock on the new file, but before calling os.rename(), I
    > can still over-write it from another process:
    >
    > $ echo "this comes from another process" > spam.txt
    > $ cat spam.txt
    > this comes from another process


    What happens if you try to delete spam.txt at this point instead of
    over-write it?
    Steve M, Dec 6, 2008
    #3
  4. Steven D'Aprano wrote:
    > I'm trying to safely rename a file without over-writing any existing
    > files, and I've run into a problem with file locks. Here's a naive way of
    > renaming without over-writing

    By default on a Linux filesystem, flock() gives you an _advisory_ lock.
    Other processes can touch the file unless they explicitly check the lock.

    Try

    man mount (see under the -o mand)

    and

    man fcntl (see under file locking)

    Jeffrey Straszheim
    Jeffrey Straszheim, Dec 6, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steven D'Aprano

    Safely renaming a file without overwriting

    Steven D'Aprano, Oct 28, 2006, in forum: Python
    Replies:
    11
    Views:
    451
    Wolfgang Draxinger
    Oct 29, 2006
  2. Bruintje Beer

    Exclusive file lock

    Bruintje Beer, Dec 21, 2007, in forum: Java
    Replies:
    6
    Views:
    472
  3. Eugene Perederey

    Setting an exclusive lock on a file

    Eugene Perederey, Mar 29, 2009, in forum: Python
    Replies:
    2
    Views:
    264
    Diez B. Roggisch
    Mar 29, 2009
  4. Sea&Gull

    Exclusive lock of a file

    Sea&Gull, Jan 11, 2005, in forum: Ruby
    Replies:
    2
    Views:
    121
    Bill Atkins
    Jan 11, 2005
  5. lokesh mavale
    Replies:
    1
    Views:
    67
Loading...

Share This Page