Safely renaming a file without overwriting

Discussion in 'Python' started by Steven D'Aprano, Oct 28, 2006.

  1. I want to rename a file, but only if the destination file name doesn't
    already exist.

    I can do this:

    if os.path.exists(dest):
    # skip file, raise an exception, make a backup...
    do_something_else()
    else:
    os.rename(src, dest)


    But on a multi-user system, it is possible that dest is created in the
    time period between checking if it exists and attempting the rename.

    Is there any way to prevent this? Or do I just try to keep the check and
    the rename as close together as possible, minimizing the chances and
    hoping for the best?


    --
    Steven.
    Steven D'Aprano, Oct 28, 2006
    #1
    1. Advertising

  2. Steven D'Aprano wrote:

    > I want to rename a file, but only if the destination file name
    > doesn't already exist.
    >
    > I can do this:
    >
    > if os.path.exists(dest):
    > # skip file, raise an exception, make a backup...
    > do_something_else()
    > else:
    > os.rename(src, dest)
    >
    >
    > But on a multi-user system, it is possible that dest is created
    > in the time period between checking if it exists and attempting
    > the rename.
    >
    > Is there any way to prevent this? Or do I just try to keep the
    > check and the rename as close together as possible, minimizing
    > the chances and hoping for the best?


    1: Open the file with os.open

    2: Lock the file exclusively -> no other process can now access
    it.

    3: Use rename to rename the file; this causes a file system level
    implicit unlink of the old file (it dissappears from the file
    system) but the opening process can still access it.

    4: close the file -> the lock is removed and the rename
    finalized.

    Wolfgang Draxinger
    --
    E-Mail address works, Jabber: , ICQ: 134682867
    GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E
    Wolfgang Draxinger, Oct 28, 2006
    #2
    1. Advertising

  3. Wolfgang Draxinger schrieb:
    > Steven D'Aprano wrote:
    >
    >> I want to rename a file, but only if the destination file name
    >> doesn't already exist.
    >>
    >> I can do this:
    >>
    >> if os.path.exists(dest):
    >> # skip file, raise an exception, make a backup...
    >> do_something_else()
    >> else:
    >> os.rename(src, dest)
    >>
    >>
    >> But on a multi-user system, it is possible that dest is created
    >> in the time period between checking if it exists and attempting
    >> the rename.
    >>
    >> Is there any way to prevent this? Or do I just try to keep the
    >> check and the rename as close together as possible, minimizing
    >> the chances and hoping for the best?

    >
    > 1: Open the file with os.open
    >
    > 2: Lock the file exclusively -> no other process can now access
    > it.
    >
    > 3: Use rename to rename the file; this causes a file system level
    > implicit unlink of the old file (it dissappears from the file
    > system) but the opening process can still access it.
    >
    > 4: close the file -> the lock is removed and the rename
    > finalized.


    Where does that help for new files? The OP was right in assuming that a
    race condition could occur when he tests for a file & then tries to
    create it, as in the meantime it could have been created.

    Diez
    Diez B. Roggisch, Oct 28, 2006
    #3
  4. Diez B. Roggisch wrote:

    >> 1: Open the file with os.open
    >>
    >> 2: Lock the file exclusively -> no other process can now
    >> access it.
    >>
    >> 3: Use rename to rename the file; this causes a file system
    >> level implicit unlink of the old file (it dissappears from the
    >> file system) but the opening process can still access it.
    >>
    >> 4: close the file -> the lock is removed and the rename
    >> finalized.


    The open is to happen on the new file name with O_CREAT | O_EXCL
    flags. Sorry, I forgot that to mention explicitly. However I
    have not tried it yet, but it should work.

    Wolfgang Draxinger
    --
    E-Mail address works, Jabber: , ICQ: 134682867
    GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E
    Wolfgang Draxinger, Oct 28, 2006
    #4
  5. Steven D'Aprano

    Chetan Guest

    Steven D'Aprano <> writes:

    > I want to rename a file, but only if the destination file name doesn't
    > already exist.
    >
    > I can do this:
    >
    > if os.path.exists(dest):
    > # skip file, raise an exception, make a backup...
    > do_something_else()
    > else:
    > os.rename(src, dest)
    >
    >
    > But on a multi-user system, it is possible that dest is created in the
    > time period between checking if it exists and attempting the rename.
    >
    > Is there any way to prevent this? Or do I just try to keep the check and
    > the rename as close together as possible, minimizing the chances and
    > hoping for the best?
    >
    >
    > --
    > Steven.


    The answer, unfortunately, depends on the platform. I haven't tried, but it
    looks like rename() will fail on Win32 if the file already exists. On Unix, you
    can use link to rename the file - which will not overwrite the file if it
    exists. Then use unlink to remove the src file.

    Chetan
    Chetan, Oct 28, 2006
    #5
  6. On Sat, 28 Oct 2006 13:38:14 +0200, Wolfgang Draxinger wrote:

    >> But on a multi-user system, it is possible that dest is created
    >> in the time period between checking if it exists and attempting
    >> the rename.
    >>
    >> Is there any way to prevent this? Or do I just try to keep the
    >> check and the rename as close together as possible, minimizing
    >> the chances and hoping for the best?

    >
    > 1: Open the file with os.open


    Open "the" file? There are potentially two files -- the source and
    destination. I only want to do the rename if the destination
    *doesn't* exist, so there is no destination file to open. How will it help
    me to lock the source file? Have I misunderstood?


    --
    Steven.
    Steven D'Aprano, Oct 28, 2006
    #6
  7. On Sat, 28 Oct 2006 16:48:37 +0200, Diez B. Roggisch wrote:

    > Where does that help for new files? The OP was right in assuming that a
    > race condition could occur when he tests for a file & then tries to
    > create it, as in the meantime it could have been created.


    Ah! "Race condition" -- that was the term I was looking for ... now maybe
    I'll have some better results with Google.



    --
    Steven.
    Steven D'Aprano, Oct 28, 2006
    #7
  8. Steven D'Aprano wrote:

    > Open "the" file? There are potentially two files -- the source
    > and destination. I only want to do the rename if the
    > destination *doesn't* exist, so there is no destination file to
    > open. How will it help me to lock the source file? Have I
    > misunderstood?


    I forgot to say, to open the source file with O_CREAT | O_EXCL.
    The open will fail if the file already exists. By locking the
    file no other process will be able to access it, but the process
    that holds the lock can do with it anything. This includes to
    rename an existing file to the name of the opened one - the
    previously created placeholder file will get unlinked before,
    but there is _probably_ no way that any process can intercept
    this. It is an interesting thing, that files that are opened
    remain fully usable and accessible if you unlink them as long
    you don't close them. You have to close it, to remove all
    remains of it.

    Wolfgang Draxinger
    --
    E-Mail address works, Jabber: , ICQ: 134682867
    GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E
    Wolfgang Draxinger, Oct 28, 2006
    #8
  9. On Sun, 29 Oct 2006 00:29:06 +0200, Wolfgang Draxinger wrote:

    > Steven D'Aprano wrote:
    >
    >> Open "the" file? There are potentially two files -- the source
    >> and destination. I only want to do the rename if the
    >> destination *doesn't* exist, so there is no destination file to
    >> open. How will it help me to lock the source file? Have I
    >> misunderstood?

    >
    > I forgot to say, to open the source file with O_CREAT | O_EXCL.
    > The open will fail if the file already exists.


    But the source file always exists, otherwise there is nothing to rename!
    Do you mean, open the destination filename?



    --
    Steven.
    Steven D'Aprano, Oct 28, 2006
    #9
  10. Steven D'Aprano wrote:

    > But the source file always exists, otherwise there is nothing
    > to rename! Do you mean, open the destination filename?


    Of course I meant the destination file. Someone please spill some
    ice chilled water over me to get me awake again. Time to go to
    bed :p before I make more dumb mistakes/typos...

    Wolfgang Draxinger
    --
    E-Mail address works, Jabber: , ICQ: 134682867
    GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E
    Wolfgang Draxinger, Oct 29, 2006
    #10
  11. Wolfgang Draxinger schrieb:
    > Steven D'Aprano wrote:
    >
    >> But the source file always exists, otherwise there is nothing
    >> to rename! Do you mean, open the destination filename?

    >
    > Of course I meant the destination file. Someone please spill some
    > ice chilled water over me to get me awake again. Time to go to
    > bed :p before I make more dumb mistakes/typos...


    But that doesn't help. Opening the destination file that way will only
    make a difference if _both_ processes that fight over that filename work
    that way. Which might be possible in the actual case, but not as a
    general recipe.

    The link/unlink trick of Chetan sounds reasonable, though.
    Diez B. Roggisch, Oct 29, 2006
    #11
  12. Diez B. Roggisch wrote:

    > The link/unlink trick of Chetan sounds reasonable, though.


    It will work only if both source and destination are on the same
    file system, which means you can't move a file between mount
    points - at least that's the way how it's defined by POSIX; NTFS
    supports hard links and soft links, too but I don't know if
    those got the same constraints.

    Wolfgang Draxinger
    --
    E-Mail address works, Jabber: , ICQ: 134682867
    GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E
    Wolfgang Draxinger, Oct 29, 2006
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. javajavalink
    Replies:
    6
    Views:
    1,187
    Michael Borgwardt
    Dec 14, 2004
  2. Stewart
    Replies:
    8
    Views:
    777
    Stewart
    Nov 5, 2004
  3. dev648237923

    Deploy without overwriting Web.Config

    dev648237923, Jan 17, 2007, in forum: ASP .Net
    Replies:
    4
    Views:
    513
    Walter Wang [MSFT]
    Jan 23, 2007
  4. Steven D'Aprano
    Replies:
    3
    Views:
    1,791
    Jeffrey Straszheim
    Dec 6, 2008
  5. Kamarulnizam Rahim
    Replies:
    4
    Views:
    200
    Robert Klemme
    Jan 28, 2011
Loading...

Share This Page