copyfile avoiding overwrites and race conditions

Discussion in 'Python' started by manuelg@gmail.com, Jun 16, 2006.

  1. Guest

    Here is a code fragment, where I am trying to copy a file, avoiding
    overwrites and race conditions. The filename gets a '02','03','04' etc
    appended to the end if a file with that name already exists.

    I know the writing of the single space is overkill, but I am surprised
    I cannot find an example of this floating around the web and newsgroups

    my understanding of 'os.open' and 'os.fdopen' is minimal

    ## start fragment

    i = 0
    while True:

    if i >= 100: raise RanOutFileNamesError(fullpath)
    i += 1

    if i > 1:
    name0 = '%s_%02i%s' % (suggested_name, i, file_extension)
    else:
    assert i == 1
    name0 = '%s%s' % (suggested_name, file_extension)
    fullpath = os.path.join(path0, name0)

    # create dummy file, force overwrite

    try:
    fd = os.open(
    fullpath,
    os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    except OSError:
    continue

    # is this really necessary?
    file = os.fdopen(fd, "w")
    file.write(' ')
    file.close()

    shutil.copyfile(original_filepath, fullpath)

    return fullpath

    ## end fragment
    , Jun 16, 2006
    #1
    1. Advertising

  2. Larry Bates Guest

    wrote:
    > Here is a code fragment, where I am trying to copy a file, avoiding
    > overwrites and race conditions. The filename gets a '02','03','04' etc
    > appended to the end if a file with that name already exists.
    >
    > I know the writing of the single space is overkill, but I am surprised
    > I cannot find an example of this floating around the web and newsgroups
    >
    > my understanding of 'os.open' and 'os.fdopen' is minimal
    >
    > ## start fragment
    >
    > i = 0
    > while True:
    >
    > if i >= 100: raise RanOutFileNamesError(fullpath)
    > i += 1
    >
    > if i > 1:
    > name0 = '%s_%02i%s' % (suggested_name, i, file_extension)
    > else:
    > assert i == 1
    > name0 = '%s%s' % (suggested_name, file_extension)
    > fullpath = os.path.join(path0, name0)
    >
    > # create dummy file, force overwrite
    >
    > try:
    > fd = os.open(
    > fullpath,
    > os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    > except OSError:
    > continue
    >
    > # is this really necessary?
    > file = os.fdopen(fd, "w")
    > file.write(' ')
    > file.close()
    >
    > shutil.copyfile(original_filepath, fullpath)
    >
    > return fullpath
    >
    > ## end fragment
    >

    I guess my approach would be different. To eliminate any race
    conditions, I would keep a small text file that always contained
    the next filename that is to be written. Something like:

    nextfiletowrite=/path/filename006.dat

    I would try to get a lock on this file, read it, extract next
    filename, increment the counter portion of the filename,
    write it back out and unlock it. Now I have the name of the
    file to write that is unique to my instance and I can write it
    without worrying about other processes.

    Hope this helps.

    -Larry Bates
    Larry Bates, Jun 19, 2006
    #2
    1. Advertising

  3. Guest

    Larry Bates wrote:

    > I guess my approach would be different. To eliminate any race
    > conditions, I would keep a small text file that always contained
    > the next filename that is to be written. Something like:
    >
    > nextfiletowrite=/path/filename006.dat
    >
    > I would try to get a lock on this file, read it, extract next
    > filename, increment the counter portion of the filename,
    > write it back out and unlock it. Now I have the name of the
    > file to write that is unique to my instance and I can write it
    > without worrying about other processes.


    Yes, that would work as well

    but I get the feeling that

    1) most people don't write to an automatically incrementing filename
    2) the few who do don't sweat the race condition

    Thanks for your reply

    Manuel
    , Jun 19, 2006
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. The Weiss Family

    race conditions/pulse width

    The Weiss Family, Oct 16, 2004, in forum: VHDL
    Replies:
    6
    Views:
    720
    Jim Lewis
    Oct 19, 2004
  2. Taras_96
    Replies:
    7
    Views:
    6,651
    Taras_96
    Apr 5, 2005
  3. mars
    Replies:
    6
    Views:
    353
    Laurent Pointal
    Feb 7, 2007
  4. Larry Bates

    Drop folder and race conditions

    Larry Bates, Oct 9, 2007, in forum: Python
    Replies:
    1
    Views:
    280
    Steven D'Aprano
    Oct 9, 2007
  5. kj
    Replies:
    4
    Views:
    297
    webEater
    Dec 22, 2006
Loading...

Share This Page