Drop folder and race conditions

Discussion in 'Python' started by Larry Bates, Oct 9, 2007.

  1. Larry Bates

    Larry Bates Guest

    I have a need to implement a drop folder upload mechanism for secure uploading
    of files to a server. At first glance this appears that it would be an easy
    application to write. Then I begin to think about the race conditions that
    exist between the process that will wake up to upload the files and the fact
    that the user can add additional files to the drop folder at any point in time.
    I would like to clear out files/folders after they have been processed but it is
    possible that while the background process was uploading files, the user added
    additional files/folders to the drop folder that make the deletion of folders
    impossible. This project has become significantly more complex than it appeared
    at first. Anyone out there have any "sage" advice on how to tackle this beast?

    Thanks in advance for any pointers.

    Regards,
    Larry Bates
     
    Larry Bates, Oct 9, 2007
    #1
    1. Advertising

  2. On Tue, 09 Oct 2007 07:05:57 -0500, Larry Bates wrote:

    > I have a need to implement a drop folder upload mechanism for secure
    > uploading of files to a server. At first glance this appears that it
    > would be an easy application to write. Then I begin to think about the
    > race conditions that exist between the process that will wake up to
    > upload the files and the fact that the user can add additional files to
    > the drop folder at any point in time. I would like to clear out
    > files/folders after they have been processed but it is possible that
    > while the background process was uploading files, the user added
    > additional files/folders to the drop folder that make the deletion of
    > folders impossible. This project has become significantly more complex
    > than it appeared at first. Anyone out there have any "sage" advice on
    > how to tackle this beast?



    Off the top of my head...

    Recursively copy the contents of each folder to the server, deepest
    first, deleting each file as it's copied. Then delete the folder as soon
    as you've emptied it (but naturally not the top-level drop folder). If it
    happens to *not* be empty (because the user has added additional files to
    it), just catch the error and ignore it. In another few minutes, you'll
    try again, copying the newly added files and then delete the folder.

    Because each file is being deleted as soon as it is copied, you'll won't
    build up an ever-increasing collection of files. At worst, you may have a
    collection of folders -- but unless you're fighting a hostile process
    that can add folders quicker than you can delete them, you'll eventually
    win.

    Maybe you need a heuristic that says "if the number of 'directory not
    empty' errors keeps increasing, assume we're dealing with a hostile
    process and take extra steps". Say, lock the drop folder for five minutes
    while you flush it.

    Another possibility is that before you start copying from the folders,
    you change their permissions to prohibit the user adding extra files into
    them. That way, nothing can be added to them once you start copying, and
    you can be sure that you can delete them once empty.


    --
    Steven
     
    Steven D'Aprano, Oct 9, 2007
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. The Weiss Family

    race conditions/pulse width

    The Weiss Family, Oct 16, 2004, in forum: VHDL
    Replies:
    6
    Views:
    723
    Jim Lewis
    Oct 19, 2004
  2. Taras_96
    Replies:
    7
    Views:
    6,677
    Taras_96
    Apr 5, 2005
  3. Replies:
    2
    Views:
    1,188
  4. mars
    Replies:
    6
    Views:
    355
    Laurent Pointal
    Feb 7, 2007
  5. kj
    Replies:
    4
    Views:
    300
    webEater
    Dec 22, 2006
Loading...

Share This Page