Drop folder and race conditions

[Larry Bates]

I have a need to implement a drop folder upload mechanism for secure uploading
of files to a server. At first glance this appears that it would be an easy
application to write. Then I begin to think about the race conditions that
exist between the process that will wake up to upload the files and the fact
that the user can add additional files to the drop folder at any point in time.
I would like to clear out files/folders after they have been processed but it is
possible that while the background process was uploading files, the user added
additional files/folders to the drop folder that make the deletion of folders
impossible. This project has become significantly more complex than it appeared
at first. Anyone out there have any "sage" advice on how to tackle this beast?

Thanks in advance for any pointers.

Regards,
Larry Bates

 

[Steven D'Aprano]

I have a need to implement a drop folder upload mechanism for secure
uploading of files to a server. At first glance this appears that it
would be an easy application to write. Then I begin to think about the
race conditions that exist between the process that will wake up to
upload the files and the fact that the user can add additional files to
the drop folder at any point in time. I would like to clear out
files/folders after they have been processed but it is possible that
while the background process was uploading files, the user added
additional files/folders to the drop folder that make the deletion of
folders impossible. This project has become significantly more complex
than it appeared at first. Anyone out there have any "sage" advice on
how to tackle this beast?

Off the top of my head...

Recursively copy the contents of each folder to the server, deepest
first, deleting each file as it's copied. Then delete the folder as soon
as you've emptied it (but naturally not the top-level drop folder). If it
happens to *not* be empty (because the user has added additional files to
it), just catch the error and ignore it. In another few minutes, you'll
try again, copying the newly added files and then delete the folder.

Because each file is being deleted as soon as it is copied, you'll won't
build up an ever-increasing collection of files. At worst, you may have a
collection of folders -- but unless you're fighting a hostile process
that can add folders quicker than you can delete them, you'll eventually
win.

Maybe you need a heuristic that says "if the number of 'directory not
empty' errors keeps increasing, assume we're dealing with a hostile
process and take extra steps". Say, lock the drop folder for five minutes
while you flush it.

Another possibility is that before you start copying from the folders,
you change their permissions to prohibit the user adding extra files into
them. That way, nothing can be added to them once you start copying, and
you can be sure that you can delete them once empty.