timing issue: shutil.rmtree and os.makedirs

Discussion in 'Python' started by Tim, Jul 29, 2013.

  1. Tim

    Tim Guest

    I have the following function (Python2.7 on FreeBSD) that results in an OSError.

    My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:

    def make_clean_dir(directory):
    if os.path.exists(directory):
    if os.path.isdir(directory):
    shutil.rmtree(directory)
    else:
    os.unlink(directory)
    os.makedirs(directory)

    The last bit of the traceback is:
    File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
    os.makedirs(directory)
    File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
    OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'

    The directory 'testing_html' existed when I executed the function;
    So I suppose the directory wasn't finished being removed by the time os.makedirs was invoked. How can avoid this? (A context manager maybe?).

    thanks,
    --Tim
     
    Tim, Jul 29, 2013
    #1
    1. Advertising

  2. On Mon, Jul 29, 2013 at 8:16 PM, Tim <> wrote:
    > My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:
    >
    > def make_clean_dir(directory):
    > if os.path.exists(directory):
    > if os.path.isdir(directory):
    > shutil.rmtree(directory)
    > else:
    > os.unlink(directory)
    > os.makedirs(directory)
    >
    > The last bit of the traceback is:
    > File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
    > os.makedirs(directory)
    > File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
    > mkdir(name, mode)
    > OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'
    >
    > The directory 'testing_html' existed when I executed the function;


    First thing I'd check is: Did rmtree succeed? Try removing the
    makedirs and test it again; then, when your process has completely
    finished, see if the directory is there. If it is, the problem is in
    rmtree - for instance:

    * You might not have permission to remove everything
    * There might be a messed-up object in the file system
    * If the directory is a remote share mount point, the other end might
    have lied about the removal
    * Something might have been created inside the directory during the removal
    * Myriad other possibilities

    As I understand rmtree's docs, any errors *that it detects* will be
    raised as exceptions (since you haven't told it to suppress or handle
    them), but possibly there's an error that it isn't able to detect.
    Worth a test, anyhow.

    ChrisA
     
    Chris Angelico, Jul 30, 2013
    #2
    1. Advertising

  3. Tim

    Tim Guest

    On Monday, July 29, 2013 7:52:36 PM UTC-4, Chris Angelico wrote:
    > On Mon, Jul 29, 2013 at 8:16 PM, Tim wrote:
    > > My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:


    > > def make_clean_dir(directory):
    > > if os.path.exists(directory):
    > > if os.path.isdir(directory):
    > > shutil.rmtree(directory)
    > > else:
    > > os.unlink(directory)
    > > os.makedirs(directory)
    > >
    > > The last bit of the traceback is:
    > > File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
    > > os.makedirs(directory)
    > > File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
    > > mkdir(name, mode)
    > > OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'
    > >
    > > The directory 'testing_html' existed when I executed the function;

    >
    > First thing I'd check is: Did rmtree succeed? Try removing the
    > makedirs and test it again; then, when your process has completely
    > finished, see if the directory is there. If it is, the problem is in
    > rmtree - for instance:


    > * You might not have permission to remove everything
    > * There might be a messed-up object in the file system
    > * If the directory is a remote share mount point, the other end might
    > have lied about the removal
    > * Something might have been created inside the directory during the removal
    > * Myriad other possibilities
    > As I understand rmtree's docs, any errors *that it detects* will be
    > raised as exceptions (since you haven't told it to suppress or handle
    > them), but possibly there's an error that it isn't able to detect.
    > Worth a test, anyhow.
    >
    > ChrisA


    Thanks Chris, but the directory was actually removed on the first run in spite of the traceback; when I run it a second time (immediately after the first time), it runs fine. That's why I thought it was a timing issue. I thought about just putting a sleep in there, but that made me feel dirty.

    hmm, now that you mention it, this is executing on a remote box with accessto the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.

    But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

    thanks,
    --Tim
     
    Tim, Jul 30, 2013
    #3
  4. On Tue, Jul 30, 2013 at 2:10 PM, Tim <> wrote:
    > hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.
    >
    > But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.


    Hmm. What system is used for the file system sharing? I know quite a
    few of them lie about whether something's been completely done or not.

    Can you use inotify to tell you when the directory's been deleted?
    Seems stupid though.

    Worst case, all you need is a quick loop at the bottom, eg:

    for delay in 100,300,600,1000,3000,5000,10000:
    if not os.path.exists(directory): break
    sleep(delay)

    That'll sleep a maximum of 20 seconds, tune as required. Of course, if
    there's a way to tune the FS to guarantee that the removal blocks
    correctly, that would be way better than sleep()!

    ChrisA
     
    Chris Angelico, Jul 30, 2013
    #4
  5. On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote:

    > for delay in 100,300,600,1000,3000,5000,10000:
    > if not os.path.exists(directory): break
    > sleep(delay)
    >
    > That'll sleep a maximum of 20 seconds, tune as required.


    Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7
    minutes (assuming the directory doesn't get deleted instantaneously).

    time.sleep() takes an argument in seconds.



    --
    Steven
     
    Steven D'Aprano, Jul 30, 2013
    #5
  6. On Tue, Jul 30, 2013 at 3:07 PM, Steven D'Aprano
    <> wrote:
    > On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote:
    >
    >> for delay in 100,300,600,1000,3000,5000,10000:
    >> if not os.path.exists(directory): break
    >> sleep(delay)
    >>
    >> That'll sleep a maximum of 20 seconds, tune as required.

    >
    > Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7
    > minutes (assuming the directory doesn't get deleted instantaneously).
    >
    > time.sleep() takes an argument in seconds.


    LOL! Whoops. That's what I get for not checking my docs. This is why
    we have public responses, my errors can be caught by someone else.

    ChrisA
     
    Chris Angelico, Jul 30, 2013
    #6
  7. Tim

    Tim Guest

    On Tuesday, July 30, 2013 9:27:10 AM UTC-4, Chris Angelico wrote:
    > On Tue, Jul 30, 2013 at 2:10 PM, Tim wrote:
    > > hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, thereis a local call to an intermediate script that connects to a socket on theremote where the above program actually runs, but the file system is the same place for both local and remote.
    > >
    > > But even so, since the script that does the rmtree and mkdir is runningon the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

    >
    > Hmm. What system is used for the file system sharing? I know quite a
    > few of them lie about whether something's been completely done or not.
    > Can you use inotify to tell you when the directory's been deleted?
    > Seems stupid though.
    > Worst case, all you need is a quick loop at the bottom, eg:
    >

    <<snip, thanks for the code and correction>>
    > ChrisA


    Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).

    thanks much!
    --Tim
     
    Tim, Jul 30, 2013
    #7
  8. On Tue, Jul 30, 2013 at 4:37 PM, Tim <> wrote:
    > Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).


    I've used several different networked file systems, including
    NetBIOS/NetBEUI/SMB/Samba/etc, sshfs/cifs, and nfs. Not one of them
    "feels" as clean as I'd like, though sshfs comes closest. There always
    seem to be hacks around.

    ChrisA
     
    Chris Angelico, Jul 30, 2013
    #8
  9. On Jul 30, 2013 3:29 PM, "Chris Angelico" <> wrote:
    >
    > On Tue, Jul 30, 2013 at 2:10 PM, Tim <> wrote:
    > > hmm, now that you mention it, this is executing on a remote box with

    access to the same file system my local calling program is on. That is,
    there is a local call to an intermediate script that connects to a socket
    on the remote where the above program actually runs, but the file system is
    the same place for both local and remote.
    > >
    > > But even so, since the script that does the rmtree and mkdir is running

    on the same machine (even though it's remote), I would think the mkdir
    couldn't execute until the rmtree was completely finished.
    >
    > Hmm. What system is used for the file system sharing? I know quite a
    > few of them lie about whether something's been completely done or not.
    >
    > Can you use inotify to tell you when the directory's been deleted?
    > Seems stupid though.


    Inotify is a linux thing, but there is kqueue for Free?BSD. OP can run the
    deletion procedure, wait for a NOTE_DELETE event, which would block, and
    create the fresh directory afterwards. It may require some C hacking
    though, in case Python lacks a kqueue wrapper. I think that this kind of
    approach would be more sound than a check-wait-loop approach.

    (I would elaborate more with pointers to appropriate documentation, but I'm
    on a silly tablet, please excuse me for that.)

    -gk
     
    Göktuğ Kayaalp, Jul 30, 2013
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    2
    Views:
    1,344
  2. Tim Chase
    Replies:
    1
    Views:
    1,519
    Sean DiZazzo
    Jul 13, 2009
  3. Fletcher Johnson

    Why is the shutil module called shutil?

    Fletcher Johnson, Sep 24, 2011, in forum: Python
    Replies:
    11
    Views:
    910
    alex23
    Sep 26, 2011
  4. Flagstaff

    rmtree and untaint

    Flagstaff, Jan 3, 2004, in forum: Perl Misc
    Replies:
    4
    Views:
    138
    Flagstaff
    Jan 5, 2004
  5. Chuck Bradley

    using wildcards in rmtree or blog on windows

    Chuck Bradley, Jul 15, 2004, in forum: Perl Misc
    Replies:
    2
    Views:
    292
    Chuck Bradley
    Jul 16, 2004
Loading...

Share This Page