bug with os.rename in 2.4.1?

Discussion in 'Python' started by t123, Apr 28, 2009.

  1. t123

    t123 Guest

    I was wondering if anyone has seen this problem before?

    I have a job that runs hourly. Part of the processing is to rename a
    file using os.rename. The file name is the same every hour.
    Sometimes the os.rename fails and the file does not get renamed. It
    only happens occasionally. Most of the time it works just fine.

    Appreciate any ideas.

    Thanks
     
    t123, Apr 28, 2009
    #1
    1. Advertising

  2. > I was wondering if anyone has seen this problem before?
    >
    > I have a job that runs hourly. Part of the processing is to rename a
    > file using os.rename. The file name is the same every hour.
    > Sometimes the os.rename fails and the file does not get renamed. It
    > only happens occasionally. Most of the time it works just fine.


    Unless you send a minimal version of the code you use and the complete
    traceback when an error occurs it will be impossible to help you.

    Cheers,
    Daniel

    --
    Psss, psss, put it down! - http://www.cafepress.com/putitdown
     
    Daniel Fetchinson, Apr 28, 2009
    #2
    1. Advertising

  3. En Tue, 28 Apr 2009 14:15:54 -0300, Daniel Fetchinson
    <> escribió:

    >> I have a job that runs hourly. Part of the processing is to rename a
    >> file using os.rename. The file name is the same every hour.
    >> Sometimes the os.rename fails and the file does not get renamed. It
    >> only happens occasionally. Most of the time it works just fine.

    >
    > Unless you send a minimal version of the code you use and the complete
    > traceback when an error occurs it will be impossible to help you.


    Also, don'f forget to tell us the OS / environment you're using.
    On Windows, you cannot rename a file while it is open by any process.

    --
    Gabriel Genellina
     
    Gabriel Genellina, Apr 28, 2009
    #3
  4. t123

    t123 Guest

    It's running on solaris 9. Here is some of the code. It's actually
    at the beginning of the job. The files are ftp'd over. The first
    thing that happens is that the files get renamed before any processing
    of the file. And when it fails, it always fails at the first file,
    comm.dat. What I can't understand is why the inconsistent behavior.

    try:
    if os.path.exists(paths.xferin_dir+'/COMM.DAT'):
    os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/
    COMM.DAT'+'.0')
    if os.path.exists(paths.xferin_dir+'/DESC.DAT'):
    os.rename(paths.xferin_dir+'/DESC.DAT',paths.xferin_dir+'/
    DESC.DAT'+'.0')
    if os.path.exists(paths.xferin_dir+'/HEADER.DAT'):
    os.rename(paths.xferin_dir+'/HEADER.DAT',paths.xferin_dir+'/
    HEADER.DAT'+'.0')
    if os.path.exists(paths.xferin_dir+'/LINE.DAT'):
    os.rename(paths.xferin_dir+'/LINE.DAT',paths.xferin_dir+'/
    LINE.DAT'+'.0')

    except:
    print "Unexpected error:", sys.exc_info()[0]
    raise

    On Apr 28, 12:25 pm, Duncan Booth <>
    wrote:
    > t123 <> wrote:
    > > I was wondering if anyone has seen this problem before?

    >
    > > I have a job that runs hourly.  Part of the processing is to rename a
    > > file using os.rename.  The file name is the same every hour.
    > > Sometimes the os.rename fails and the file does not get renamed.  It
    > > only happens occasionally.  Most of the time it works just fine.

    >
    > > Appreciate any ideas.

    >
    > My crystal ball says you are using Windows and don't always close the file
    > before you try to rename it.
     
    t123, Apr 28, 2009
    #4
  5. t123

    MRAB Guest

    t123 wrote:
    > I was wondering if anyone has seen this problem before?
    >
    > I have a job that runs hourly. Part of the processing is to rename a
    > file using os.rename. The file name is the same every hour.
    > Sometimes the os.rename fails and the file does not get renamed. It
    > only happens occasionally. Most of the time it works just fine.
    >
    > Appreciate any ideas.
    >

    This can happen if an antivirus checker, or Windows indexing service
    (assuming you're using Windows), or some such background task, is
    accessing the file at the time. It's a good idea to wait a short time
    and then retry, perhaps more than once, before giving up. Just write a
    short function to do this.
     
    MRAB, Apr 28, 2009
    #5
  6. En Tue, 28 Apr 2009 14:43:01 -0300, t123 <> escribió:

    > It's running on solaris 9. Here is some of the code. It's actually
    > at the beginning of the job. The files are ftp'd over. The first
    > thing that happens is that the files get renamed before any processing
    > of the file. And when it fails, it always fails at the first file,
    > comm.dat. What I can't understand is why the inconsistent behavior.
    >
    > try:
    > if os.path.exists(paths.xferin_dir+'/COMM.DAT'):
    > os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/
    > COMM.DAT'+'.0')
    > if os.path.exists(paths.xferin_dir+'/DESC.DAT'):
    > os.rename(paths.xferin_dir+'/DESC.DAT',paths.xferin_dir+'/
    > DESC.DAT'+'.0')
    > if os.path.exists(paths.xferin_dir+'/HEADER.DAT'):
    > os.rename(paths.xferin_dir+'/HEADER.DAT',paths.xferin_dir+'/
    > HEADER.DAT'+'.0')
    > if os.path.exists(paths.xferin_dir+'/LINE.DAT'):
    > os.rename(paths.xferin_dir+'/LINE.DAT',paths.xferin_dir+'/
    > LINE.DAT'+'.0')
    >
    > except:
    > print "Unexpected error:", sys.exc_info()[0]
    > raise


    And the error is...?

    --
    Gabriel Genellina
     
    Gabriel Genellina, Apr 28, 2009
    #6
  7. On Tue, 28 Apr 2009 14:30:02 -0500, Nick Craig-Wood wrote:

    > t123 <> wrote:
    >> It's running on solaris 9. Here is some of the code. It's actually
    >> at the beginning of the job. The files are ftp'd over. The first
    >> thing that happens is that the files get renamed before any processing
    >> of the file. And when it fails, it always fails at the first file,
    >> comm.dat. What I can't understand is why the inconsistent behavior.
    >>
    >> try:
    >> if os.path.exists(paths.xferin_dir+'/COMM.DAT'):
    >> os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/

    >
    > This code is inherently racy... What if two copies of your code started
    > simultaneously? They might both run the os.path.exists but only one
    > will succeed in the os.rename.
    >
    > You could write instead
    >
    > try:
    > os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/

    COMM.DAT'+'.0')
    > except OSError:
    > pass
    >
    > Which isn't racy.



    The race condition is still there. The only difference is that in the
    first case it fails noisily, with an exception, and in the second it
    fails quietly and does nothing.



    --
    Steven
     
    Steven D'Aprano, Apr 29, 2009
    #7
  8. On Thu, 30 Apr 2009 03:30:04 -0500, Nick Craig-Wood wrote:

    >> The race condition is still there. The only difference is that in the
    >> first case it fails noisily, with an exception, and in the second it
    >> fails quietly and does nothing.

    >
    > I'd argue that since os.rename implements the syscall rename() and that
    > is defined to be atomic (give or take) then the above is atomic and
    > can't possibly be racy.


    Ah, you're right, at least for systems where rename is atomic.

    What I was trying to say is that the risk of the file *not* being renamed
    is still there, but in the first case the code failed noisily with an
    exception and in the second it just failed quietly. But that's wrong, the
    test is to see if the source file exists, not whether the destination
    file exists. If the destination file exists, and you have write-access to
    it, then it will be over-written no matter what.


    --
    Steven
     
    Steven D'Aprano, May 1, 2009
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hugz
    Replies:
    3
    Views:
    593
    Harmit Jhajj
    Feb 25, 2004
  2. Kathy Burke
    Replies:
    1
    Views:
    454
    Ron Vecchi
    Jul 26, 2003
  3. Kathy Burke
    Replies:
    0
    Views:
    428
    Kathy Burke
    Jul 27, 2003
  4. Marina
    Replies:
    3
    Views:
    2,402
    Himanshu Dhami
    Aug 15, 2003
  5. THY
    Replies:
    1
    Views:
    378
Loading...

Share This Page