bug with os.rename in 2.4.1?

T

t123

I was wondering if anyone has seen this problem before?

I have a job that runs hourly. Part of the processing is to rename a
file using os.rename. The file name is the same every hour.
Sometimes the os.rename fails and the file does not get renamed. It
only happens occasionally. Most of the time it works just fine.

Appreciate any ideas.

Thanks
 
D

Daniel Fetchinson

I was wondering if anyone has seen this problem before?
I have a job that runs hourly. Part of the processing is to rename a
file using os.rename. The file name is the same every hour.
Sometimes the os.rename fails and the file does not get renamed. It
only happens occasionally. Most of the time it works just fine.

Unless you send a minimal version of the code you use and the complete
traceback when an error occurs it will be impossible to help you.

Cheers,
Daniel
 
G

Gabriel Genellina

En Tue, 28 Apr 2009 14:15:54 -0300, Daniel Fetchinson
Unless you send a minimal version of the code you use and the complete
traceback when an error occurs it will be impossible to help you.

Also, don'f forget to tell us the OS / environment you're using.
On Windows, you cannot rename a file while it is open by any process.
 
T

t123

It's running on solaris 9. Here is some of the code. It's actually
at the beginning of the job. The files are ftp'd over. The first
thing that happens is that the files get renamed before any processing
of the file. And when it fails, it always fails at the first file,
comm.dat. What I can't understand is why the inconsistent behavior.

try:
if os.path.exists(paths.xferin_dir+'/COMM.DAT'):
os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/
COMM.DAT'+'.0')
if os.path.exists(paths.xferin_dir+'/DESC.DAT'):
os.rename(paths.xferin_dir+'/DESC.DAT',paths.xferin_dir+'/
DESC.DAT'+'.0')
if os.path.exists(paths.xferin_dir+'/HEADER.DAT'):
os.rename(paths.xferin_dir+'/HEADER.DAT',paths.xferin_dir+'/
HEADER.DAT'+'.0')
if os.path.exists(paths.xferin_dir+'/LINE.DAT'):
os.rename(paths.xferin_dir+'/LINE.DAT',paths.xferin_dir+'/
LINE.DAT'+'.0')

except:
print "Unexpected error:", sys.exc_info()[0]
raise
 
M

MRAB

t123 said:
I was wondering if anyone has seen this problem before?

I have a job that runs hourly. Part of the processing is to rename a
file using os.rename. The file name is the same every hour.
Sometimes the os.rename fails and the file does not get renamed. It
only happens occasionally. Most of the time it works just fine.

Appreciate any ideas.
This can happen if an antivirus checker, or Windows indexing service
(assuming you're using Windows), or some such background task, is
accessing the file at the time. It's a good idea to wait a short time
and then retry, perhaps more than once, before giving up. Just write a
short function to do this.
 
G

Gabriel Genellina

It's running on solaris 9. Here is some of the code. It's actually
at the beginning of the job. The files are ftp'd over. The first
thing that happens is that the files get renamed before any processing
of the file. And when it fails, it always fails at the first file,
comm.dat. What I can't understand is why the inconsistent behavior.

try:
if os.path.exists(paths.xferin_dir+'/COMM.DAT'):
os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/
COMM.DAT'+'.0')
if os.path.exists(paths.xferin_dir+'/DESC.DAT'):
os.rename(paths.xferin_dir+'/DESC.DAT',paths.xferin_dir+'/
DESC.DAT'+'.0')
if os.path.exists(paths.xferin_dir+'/HEADER.DAT'):
os.rename(paths.xferin_dir+'/HEADER.DAT',paths.xferin_dir+'/
HEADER.DAT'+'.0')
if os.path.exists(paths.xferin_dir+'/LINE.DAT'):
os.rename(paths.xferin_dir+'/LINE.DAT',paths.xferin_dir+'/
LINE.DAT'+'.0')

except:
print "Unexpected error:", sys.exc_info()[0]
raise

And the error is...?
 
S

Steven D'Aprano

This code is inherently racy... What if two copies of your code started
simultaneously? They might both run the os.path.exists but only one
will succeed in the os.rename.

You could write instead

try:
os.rename(paths.xferin_dir+'/COMM.DAT',paths.xferin_dir+'/ COMM.DAT'+'.0')
except OSError:
pass

Which isn't racy.


The race condition is still there. The only difference is that in the
first case it fails noisily, with an exception, and in the second it
fails quietly and does nothing.
 
S

Steven D'Aprano

I'd argue that since os.rename implements the syscall rename() and that
is defined to be atomic (give or take) then the above is atomic and
can't possibly be racy.

Ah, you're right, at least for systems where rename is atomic.

What I was trying to say is that the risk of the file *not* being renamed
is still there, but in the first case the code failed noisily with an
exception and in the second it just failed quietly. But that's wrong, the
test is to see if the source file exists, not whether the destination
file exists. If the destination file exists, and you have write-access to
it, then it will be over-written no matter what.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top