Threading issue: [Error 9]: bad file descriptor

[Kevin]

Has anyone else run into random IOErrors ([Error 9] bad file descriptor) in
multi-threaded Python apps?

I've searched the newsgroups, and the only references I can find to that
seem to be sporadically related to threading, but nobody seems to have a
"fix" or even a clear understanding of what the issue is.

The app I'm running into this with (once in a while... not really repeatable
predictably) is using 3 threads, and I'm using locks around all shared
objects. The error seems to be happening mostly when I'm trying to delete a
temporary file that was originally created by a different thread.

ANy suggestions? Thanks,

Kevin.

 

[Andrew MacIntyre]

Has anyone else run into random IOErrors ([Error 9] bad file descriptor) in
multi-threaded Python apps?

Not indicating which of the platforms with Python supported threads you're
experiencing this on doesn't give much scope for others to help...

 

[Aahz]

Has anyone else run into random IOErrors ([Error 9] bad file
descriptor) in multi-threaded Python apps?

The app I'm running into this with (once in a while... not really
repeatable predictably) is using 3 threads, and I'm using locks around
all shared objects. The error seems to be happening mostly when I'm
trying to delete a temporary file that was originally created by a
different thread.

Sounds like the problem isn't locking per se, but lack of
synchronization. The simple answer: never use an external object in
multiple threads.

 

[Cliff Wells]

My oversight, thanks!

I'm testing on Windows 2000 Professional, but have had other users report
the same problem on other Windows flavours (NT, 98).

I experienced a similar problem a few years ago (on Windows NT Server).
A long-running threaded application would sporadically raise an
exception when writing to a log file. Never figured out the cause, but
"solved" it by catching the exception and reopening the file (IIRC).

Regards,

 

[David Bolen]

My oversight, thanks!

I'm testing on Windows 2000 Professional, but have had other users report
the same problem on other Windows flavours (NT, 98).

I can only speak for NT, but I have seen timing issues in the past
where there is some additional latency imposed by the system between
when I release all references to a file and when it can be removed
from the filesystem by the same process. This has affected me in C
code as well as Python. Perhaps you're running into the same thing,
although for me it normally showed up as a permission denied error.

Sometimes the higher level Python I/O exceptions can be a bit vague
(by their nature, a lot of underlying Win32 error codes eventually get
translated into a far fewer number of C RTL error codes, which in turn
bubble up as Python exceptions). One thing you could try, depending
on the operation in question, is to replace the Python operation (such
as os.remove) with a matching win32all module operation (such as
win32file.DeleteFile), which should raise a more specific exception.

In my past experiences, the most practical, albeit inelegant,
workaround was to retry a failed removal operation after several
seconds. Definitely a kluge, but I was never able to isolate any more
well-defined approach.

This of course assumes that you don't really have a threading race
condition under which you still do hold active handles to the file
that you are trying to remove. The fact that you're getting back an
invalid handle error certainly makes it seem that you make still be
looking at a race condition or inter-thread data corruption within
your application.

Depending on the structure of the application, if you can wrap (or if
you have already) all access to your file handles through a common
class you should be able to instrument it in order to at least get
some sort of trace as to how they are accessed, and perhaps catch one
of the failures. But as you probably know, debugging this sort of
thing can be nasty, particularly if it's truly sporadic in nature.

-- David