OSError: [Errno 26] Text file busy during subprocess.check_call()

[harijay]

Hi,
I am writing some multithreaded code which aims to automate three
sequential data processing applications and distribute the processing
on my 16GB RAM, 64 bit Ubuntu box running Python 2.6.5

The basic class that orchestrates these jobs use Queue.Queue() to feed
the product of the first job into the Queue for the next job.

Each Thread receives a dynamically generated shell script from some
classes I wrote and then runs the script using

subprocess.call(["shell_script_file.sh"])

I tested the code on a mac laptop and also on ubuntu. Curiously on Mac
OSX 32 bit Core duo running snow leopard, the code always runs fine.
However on my ubuntu box I get sporadic errors detailed below.

I tried replacing the
subprocess.call() with

subprocess.Popen(["shellscriptfile.sh"],stdout=open("unique_process_id.log","w"),stderr=open("unique_error_log.log","w")

But I get the same "OSError: [Errno 26] Text file busy" error

Everytime I run the same job queue a different part of the job fails.

Unfortunately I dont see anybody else reporting this OSError. ANy help
in troubleshooting my "newbie" thread code will be greatly
appreciated.

Thanks
hari

The orchestrator class is at:
https://github.com/harijay/auriga/blob/master/process_multi.py

A sample thread subclass is at :
https://github.com/harijay/auriga/blob/master/scatomtzrunthread.py


Detailed error:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
self.run()
File "/home/hari/Dropbox/auriga/scatomtzrunthread.py", line 28, in
run
stat = subprocess.call([file])
File "/usr/lib/python2.6/subprocess.py", line 480, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in
_execute_child
raise child_exception
OSError: [Errno 26] Text file busy

 

[Thomas L. Shinnick]

Hi,
I am writing some multithreaded code which aims to automate three
sequential data processing applications and distribute the processing
on my 16GB RAM, 64 bit Ubuntu box running Python 2.6.5

The basic class that orchestrates these jobs use Queue.Queue() to feed
the product of the first job into the Queue for the next job.

Each Thread receives a dynamically generated shell script from some
classes I wrote and then runs the script using

subprocess.call(["shell_script_file.sh"])

You say dynamically generated. Any chance you are (re)using the same
filename each time? Is it possible that two uses of that filename
could occur at the same time? That is, is it possible that at the
same time while one process is running from the script file, another
process is trying to re-write the script file? And so maybe you need
to have dynamically generated and unique filenames

Most often I see references to binary executable files for the error
message, but I've also seen references to script files, e.g.
http://www.cyberciti.biz/faq/binbash-bad-interpreter-text-file-busy/

I tested the code on a mac laptop and also on ubuntu. Curiously on Mac
OSX 32 bit Core duo running snow leopard, the code always runs fine.
However on my ubuntu box I get sporadic errors detailed below.

I tried replacing the
subprocess.call() with

subprocess.Popen(["shellscriptfile.sh"],stdout=open("unique_process_id.log","w"),stderr=open("unique_error_log.log","w")

But I get the same "OSError: [Errno 26] Text file busy" error

Everytime I run the same job queue a different part of the job fails.

Unfortunately I dont see anybody else reporting this OSError. ANy help
in troubleshooting my "newbie" thread code will be greatly
appreciated.

Thanks
hari

The orchestrator class is at:
https://github.com/harijay/auriga/blob/master/process_multi.py

A sample thread subclass is at :
https://github.com/harijay/auriga/blob/master/scatomtzrunthread.py


Detailed error:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
self.run()
File "/home/hari/Dropbox/auriga/scatomtzrunthread.py", line 28, in
run
stat = subprocess.call([file])
File "/usr/lib/python2.6/subprocess.py", line 480, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in
_execute_child
raise child_exception
OSError: [Errno 26] Text file busy

 

[Terry Reedy]

"OSError: [Errno 26] Text file busy" error

Searching 'errno 26', the third Google response suggests that you are
trying to write to a file (especially an executable or shared library?)
that is already in use. Perhaps just trying to read when locked will
trigger?

 

[harijay]

I am writing to a unique script file . Each script file has prefixes
like script1.sh script2.sh and they reside in different directories .
The scripts will never trample each other since they are all
sequential and shell scripts and no directory will have more than one
shell script.

The only thing I am doing is dir1 and dir2 will each have a
script1.sh . But that still makes it mutually exclusive paths.

Hari

Hi,
I am writing some multithreaded code which aims to automate three
sequential data processing applications and distribute the processing
on my 16GB RAM, 64 bit Ubuntu box running Python 2.6.5

The basic class that orchestrates these jobs use Queue.Queue() to feed
the product of the first job into the Queue for the next job.

Each Thread receives a dynamically generated shell script from some
classes I wrote and then runs the script using

subprocess.call(["shell_script_file.sh"])

You say dynamically generated.  Any chance you are (re)using the same
filename each time?  Is it possible that two uses of that filename
could occur at the same time?  That is, is it possible that at the
same time while one process is running from the script file, another
process is trying to re-write the script file?  And so maybe you need
to have dynamically generated and unique filenames

Most often I see references to binary executable files for the error
message, but I've also seen references to script files, e.g.
     http://www.cyberciti.biz/faq/binbash-bad-interpreter-text-file-busy/

I tested the code on a mac laptop and also on ubuntu. Curiously on Mac
OSX 32 bit Core duo running snow leopard, the code always runs fine.
However on my ubuntu box I get sporadic errors detailed below.

I tried replacing the
subprocess.call() with

subprocess.Popen(["shellscriptfile.sh"],stdout=open("unique_process_id..log ","w"),stderr=open("unique_error_log.log","w")

But I get the same "OSError: [Errno 26] Text file busy"  error

Everytime I run the same job queue a different part of the job fails.

Unfortunately I dont see anybody else reporting this OSError. ANy help
in troubleshooting my "newbie" thread code will be greatly
appreciated.

The orchestrator class is at:
https://github.com/harijay/auriga/blob/master/process_multi.py

A sample thread subclass is at :
https://github.com/harijay/auriga/blob/master/scatomtzrunthread.py

Detailed error:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
    self.run()
  File "/home/hari/Dropbox/auriga/scatomtzrunthread.py", line 28, in
run
    stat = subprocess.call([file])
  File "/usr/lib/python2.6/subprocess.py", line 480, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1139, in
_execute_child
    raise child_exception
OSError: [Errno 26] Text file busy

 

[Steven D'Aprano]

On Thu, 30 Dec 2010 13:46:35 -0800, harijay wrote:

[...]

But I get the same "OSError: [Errno 26] Text file busy" error

Everytime I run the same job queue a different part of the job fails.

Unfortunately I dont see anybody else reporting this OSError. ANy help
in troubleshooting my "newbie" thread code will be greatly appreciated.

Try catching the OSError exception and inspecting the filename attribute.
Something like this might help:

# Untested
def report_error(type, value, traceback):
if type is OSError:
print value.filename
sys.__excepthook__(type, value, traceback)


sys.excepthook = report_error

 

[Nobody]

Each Thread receives a dynamically generated shell script from some
classes I wrote and then runs the script using

subprocess.call(["shell_script_file.sh"])

But I get the same "OSError: [Errno 26] Text file busy" error

"Text file busy" aka ETXTBSY occurs if you try to execute a file which is
open for write by any process.

Be sure to explicitly close() the script before attempting to execute it.
Don't rely upon the destructor closing it, as that may be deferred.

Also, if you spawn any child processes, ensure that they don't inherit the
descriptor being used to write to the script. Ideally, you should set the
close-on-exec flag on the descriptor as soon as the file is opened. Using
close_fds=True in subprocess.Popen() will solve the issue for processes
spawned that way, but you also need to consider subprocesses which may be
spawned "behind the scenes" by library functions.

 

[harijay]

Thanks Steven and other user for the suggestion on finding out which file is blocking and ensure closing of file descriptors when running.

Here is my code ..will report back if it did work



Hari

class ScaToMtzRunThread(Thread):
    def __init__(self, in_queue, out_queue,auriga_output_directory_root):
        Thread.__init__(self)
        self.in_queue = in_queue
        self.out_queue = out_queue
        self.auriga_output_directory_root = auriga_output_directory_root
        sys.excepthook = self.report_error
        
    def report_error(type,value,traceback):
        if type is OSError:
            print value.filename
            sys.__excepthook__(type,value,traceback)
        else:
            sys.__excepthook__(type,value,traceback)

            
    def run(self):
        while True:
            path = self.in_queue.get()
#            sys.stdout.flush()
#            if path is None:
#                self.in_queue.task_done()
#                self.join()
            myfile = ScalePackToMtzRunscriptCreator(path,self.auriga_output_directory_root)
            scrfile = myfile.create_and_return_runscript_file()
            try:
                subprocess.check_call([scrfile],close_fds=True)
                time.sleep(2)
            except subprocess.CalledProcessError:
                self.in_queue.put(path)
            self.out_queue.put(myfile.mtzoutpath())

Also I am using this function to write my shell scripts ( it does explicitly close file descriptor):

def safe_write_script(string,filehandle):
    mutex.acquire()
    filehandle.write(string)
    filehandle.close()
    os.chmod(filehandle.name,0o755)
    mutex.release()

On Thu, 30 Dec 2010 13:46:35 -0800, harijay wrote:

[...]
> But I get the same "OSError: [Errno 26] Text file busy" error
>
> Everytime I run the same job queue a different part of the job fails.
>
> Unfortunately I dont see anybody else reporting this OSError. ANy help
> in troubleshooting my "newbie" thread code will be greatly appreciated.

Try catching the OSError exception and inspecting the filename attribute.
Something like this might help:

# Untested
def report_error(type, value, traceback):
if type is OSError:
print value.filename
sys.__excepthook__(type, value, traceback)


sys.excepthook = report_error


--
Steven

 

[harijay]

Cannot avoid OSError 26

I continue geting these OSErrors Errno 26 on ubuntu 32 and 64 bit despite setting close_fds=True , adding wait blocks between various disk i/o steps.

I never get the error on OSX

Also having some problems capturing the file that is busy and leading to the error ( as per Stevens suggestion). I am using the same file descriptors. But each thread is its own subclass , so arent these really different descriptor objects? Also the scripts end up in different directories..and other than being called once..They are never really called again


Somehow I think the information in this issue about sys.excepthook in a threaded subclass context affects me when trying to find out which files are busy and leading to Errno 26:

issue1230540


My Thread class is reproduced below

#!/usr/bin/python
from scalepacktomtzrunscriptcreator import ScalePackToMtzRunscriptCreator
from process_multi_util_functions import report
from threading import Thread
import sys
import subprocess
import time
import pprint

class ScaToMtzRunThread(Thread):
    def __init__(self, in_queue, out_queue,auriga_output_directory_root):
        Thread.__init__(self)
        self.in_queue = in_queue
        self.out_queue = out_queue
        self.auriga_output_directory_root = auriga_output_directory_root
        sys.excepthook = self.report_error

    def report_error(type=None,value=None,traceback=None):
        if type is OSError:
            print ("***FilenameError***",value.filename)
            sys.__excepthook__(type,value,traceback)
        else:
            sys.__excepthook__(type,value,traceback)


    def run(self):
        while True:
            path = self.in_queue.get()
#            sys.stdout.flush()
#            if path is None:
#                self.in_queue.task_done()
#                self.join()
            myfile = ScalePackToMtzRunscriptCreator(path,self.auriga_output_directory_root)
            scrfile = myfile.create_and_return_runscript_file()
            time.sleep(2)
            try:
                subprocess.Popen([scrfile],close_fds=True).wait()
                time.sleep(2)
            except subprocess.CalledProcessError:
                self.in_queue.put(path)
            except OSError as e :
                print "CAUGHT an Oesserror",e
                afile = open("error.txt","a")
                afile.write(self.report_error(sys.exc_info()))
                afile.close()

            self.out_queue.put(myfile.mtzoutpath())

 

[harijay]

Used Queue properly to handle OSErrors

Hi,
I solved my premature termination errors , by handling the OSErrors and inserting items back into the Queue task list.

The new Thread code is pasted here:
I still need to figure out how to exit without a CTRL C. But I shall look arounf for how to do that.

Thanks a tonne for your help everyone

Hari

#!/usr/bin/python
from scalepacktomtzrunscriptcreator import ScalePackToMtzRunscriptCreator
from process_multi_util_functions import report
from threading import Thread
import sys
import subprocess
import time
import pprint

class ScaToMtzRunThread(Thread):
    def __init__(self, in_queue, out_queue,auriga_output_directory_root):
        Thread.__init__(self)
        self.in_queue = in_queue
        self.out_queue = out_queue
        self.auriga_output_directory_root = auriga_output_directory_root
        sys.excepthook = self.report_error
        
    def report_error(type=None,value=None,traceback=None):
        if type is OSError:
            print ("***FilenameError***",value.filename)
            sys.__excepthook__(type,value,traceback)
        else:
            sys.__excepthook__(type,value,traceback)

            
    def run(self):
        while True:
            path = self.in_queue.get()
            myfile = ScalePackToMtzRunscriptCreator(path,self.auriga_output_directory_root)
            scrfile = myfile.create_and_return_runscript_file()
            time.sleep(2)
	    try:
                subprocess.Popen([scrfile],close_fds=True).wait()
                time.sleep(2)
            except subprocess.CalledProcessError:
                self.in_queue.put(path)
            except OSError:
                self.in_queue.put(path)
	    self.out_queue.put(myfile.mtzoutpath())
            self.in_queue.task_done()