threading : make stop the caller

[Laurent Claessens]

Hello


I've a list of tasks to perform. Each of them is a threading.Thread.
Basically I have :

while task_list :
task = task_list[0]
task.run()
task_list.remove(task)


Now I want, in some circumstance to raise errors that make the loop stop.

In order IOError to make stop the loop, I tried this :

no_error = True
while task_list and no_error:
task = task_list[0]
try :
task.run()
except IOError :
no_error = False
task_list.remove(task)

But it does not work (I think the exception is raised too late).

I prefer that the taske are not aware to be included in a loop, but I
can make them raise personnal exceptions.
How can I do ?


Thanks
Laurent

 

[Chris Angelico]

  Hello


I've a list of tasks to perform. Each of them is a threading.Thread.
Basically I have :

while task_list :
  task = task_list[0]
  task.run()
  task_list.remove(task)

I'm not understanding what you're doing with threads here. Are you
using threading.Thread but then calling its run() method
synchronously?

Normally threads are used for asynchronous operations. You would then
use the start() method to spin the thread off; it will return almost
immediately, and the thread will run to completion in parallel with
you. But then you can't halt the main loop, because it will have
already finished by the time you detect the IOError (starting a bunch
of threads is pretty quick). On the other hand, the code you're
showing seems to simply call each thread's run() method one by one,
which should propagate any exceptions in the same way that function
calls usually do.

Can you share the code for one of the tasks, and show what happens
when it raises an exception?

Chris Angelico

 

[Laurent Claessens]

Le 19/06/2011 17:19, Chris Angelico a écrit :

Hello


I've a list of tasks to perform. Each of them is a threading.Thread.
Basically I have :

while task_list :
task = task_list[0]
task.run()
task_list.remove(task)

I'm not understanding what you're doing with threads here. Are you
using threading.Thread but then calling its run() method
synchronously?

Woops yes. I missprinted my example. I was using task.start() of course.

The aim is to copy the content of a repertory (with some conditions on
each file, so I cannot use shutils or something).

I've one thread that runs over the repertory and fill the list
'task_list' with taskes from the following class :


class FileCopyTask(threading.Thread):
def __init__(self,source,destination,old_version):
threading.Thread.__init__(self)
self.source = source
self.destination = destination
def run(self):
try :
shutil.copy(self.source,self.destination)
except (IOError,OSError),data :
<WHAT TO PUT HERE ??>
else :
print "file copied"


In the same time I've a thread that read the list and perform the
operations:

def run():
while task_list :
task = task_list[0]
task_list.remove(task)
task.start()


My problem is that when FileToCopyTask raises an error, the program does
not stop.
In fact when the error is Disk Full, I want to stop the whole program
because I know that the next task will fail too.

thanks for any help
Laurent

 

[Laurent Claessens]

I read the library documentation. I think that if I get a trick to kill
a thread, then I'm done.

Is there a way ?

Laurent


Le 19/06/2011 17:39, Laurent Claessens a écrit :

 

[Chris Angelico]

My problem is that when FileToCopyTask raises an error, the program does not
stop.
In fact when the error is Disk Full, I want to stop the whole program
because I know that the next task will fail too.

If you're starting a thread for every file you're copying, you're
starting a huge number of threads that probably will just end up
fighting over the disk. To get a reasonably efficient early-abort, I'd
recommend having a fixed pool of worker threads (say, ten of them),
and have each thread (a) check if the early-abort flag is set, and
then (b) start copying the next file in queue. Once the queue's empty
or the early-abort flag is set, all ten threads will terminate when
they finish their current transfers.

(The ten threads figure is arbitrary. Optimum value for performance
will come by adjusting this.)

ChrisA

 

[Laurent Claessens]

Le 19/06/2011 18:03, Chris Angelico a écrit :

If you're starting a thread for every file you're copying, you're
starting a huge number of threads that probably will just end up
fighting over the disk. To get a reasonably efficient early-abort, I'd
recommend having a fixed pool of worker threads (say, ten of them),
and have each thread (a) check if the early-abort flag is set, and
then (b) start copying the next file in queue. Once the queue's empty
or the early-abort flag is set, all ten threads will terminate when
they finish their current transfers.

Yes, my example was simplified, but I do that

Laurent

 

[Terry Reedy]

In the same time I've a thread that read the list and perform the
operations:

def run():
while task_list :
task = task_list[0]
task_list.remove(task)
task.start()

Popping task off the end of the list is more efficient:

while task_list:
task_list.pop().start()

or if the list is static

for task in task_list:
task.start()

 

[Laurent Claessens]

Popping task off the end of the list is more efficient:

while task_list:
task_list.pop().start()

That's cool. In my case it's better to do
task_list.pop(0).start

in order to pop the first element.

or if the list is static

No, my list is dynamic and is feeded by an other thread (which also has
to be closed).

Finally, I solved the problem by using a global flag looked at each
iteration of the loop. A break is done if the global flag reports an error.

That was suggested by Alain Ketterlin from the French speaking python's
usenet and Chris Angelico here.

Thanks all
Laurent

 

[Terry Reedy]

If you're starting a thread for every file you're copying, you're
starting a huge number of threads that probably will just end up
fighting over the disk. To get a reasonably efficient early-abort, I'd
recommend having a fixed pool of worker threads (say, ten of them),
and have each thread (a) check if the early-abort flag is set, and
then (b) start copying the next file in queue. Once the queue's empty
or the early-abort flag is set, all ten threads will terminate when
they finish their current transfers.

(The ten threads figure is arbitrary. Optimum value for performance
will come by adjusting this.)

I an not convinced that much of anything is gained by having multiple
copying threads. It certainly adds complication -- hence this thread. I
would just use shutil to copy files or directories of files.

Note that copytree has an option to pass a filter function:
"If ignore is given, it must be a callable that will receive as its
arguments the directory being visited by copytree(), and a list of its
contents, as returned by os.listdir(). Since copytree() is called
recursively, the ignore callable will be called once for each directory
that is copied. The callable must return a sequence of directory and
file names relative to the current directory (i.e. a subset of the items
in its second argument); these names will then be ignored in the copy
process. ignore_patterns() can be used to create such a callable that
ignores names based on glob-style patterns."

 

[Lie Ryan]

That's cool. In my case it's better to do
task_list.pop(0).start

in order to pop the first element.

then you really wanted a queue instead of a list.

There is a thread-safe `queue` module in the standard library.