repeat something in a thread, but stop when the program stops

[Harald Armin Massa]

I need to do some synchronisations like in a cron.job

import time
from threading import Thread

class updater(Thread):
def run(self):
while True:
do_updates()
time.sleep(600)

updater().start()

# regular program flow continues ....
print "tralalala"
while True:
do_very_important_stuff()
if ask_user_if_I_should_stop():
break

print "programm stopped because YOU decided ..."


#################################
BUT ... the updater() thread keeps running. The software will not end
until forced with keyboard interrupt.

I would like to have a way to stop that thread in a automatic way.

one obvious solution would be:

import time
from threading import Thread
import Queue

class updater(Thread):
stopper=Queue.Queue()
def run(self):
self.timegone=0
while self.stopper.empty():
time.sleep(0.5)
self.timegone+=1
if self.timegone>=600:
print "you waited, time passed, I will do updates"
do_updates()
self.timegone=0
print "updater stopped also"

myupdater=updater()
myupdater.start()

while True:
print "doing important stuff"
do_very_important_stuff()
a=raw_input("stop?")
if a=="y":
myupdater.stopper.put("stop this thread, will you??")
break

print "programm stopped because YOU decided ..."

############
I use a Queue to communicate the stop-message to the updater-thread.
Queue is documented as being thread-safe, so no hobbling with mutex or
stuff.

BUT... just to look at that queue the thread has to be activated. And
.... the usual run-time of the software will be around 10hours, the
update has to be done 20 times. And only in the evening the programm
will be ended (shutdown of the computer). And more over: it will be in
the background. time.sleep() takes essentially no processor time.

Checking the queue will not stop the computer, but takes mor than
necessary.

Is there any better approach ?

Harald

 

[Peter Otten]

I would like to have a way to stop that thread in a automatic way.

one obvious solution would be:

import time
from threading import Thread
import Queue

class updater(Thread):
stopper=Queue.Queue()
def run(self):
self.timegone=0
while self.stopper.empty():
time.sleep(0.5)
self.timegone+=1
if self.timegone>=600:
print "you waited, time passed, I will do updates"
do_updates()
self.timegone=0
print "updater stopped also"

As of 2.3 (I think) Queue.get() accepts a timeout parameter. You could
change the above to

from Queue import Queue, Empty

class updater(Thread):
stopper = Queue()
def run(self):
while True:
do_updates()
try:
self.stopper.get(True, 600)
except Empty:
pass
else:
break

and the sleeping thread should immediately wake up when it gets the stop
notification.

Peter

 

[Tim Peters]

[Harald Armin Massa]

I need to do some synchronisations like in a cron.job

import time
from threading import Thread

class updater(Thread):
def run(self):
while True:
do_updates()
time.sleep(600)

updater().start()

# regular program flow continues ....
print "tralalala"
while True:
do_very_important_stuff()
if ask_user_if_I_should_stop():
break

print "programm stopped because YOU decided ..."

#################################
BUT ... the updater() thread keeps running. The software will not end
until forced with keyboard interrupt.

It keeps running because you didn't mark your thread as a daemon
thread. By default, Python waits for all threads to complete before
it will exit. If you don't want Python to wait for some particular
thread T, do T.setDaemon(True) before starting T. You have to accept
that T will then get killed ungracefully by your operating system when
Python exits (on most platforms).

Peter Otten suggested Queue.get() with a timeout, and that can also be
used. If you do that, the thread will wake up about 20 times per
second to check the queue. The logic here is better suited to a
Threading.Event, though (Queue is overkill; Event.wait() can also be
given a timeout).

....

BUT... just to look at that queue the thread has to be activated. And
... the usual run-time of the software will be around 10hours, the
update has to be done 20 times. And only in the evening the programm
will be ended (shutdown of the computer). And more over: it will be in
the background. time.sleep() takes essentially no processor time.

Checking the queue will not stop the computer, but takes mor than
necessary.

Measure it: you won't be able to see a difference. Modern processors
cram on the order of 1E9 cycles into each second. Doing a tiny amount
of work 20 times per second is trivial.

 

[Harald Massa]

Thank you very much, Peter and Tim!

I was really not aware of the timeout-parameter of Queue and did not
understand demons!

Measure it: you won't be able to see a difference. Modern processors
cram on the order of 1E9 cycles into each second. Doing a tiny amount
of work 20 times per second is trivial.

You are surely right, Tim. Measuring did not detect the difference. I just
see my tasktray collecting more and more icons of programs which just do a
very tiny amount of work every half a second, and ... as them getting many,
the computer begins to feel more sluggish.

Harald

 

[Dennis Lee Bieber]

see my tasktray collecting more and more icons of programs which just do a
very tiny amount of work every half a second, and ... as them getting many,
the computer begins to feel more sluggish.

Is this under Windows?

I've never seen Python threads creating icons. Are your threads
spawning (os.system, the popen* family, etc.) separate programs? It may
be that those spawns are not closing down properly (or the program
/they/ start never exits).

Or were you creating a new thread during each "activation"... If
the latter, you might need to have something doing a thread.join to
clean them out when they finish the work. Maybe a queue that each thread
puts its ID onto when finished, so the main program can get that ID and
perform a join. Daemons go away when the main program does, but
otherwise are designed to always be present to accept input (queue,
perhaps) If you start five daemons, but each only does one pass at a
task you may still have five idle daemons hanging around.

--

 

[Tim Peters]

[Tim Peters, on waking up a thread to check a queue 20 times a second]
[Harald Massa]

You are surely right, Tim. Measuring did not detect the difference. I just
see my tasktray collecting more and more icons of programs which just do a
very tiny amount of work every half a second, and ... as them getting many,
the computer begins to feel more sluggish.

Those are quite different, and they do more work than you may think.
For example, if a program puts an icon in the Windows systray, then it
has a graphical UI too, and has to run a Windows message pump to
detect stuff like "is the mouse hovering over my icon now? did they
click on me? did they right-click on me? did they double-click on
me?". Apps in the systray often do a lot more than just that too.
For example, one app in my systray right now hooks itself into the
Windows clipboard chain, and intercepts (and so also slows, by some
amount I couldn't care less about) all clipboard operations. Another
hooks itself into the file system at a low level, tracking all disk
changes, and so slows (by a measurable amount!) all disk writes.
Heck, the little systray icon showing the pressure on my laptop's
touchpad represents a process that has read over a billion bytes since
I last booted. Etc.

A Python thread waking up to see whether a Python Queue has something
in it does none of that stuff. If you have hundreds of Python threads
doing this, then you can start to think about maybe worrying when it
gets to thousands <wink>.

 

[Harald Massa]

Dennis,

Is this under Windows?
I've never seen Python threads creating icons.

For sure you have not. It is quite some work to create task-tray icons
using python. A simple thread hopefully will never do that.

The problem is not that severe than you seem to have got the impression
from my talks. I just am very housewifely about computer ressources. And
the programms in the systray are different programs, most not in Python.
Just used that as an explanation, WHY I am looking at "cycles that get
wasted waiting for nothing"

Harald

 

[Dennis Lee Bieber]

The problem is not that severe than you seem to have got the impression
from my talks. I just am very housewifely about computer ressources. And
the programms in the systray are different programs, most not in Python.
Just used that as an explanation, WHY I am looking at "cycles that get
wasted waiting for nothing"

Acknowledged -- my initial read was that you were seeing some
icon multiplying with each "activation".

--

 

[Harald Massa]

Tim,

as them getting many, the computer begins to feel more sluggish.

Those are quite different, and they do more work than you may think.
For example, if a program puts an icon in the Windows systray, then it
has a graphical UI too, and has to run a Windows message pump to
detect stuff like "is the mouse hovering over my icon now? did they
click on me?

My impression from reading dispatcher-code was more in the area of:

a=get_message_from_windows()

if a==waaaah_they_clicked_on_me:
give_it_to_them_REAL_hard()
elif a==they_hover_over_me:
do_some_hover_action()

with "get_message_from_windows()" silently and cycle-free blocking until
windows decides to distribute a message. Something like a blocking
socket, which makes a program wait.

A Python thread waking up to see whether a Python Queue has something
in it does none of that stuff. If you have hundreds of Python threads
doing this, then you can start to think about maybe worrying when it
gets to thousands <wink>.

There was a young monk helping the wise ZEN-master taking a bath. After
he filled the bath-tub for the master, he had some water left, which he
poured away. The master got very harsh to him and asked him, who allowed
him to spill that water.
The young monk got some enlightenment and learned that this is some
essence of ZEN: to cherish every drop, not to waste anything; and changed
his name to "drop of water"

So please excuse my austerity, I also did not know "Queue.get(200)"
and feared to use:

while True:
time.sleep(0.5)
if Queue.empty():
pass
else
break

which really takes some Python bytecodes to toast some cycles )))

Thank you very much for your detailled explanation,

Harald

 

[Jeff Shannon]

My impression from reading dispatcher-code was more in the area of:

a=get_message_from_windows()

if a==waaaah_they_clicked_on_me:
give_it_to_them_REAL_hard()
elif a==they_hover_over_me:
do_some_hover_action()

with "get_message_from_windows()" silently and cycle-free blocking until
windows decides to distribute a message. Something like a blocking
socket, which makes a program wait.

Essentially true, but Windows sends a lot more messages than most people
realize, even to "dormant" apps sitting in the system tray. Even if you
code your own windowproc and handle a bunch of messages yourself, the
vast majority of Windows messages get passed on to DefWindowProc()
(provided by Windows).

Of course, the point is that in practice, it hardly matters. On any
reasonably recent PC, handling a few hundred messages every second is
trivial. (As long as the message handlers don't access "slow"
resources, at least...)

Jeff Shannon
Technician/Programmer
Credit International