Spawing a thread and printing dots until it finishes

[sophie_newbie]

Hi, I'm trying to write a piece of code that spawns a thread and
prints dots every half second until the thread spawned is finished.
Code is
something like this:

import threading
class MyThread ( threading.Thread ):
def run ( self ):
myLongCommand()...

import time

t = MyThread()
t.start()

while t.isAlive():
print "."
time.sleep(.5)

print "OK"

The thing is this doesn't print a dot every half second. It just
pauses for ages until the thread is finished and prints prints ".OK".
But if I take out the "time.sleep(.5)" line it will keep printing dots
really fast until the thread is finished. So it looks like its the
time.sleep(.5) bit that is messing this up somehow?

Any ideas?

Thanks!

 

[D'Arcy J.M. Cain]

import threading
class MyThread ( threading.Thread ):
def run ( self ):
myLongCommand()...

import time

t = MyThread()
t.start()

while t.isAlive():
print "."
time.sleep(.5)

print "OK"

The thing is this doesn't print a dot every half second. It just
pauses for ages until the thread is finished and prints prints ".OK".
But if I take out the "time.sleep(.5)" line it will keep printing dots
really fast until the thread is finished. So it looks like its the
time.sleep(.5) bit that is messing this up somehow?

We know that your main routine gives up the processor but without a
full definition of MyThread how do we know that it ever does? I
suspect that it hits sleep once, if at all, and then goes to the final
print statement. In fact, I suspect that this is not exactly what you
tried anyway. This code would not have printed ".OK" whether it
entered the loop or not. It could have printed this;

..
OK

because the print statement in the loop will print a dot on a line by
itself.

When looking for these sorts of answers you should really try to create
a smallest program that exhibits the behaviour you are questioning and
then cut and paste the entire script into your message unedited. Often
enough you will even answer your own question in the process.

 

[sophie_newbie]

We know that your main routine gives up the processor but without a
full definition of MyThread how do we know that it ever does? I
suspect that it hits sleep once, if at all, and then goes to the final
print statement. In fact, I suspect that this is not exactly what you
tried anyway. This code would not have printed ".OK" whether it
entered the loop or not. It could have printed this;

.
OK

because the print statement in the loop will print a dot on a line by
itself.

When looking for these sorts of answers you should really try to create
a smallest program that exhibits the behaviour you are questioning and
then cut and paste the entire script into your message unedited. Often
enough you will even answer your own question in the process.

"myLongCommand()... " is a call to an function in R (the statistical
programming language) via Rpy (A python module that allows calls to
R). The call takes a couple of minutes to execute. I'm trying to build
a web front end to this R function and instead of the user looking at
a blank screen for 2-3 mins, I want to print dots to let them feel
like the program isn't hanging.

What I am saying is that without the "time.sleep(.5)" line, the above
code will print dots on the screen continuously for 2-3 mins, filling
it up with a ridiculous ammount of dots.

Whereas with the time.sleep line, instead of pausing for half a second
between dots, its seems to print, as you correctly pointed out:

..
OK

With a pause of 2-3 minutes between the . and the OK.

I hope that clears things up a little. I haven't the faintest idea why
the code above doesn't work but hope someone has an idea. It wouldn't
be something to do with python not being able to handle multiple
threads at the same time or something? I hope there is a workaround.

 

[D'Arcy J.M. Cain]

"myLongCommand()... " is a call to an function in R (the statistical
programming language) via Rpy (A python module that allows calls to
R). The call takes a couple of minutes to execute. I'm trying to build
a web front end to this R function and instead of the user looking at
a blank screen for 2-3 mins, I want to print dots to let them feel
like the program isn't hanging.

What I am saying is that without the "time.sleep(.5)" line, the above
code will print dots on the screen continuously for 2-3 mins, filling
it up with a ridiculous ammount of dots.

Does it ever finish? It seems to me that it would loop forever if you
never give up the processor. Remember, threads are not processes.

Whereas with the time.sleep line, instead of pausing for half a second
between dots, its seems to print, as you correctly pointed out:

.
OK

With a pause of 2-3 minutes between the . and the OK.

Exactly. It prints the first dot and then gives up the processor to
your other thread. That thread runs for 2 to 3 minutes and then
completes giving up the processor. At that point your sleep has slept
for at least .5 seconds so the first thread if free to run. It then
checks the condition of the loop, sees that it is finished and
continues on the the next statement which prints the "OK" and exits.

I hope that clears things up a little. I haven't the faintest idea why
the code above doesn't work but hope someone has an idea. It wouldn't
be something to do with python not being able to handle multiple
threads at the same time or something? I hope there is a workaround.

I think that there are two things that you need to wrap your head
around before understanding what is happening here. First, threads are
NOT pre-emptive. Unless your thread gives up the processor it will run
forever. The sleep call is one way to give up the processor.

Second, sleep() does not return as soon as the time given has expired.
The argument is the MINIMUM amount of time that it waits. After that
the thread that slept is put back onto the run queue and is now a
candidate to be given the processor. Your other thread still has to
give up the processor before it can run again and even then there may
be other threads on the queue ahead of yours.

So, does your thread ever give up the processor other than by dying?

 

[Gabriel Genellina]

"myLongCommand()... " is a call to an function in R (the statistical
programming language) via Rpy (A python module that allows calls to
R). The call takes a couple of minutes to execute. I'm trying to build
a web front end to this R function and instead of the user looking at
a blank screen for 2-3 mins, I want to print dots to let them feel
like the program isn't hanging.

What I am saying is that without the "time.sleep(.5)" line, the above
code will print dots on the screen continuously for 2-3 mins, filling
it up with a ridiculous ammount of dots.

Whereas with the time.sleep line, instead of pausing for half a second
between dots, its seems to print, as you correctly pointed out:

.
OK

With a pause of 2-3 minutes between the . and the OK.

A possible explanation is that the RPy call does not release the GIL; once the main thread loses it due to sleep, it can never reacquire it again until the RPy call finishes.
Try contacting the RPy author, or perhaps there is a specific mailing list for RPy questions.

 

[Diez B. Roggisch]

I think that there are two things that you need to wrap your head

around before understanding what is happening here. First, threads are
NOT pre-emptive. Unless your thread gives up the processor it will run
forever. The sleep call is one way to give up the processor.

That is not correct, at least not on usual OSes. The posix-threads as
well as windows threads *are* preemptive.

Second, sleep() does not return as soon as the time given has expired.
The argument is the MINIMUM amount of time that it waits. After that
the thread that slept is put back onto the run queue and is now a
candidate to be given the processor. Your other thread still has to
give up the processor before it can run again and even then there may
be other threads on the queue ahead of yours.

So, does your thread ever give up the processor other than by dying?

It shouldn't need to. It will be rescheduled.

The code looks ok to me - the problem seems to be in the R-Python as
Gabriel pointed out.

Diez

 

[Diez B. Roggisch]

For a web front end you wouldn't go this route at all. You would get a
progressive .GIF file that gets loaded into the client's browser and shows
"activity" while the server does its thing. When the browser refreshes
(after the server application completes) it would go away. You can't
update a client's browser by writing dots to anything.

Yes and no. You are right of course that the dot-thread is not working
that way. But if you replace the dot-thread with the
http-request-thread, the question remains: why is it blocking? Your
approach doesn't tackle that.

If nothing else helps, a subprocess must be spawned.

Diez

 

[D'Arcy J.M. Cain]

When did that change take place?

As I recall, the Python interpreter is supposed to preempt a (pure
Python) thread after some 10 or 100 (I think the value changed some
years ago) bytecodes.

It sounds to me like you are talking about when the interpreter grabs
and releases the GIL but I was talking about when it releases the
processor. I certainly never said that sleep() was the only way to
release the processor. I was not actually aware that running a certain
number of bytecodes was another way but as I said, we never saw the
code for the thread so we don't know what it is doing or what
extensions it might be calling. It *may* be pure Python but we don't
know.

 

[sophie_newbie]

Hi, I'm trying to write a piece of code that spawns a thread and
prints dots every half second until the thread spawned is finished.
Code is
something like this:

import threading
class MyThread ( threading.Thread ):
def run ( self ):
myLongCommand()...

import time

t = MyThread()
t.start()

while t.isAlive():
print "."
time.sleep(.5)

print "OK"

The thing is this doesn't print a dot every half second. It just
pauses for ages until the thread is finished and prints prints ".OK".
But if I take out the "time.sleep(.5)" line it will keep printing dots
really fast until the thread is finished. So it looks like its the
time.sleep(.5) bit that is messing this up somehow?

Any ideas?

Thanks!

As it happens I've managed to come up with a solution to this problem
using a subprocess rather than a thread. Its not exactly rocket
science but I thought I'd post it anyway. There are 3 files:

########## dots.py #######################
# a script to print a dot every half second until it is terminated

import time
import sys

while 1 == 1:

sys.stdout.write(".")
sys.stdout.flush()

time.sleep(.5)


######### PrintDots.py ######################

# This is a simple class to spawn off another process that prints dots
repeatedly on screen
# when printDots() is called and stops when stopDots is called. It is
useful in cgi-scripts
# where you may want to let the user know that something is happening,
rather than looking
# at a blank screen for a couple of minutes.

import time
import subprocess
import os
from signal import SIGTERM

class PrintDots:

# the constructor, called when an object is created.
def __init__(self):

self.pid = 0

# the location of the script that prints the dots
self.dotsScript = "dots.py"

def printDots(self):

self.pid = subprocess.Popen( [ "python", self.dotsScript] ).pid

def stopDots(self):

os.kill(self.pid, SIGTERM)



############ mainFile.py ##############################
# The above can then be called from any cgi-script as follows

from PrintDots import PrintDots
p = PrintDots()
p.printDots()
print "Doing R Stuff"
my_Call_To_R_That_Takes_A_Long_Time()
p.stopDots()
print "OK"

############

And low and behold dots are printed on screen every half second while
python is talking to R, with an output like this:

Doing R Stuff.................................OK

 

[sophie_newbie]

Hi, I'm trying to write a piece of code that spawns a thread and
prints dots every half second until the thread spawned is finished.
Code is
something like this:

import threading
class MyThread ( threading.Thread ):
def run ( self ):
myLongCommand()...

import time

t = MyThread()
t.start()

while t.isAlive():
print "."
time.sleep(.5)

print "OK"

The thing is this doesn't print a dot every half second. It just
pauses for ages until the thread is finished and prints prints ".OK".
But if I take out the "time.sleep(.5)" line it will keep printing dots
really fast until the thread is finished. So it looks like its the
time.sleep(.5) bit that is messing this up somehow?

Any ideas?

Thanks!

As it happens I've managed to come up with a solution to this problem
using a subprocess rather than a thread. Its not exactly rocket
science but I thought I'd post it anyway. There are 3 files:

########## dots.py #######################
# a script to print a dot every half second until it is terminated

import time
import sys

while 1 == 1:

sys.stdout.write(".")
sys.stdout.flush()

time.sleep(.5)

######### PrintDots.py ######################

# This is a simple class to spawn off another process that prints dots
repeatedly on screen
# when printDots() is called and stops when stopDots is called. It is
useful in cgi-scripts
# where you may want to let the user know that something is happening,
rather than looking
# at a blank screen for a couple of minutes.

import time
import subprocess
import os
from signal import SIGTERM

class PrintDots:

# the constructor, called when an object is created.
def __init__(self):

self.pid = 0

# the location of the script that prints the dots
self.dotsScript = "dots.py"

def printDots(self):

self.pid = subprocess.Popen( [ "python", self.dotsScript] ).pid

def stopDots(self):

os.kill(self.pid, SIGTERM)

############ mainFile.py ##############################
# The above can then be called from any cgi-script as follows

from PrintDots import PrintDots
p = PrintDots()
p.printDots()
print "Doing R Stuff"
my_Call_To_R_That_Takes_A_Long_Time()
p.stopDots()
print "OK"

############

And low and behold dots are printed on screen every half second while
python is talking to R, with an output like this:

Doing R Stuff.................................OK

Whoops that last bit of code should read as follows:

from PrintDots import PrintDots
p = PrintDots()
print "Doing R Stuff"
p.printDots()
my_Call_To_R_That_Takes_A_Long_Time()
p.stopDots()
print "OK"