Strange threading behaviour

[Rotwang]

Hi all, I'm using Python 2.7.2 on Windows 7 and a module I've written is
acting strangely. I can reproduce the behaviour in question with the
following:

--- begin bugtest.py ---

import threading, Tkinter, os, pickle

class savethread(threading.Thread):
def __init__(self, value):
threading.Thread.__init__(self)
self.value = value
def run(self):
print 'Saving:',
with open(os.path.join(os.getcwd(), 'bugfile'), 'wb') as f:
pickle.dump(self.value, f)
print 'saved'

class myclass(object):
def gui(self):
root = Tkinter.Tk()
root.grid()
def save(event):
savethread(self).start()
root.bind('s', save)
root.wait_window()

m = myclass()
m.gui()

--- end bugtest.py ---


Here's the problem: suppose I fire up Python and type

and then click on the Tk window that spawns and press 's'. Then
'Saving:' gets printed, and an empty file named 'bugfile' appears in my
current working directory. But nothing else happens until I close the Tk
window; as soon as I do so the file is written to and 'saved' gets
printed. If I subsequently type

and then click on the resulting window and press 's', then 'Saving:
saved' gets printed and the file is written to immediately, exactly as I
would expect. Similarly if I remove the call to m.gui from the module
and just call it myself after importing then it all works fine. But it
seems as if calling the gui within the module itself somehow stops
savethread(self).run from finishing its job while the gui is still alive.

Can anyone help?

 

[Dave Angel]

Hi all, I'm using Python 2.7.2 on Windows 7 and a module I've written
is acting strangely. I can reproduce the behaviour in question with
the following:

--- begin bugtest.py ---

import threading, Tkinter, os, pickle

class savethread(threading.Thread):
def __init__(self, value):
threading.Thread.__init__(self)
self.value = value
def run(self):
print 'Saving:',
with open(os.path.join(os.getcwd(), 'bugfile'), 'wb') as f:
pickle.dump(self.value, f)
print 'saved'

class myclass(object):
def gui(self):
root = Tkinter.Tk()
root.grid()
def save(event):
savethread(self).start()
root.bind('s', save)
root.wait_window()

m = myclass()
m.gui()

--- end bugtest.py ---


Here's the problem: suppose I fire up Python and type


and then click on the Tk window that spawns and press 's'. Then
'Saving:' gets printed, and an empty file named 'bugfile' appears in
my current working directory. But nothing else happens until I close
the Tk window; as soon as I do so the file is written to and 'saved'
gets printed. If I subsequently type


and then click on the resulting window and press 's', then 'Saving:
saved' gets printed and the file is written to immediately, exactly as
I would expect. Similarly if I remove the call to m.gui from the
module and just call it myself after importing then it all works fine.
But it seems as if calling the gui within the module itself somehow
stops savethread(self).run from finishing its job while the gui is
still alive.

Can anyone help?

I did not study your code, as I'm not very familiar with tkinter.
However, I think I know your problem:

You do not want to try to start up threads from within a import. An
import is special, and somehow blocks threading while it's running.

Consequently, a module should not try to do anything too fancy from
within its top-level code. Add in the traditional:

def main():
m = myclass()
m.gui()

if __name__ == "__main__":
main()

and just run it from the command line, as python bugtest.py

And if you want to run it from a interactive python session, do the call
to main() after importing it:

 

[Temia Eszteri]

Hi all, I'm using Python 2.7.2 on Windows 7 and a module I've written is
acting strangely. I can reproduce the behaviour in question with the
following:

--- begin bugtest.py ---

import threading, Tkinter, os, pickle

class savethread(threading.Thread):
def __init__(self, value):
threading.Thread.__init__(self)
self.value = value
def run(self):
print 'Saving:',
with open(os.path.join(os.getcwd(), 'bugfile'), 'wb') as f:
pickle.dump(self.value, f)
print 'saved'

class myclass(object):
def gui(self):
root = Tkinter.Tk()
root.grid()
def save(event):
savethread(self).start()
root.bind('s', save)
root.wait_window()

m = myclass()
m.gui()

--- end bugtest.py ---


Here's the problem: suppose I fire up Python and type


and then click on the Tk window that spawns and press 's'. Then
'Saving:' gets printed, and an empty file named 'bugfile' appears in my
current working directory. But nothing else happens until I close the Tk
window; as soon as I do so the file is written to and 'saved' gets
printed. If I subsequently type


and then click on the resulting window and press 's', then 'Saving:
saved' gets printed and the file is written to immediately, exactly as I
would expect. Similarly if I remove the call to m.gui from the module
and just call it myself after importing then it all works fine. But it
seems as if calling the gui within the module itself somehow stops
savethread(self).run from finishing its job while the gui is still alive.

Can anyone help?

Try appending the dump command with f.flush() and os.fsync().

~Temia

 

[Temia Eszteri]

Try appending the dump command with f.flush() and os.fsync().

~Temia

Actually, wait, no. The behavior you're describing is indicating that
the thread in question isn't even getting a chance to execute at all.
I'd recommend going with Dave's idea then, see how it pans out.

Still, a good set of commands to keep in mind.

~Temia

 

[Dieter Maurer]

Hi all, I'm using Python 2.7.2 on Windows 7 and a module I've written
is acting strangely. I can reproduce the behaviour in question with
the following:

--- begin bugtest.py ---

import threading, Tkinter, os, pickle

class savethread(threading.Thread):
def __init__(self, value):
threading.Thread.__init__(self)
self.value = value
def run(self):
print 'Saving:',
with open(os.path.join(os.getcwd(), 'bugfile'), 'wb') as f:
pickle.dump(self.value, f)
print 'saved'

class myclass(object):
def gui(self):
root = Tkinter.Tk()
root.grid()
def save(event):
savethread(self).start()
root.bind('s', save)
root.wait_window()

m = myclass()
m.gui()

--- end bugtest.py ---


Here's the problem: suppose I fire up Python and type


and then click on the Tk window that spawns and press 's'. Then
'Saving:' gets printed, and an empty file named 'bugfile' appears in
my current working directory. But nothing else happens until I close
the Tk window; as soon as I do so the file is written to and 'saved'
gets printed. If I subsequently type


and then click on the resulting window and press 's', then 'Saving:
saved' gets printed and the file is written to immediately, exactly as
I would expect. Similarly if I remove the call to m.gui from the
module and just call it myself after importing then it all works
fine. But it seems as if calling the gui within the module itself
somehow stops savethread(self).run from finishing its job while the
gui is still alive.

Can anyone help?

It looks as if some waiting operation
in the "wait_window" call did not release the GIL (the "Global Interpreter
Lock" which ensures that at most one Python thread can run at a given
time and protects the Python data structures such as the reference counts
and interpreter state).

In this case, you could expect some non-deterministic behaviour.
If your thread is fast enough to finish before the internal
activity inside "wait_window" gets the GIL again,
everything completes immediately; otherwise, things complete
only after the internal waiting ends and Python code is again executed.


It might well be possible that "TKinter" has not been designed for
a multi threaded environment; alternatively, there might be a bug.
If "TKinter" truely supports multithreaded applications, any call
to "tk" would need to release the GIL and any callback into Python
reacquire it. Strange things of the kind you observe could happen
when this is forgotten at a single place.

 

[Dennis Lee Bieber]

import threading, Tkinter, os, pickle

class savethread(threading.Thread):
def __init__(self, value):
threading.Thread.__init__(self)
self.value = value
def run(self):
print 'Saving:',
with open(os.path.join(os.getcwd(), 'bugfile'), 'wb') as f:
pickle.dump(self.value, f)
print 'saved'

class myclass(object):
def gui(self):
root = Tkinter.Tk()
root.grid()
def save(event):
savethread(self).start()
root.bind('s', save)

Each time your "s" runs you are creating a NEW thread.

root.wait_window()

m = myclass()
m.gui()

--- end bugtest.py ---


Here's the problem: suppose I fire up Python and type

Running in the interpreter rather than as a program...

and then click on the Tk window that spawns and press 's'. Then
'Saving:' gets printed, and an empty file named 'bugfile' appears in my
current working directory. But nothing else happens until I close the Tk
window; as soon as I do so the file is written to and 'saved' gets
printed. If I subsequently type


and then click on the resulting window and press 's', then 'Saving:
saved' gets printed and the file is written to immediately, exactly as I
would expect. Similarly if I remove the call to m.gui from the module
and just call it myself after importing then it all works fine. But it
seems as if calling the gui within the module itself somehow stops
savethread(self).run from finishing its job while the gui is still alive.

Since you never kill the previous thread, you might have a binding
to left-over multiple threads; at the least, the Python scheduler may be
swapping between threads.

The task swapping probably gives the threads time to flush output.
With just the first run thread condition, the process may go back to the
Tk mainloop and not return to the thread to completely flush data.

Especially with the use of .wait_window(), which is supposed to lock
the application until the window is destroyed. If overly aggressive, it
might even lock up swapping back to the thread to allow it to flush.

Can anyone help?

Don't run from the interactive interpreter?

 

[Rotwang]

I did not study your code, as I'm not very familiar with tkinter.
However, I think I know your problem:

You do not want to try to start up threads from within a import. An
import is special, and somehow blocks threading while it's running.

That would explain it.

Consequently, a module should not try to do anything too fancy from
within its top-level code. Add in the traditional:

def main():
m = myclass()
m.gui()

if __name__ == "__main__":
main()

and just run it from the command line, as python bugtest.py

In fact, running the module directly from the command line makes it work
properly as-is.

Thanks for your post.

 

[Rotwang]

Actually, wait, no. The behavior you're describing is indicating that
the thread in question isn't even getting a chance to execute at all.

Surely not? The fact that it got as far as executing the first print
statement and creating the file suggests that the problem happens
somewhere between the dump command and leaving the with statement,
doesn't it? The suggestion didn't make any difference, though.

I'd recommend going with Dave's idea then, see how it pans out.

Still, a good set of commands to keep in mind.

Thanks, will do.

 

[Rotwang]

[...]

Can anyone help?

It looks as if some waiting operation
in the "wait_window" call did not release the GIL (the "Global Interpreter
Lock" which ensures that at most one Python thread can run at a given
time and protects the Python data structures such as the reference counts
and interpreter state).

In this case, you could expect some non-deterministic behaviour.
If your thread is fast enough to finish before the internal
activity inside "wait_window" gets the GIL again,
everything completes immediately; otherwise, things complete
only after the internal waiting ends and Python code is again executed.

OK, that makes sense - it could certainly explain why the print and open
commands worked fine but the thread got stuck at pickle.dump. Thanks.

 

[Rotwang]

Each time your "s" runs you are creating a NEW thread.

Yes, that's the whole point: I want the object's state to be saved in
the background while I can continue using the GUI without having to wait
(in the original module where I noticed the problem, the thread pickles
a copy of the object whose gui method created it, rather than the object
itself - that way I can make changes to the object without interfering
with the work being done by pickle.dump).

Running in the interpreter rather than as a program...

Since you never kill the previous thread, you might have a binding
to left-over multiple threads; at the least, the Python scheduler may be
swapping between threads.

Which previous thread do you mean? The problem happens when save has
only been called once.

The task swapping probably gives the threads time to flush output.
With just the first run thread condition, the process may go back to the
Tk mainloop and not return to the thread to completely flush data.

Especially with the use of .wait_window(), which is supposed to lock
the application until the window is destroyed. If overly aggressive, it
might even lock up swapping back to the thread to allow it to flush.


Don't run from the interactive interpreter?

Thanks.

 

[Dave Angel]

That would explain it.



In fact, running the module directly from the command line makes it
work properly as-is.

But if you leave it as-is, then you can't run it from the interactive
interpreter.

Thanks for your post.

just for completeness, see link;
http://docs.python.org/library/threading.html

" ... an import should not have the side effect of spawning a new thread
and then waiting for that thread in any way..."

 

[Rotwang]

[...]

In fact, running the module directly from the command line makes it
work properly as-is.

But if you leave it as-is, then you can't run it from the interactive
interpreter.

Right, thanks.

just for completeness, see link;
http://docs.python.org/library/threading.html

" ... an import should not have the side effect of spawning a new thread
and then waiting for that thread in any way..."

Ah, I missed that.