wxPython and threads again

[perchef]

Hi,

I have several files to download and a GUI to update. I know this is a
frequently asked question but i can't find an appropriate solution.
My Downloader extends threading.Thread and update a wx.Gauge in GUI
during the process.

for src in urls:
downloader = Downloader( src, destination, GUI )
downloader.start()

#work with the downloaded files...

If i don't use a downloader.join() in this for loop, I launch several
threads at the same time and so my wx.Gauge is bouncing up and down.
If i do add the downloader.join() my GUI is no more updated ( in fact,
nothing appears, it's frozen )
How can I wait the end of the thread and also be able to update the GUI
?
( I have to wait, otherwise I will work uncompleted files )

Any way to work around it?
Thanks.

 

[David E. Konerding DSD staff]

Hi,

I have several files to download and a GUI to update. I know this is a
frequently asked question but i can't find an appropriate solution.
My Downloader extends threading.Thread and update a wx.Gauge in GUI
during the process.

for src in urls:
downloader = Downloader( src, destination, GUI )
downloader.start()

#work with the downloaded files...

If i don't use a downloader.join() in this for loop, I launch several
threads at the same time and so my wx.Gauge is bouncing up and down.
If i do add the downloader.join() my GUI is no more updated ( in fact,
nothing appears, it's frozen )
How can I wait the end of the thread and also be able to update the GUI

Well, the constraints are this: the main thread is running the wx main loop,
and thus, cannot block by calling join on the downloader thread. Further, calling
wx from a thread other than the one running the event loop is deep voodoo and should typically
be avoided.

You need another way to pass completion information between the downloader
thread and the main thread; the simplest way is to define a custom wx
Event, and wxPostEvent from the downloader thread when it completes (
and when the gauge should be updated). wxPostEvent is safe to call from non-eventloop threads.
The main thread's wx event loop just spins, properly updating all other
parts of the GUI, and receiving events from the downloader thread.

ANother approach is to have a thread-safe Queue and have the main thread/event loop
poll the queue with queue.get_nowait() periodically (typically 0.1-1 sec).
The downloader thread shares the queue object and puts data structures (typically
class instances, strings, or ints) that indicate status updates.

The easiest approach, though, is to use the threadedselectreactor in Twisted (you need
to check the HEAD branch out with subversion, because that reactor isn't included in any releases).
With threadedselectreactor, it's easy to incorporate both the GUI event loop and the twisted reactor.
Twisted already includes lots of code for doing asynchronous callback-style IO for
IO bound processes like downloading. Further, you don't even think in an explicitly threaded way-
createing a whole thread just to manage a download process which is motly IO and a little bookkeeping is
silly. Twisted's approach just makes a lot more sense and simplifies the code too.

Dave

 

[Bryan Olson]

David E. Konerding DSD staff wrote:
[...]

> You need another way to pass completion information between the downloader
> thread and the main thread; the simplest way is to define a custom wx
> Event, and wxPostEvent from the downloader thread when it completes (
> and when the gauge should be updated). wxPostEvent is safe to call from non-eventloop threads.
> The main thread's wx event loop just spins, properly updating all other
> parts of the GUI, and receiving events from the downloader thread.
>
> ANother approach is to have a thread-safe Queue and have the main thread/event loop
> poll the queue with queue.get_nowait() periodically (typically 0.1-1 sec).
> The downloader thread shares the queue object and puts data structures (typically
> class instances, strings, or ints) that indicate status updates.

The way-cool things to transmit, in either the queue or the
event data, are tuples of:

(func, args, kwargs)

The so-called-'main' thread gets these, and blindly calls
func(*args, **kwargs). Since only the main thread can safely
update the GUI, other threads pass GUI-updating functions to be
called by the main thread.

The technique is beautifully general. The worker thread does
it's long, blocking operations independently, and when it needs
to update the GUI it sends the main thread a quick, non-blocking
function.

In wxPython, custom events can carry arbitrary data, so the easy
thing to do is just pass the (func, args, kwargs) across with
wxPostEvent (or so I've read; I'm not a wxPython user).

TkInter has no equivalent to wxPostEvent. Contrary to popular
belief, TkInter's event_generate is not thread-safe. The usual
TkInter solution is a queue, which the main thread periodically
polls via the 'after' function.

> The easiest approach, though, is to use the threadedselectreactor in Twisted (you need
> to check the HEAD branch out with subversion, because that reactor

isn't included in any releases).

> With threadedselectreactor, it's easy to incorporate both the GUI

event loop and the twisted reactor.

> Twisted already includes lots of code for doing asynchronous callback-style IO for
> IO bound processes like downloading. Further, you don't even think in an explicitly threaded way-
> createing a whole thread just to manage a download process which is

motly IO and a little bookkeeping is

> silly. Twisted's approach just makes a lot more sense and simplifies

the code too.

I couldn't disagree more about that being easier and simplifying
the code. "Creating a whole thread" is trivial.

 

[Peter Hansen]

Further, calling wx from a thread other than the one running the event
> loop is deep voodoo and should typically be avoided.

"Typically"? Let's just say "always" and maybe use the phrase "certain
to corrupt wx and crash the app" instead of "deep voodoo". At least
that way the OP won't waste time experimenting...

You need another way to pass completion information between the downloader
thread and the main thread; the simplest way is to define a custom wx
Event, and wxPostEvent from the downloader thread when it completes (
and when the gauge should be updated). wxPostEvent is safe to call from non-eventloop threads.
The main thread's wx event loop just spins, properly updating all other
parts of the GUI, and receiving events from the downloader thread.

Even simpler for some purposes is wx.CallAfter(), which provides the
asynchronous performance of wxPostEvent with the "beautifully general"
approach of passing callables through a Queue which Bryan Olson
described in his post. (That is, you don't need to do polling with a
non-blocking get() on the Queue this way.)

-Peter

 

[perchef]

thanks for all these advices.
I think a custom event will be the way to go.
for the moment I use something _really_ ugly : a mix between GUI,
threads and recursive fonctions.

 

[Bryan Olson]

>
> "Typically"? Let's just say "always" and maybe use the phrase "certain
> to corrupt wx and crash the app" instead of "deep voodoo". At least
> that way the OP won't waste time experimenting...

Come to think of it, wouldn't it be a good idea for a GUI
toolkit to to do something like:

import thread

# ...

def WhateverToolKitInitFunction(*args):
global _thread_of_record
_thread_of_record = thread.get_ident()
# ...


def check_thread():
if thread.get_ident() != _thread_of_record:
raise RuntimeError('Attempt to update GUI from foreign
thread.')


And then begin each non-thread-safe function like:


def SomeUpdateFunction(*args):
check_thread()
# ...


[...]

> Even simpler for some purposes is wx.CallAfter(),

Ah, Nice. Same method under the hood, but hides the complexity.

 

[Christopher Subich]

The easiest approach, though, is to use the threadedselectreactor in Twisted (you need
to check the HEAD branch out with subversion, because that reactor isn't included in any releases).
With threadedselectreactor, it's easy to incorporate both the GUI event loop and the twisted reactor.
Twisted already includes lots of code for doing asynchronous callback-style IO for
IO bound processes like downloading. Further, you don't even think in an explicitly threaded way-
createing a whole thread just to manage a download process which is motly IO and a little bookkeeping is
silly. Twisted's approach just makes a lot more sense and simplifies the code too.

Or, don't use threadedselectreactor, but instead just use normal
threading and reactor.callFromThread.

 

[David E. Konerding DSD staff]

isn't included in any releases).
event loop and the twisted reactor.
motly IO and a little bookkeeping is
the code too.

I couldn't disagree more about that being easier and simplifying
the code. "Creating a whole thread" is trivial.

I've done both styles. Actually, I greatly prefer the single threaded approach now; conceptually,
the threaded approach is very simple, but your program ends up getting complex because the data-passing
infrastructure. And you end up structuring your logic in somewhat more convoluted ways. And
most people have such a hard time dealing with data synch between threads...

Dave

 

[David E. Konerding DSD staff]

"Typically"? Let's just say "always" and maybe use the phrase "certain
to corrupt wx and crash the app" instead of "deep voodoo". At least
that way the OP won't waste time experimenting...

Not really certain:
http://wxwidgets.org/manuals/2.6.1/wx_threadfunctions.html#threadfunctions

This strongly suggests you can arbitrarily grab the wx GUI lock and call GUI
functions from any thread. It's still voodoo. But we're adults here, and practicing
voodoo isn't proscribed.

Even simpler for some purposes is wx.CallAfter(), which provides the
asynchronous performance of wxPostEvent with the "beautifully general"
approach of passing callables through a Queue which Bryan Olson
described in his post. (That is, you don't need to do polling with a
non-blocking get() on the Queue this way.)

Very good point. I wasn't aware CallAfter had those semantics; I always used it from
the main thread. But won't wx.CallAfter cause a bit of a delay since it will wait until
all pending events are processed and only handle the function afterwards, thus inducing
an extra event loop cycle/redraw?

Dave

 

[Peter Hansen]

http://wxwidgets.org/manuals/2.6.1/wx_threadfunctions.html#threadfunctions

This strongly suggests you can arbitrarily grab the wx GUI lock and call GUI
functions from any thread. It's still voodoo. But we're adults here, and practicing
voodoo isn't proscribed.

Oh, very nice... for my automatic testing requirements anyway. I did
not know those existed...

I agree using this in real code is probably not wise, and you're right
that it is best described as "voodoo".

But won't wx.CallAfter cause a bit of a delay since it will wait until
all pending events are processed and only handle the function afterwards,
> thus inducing an extra event loop cycle/redraw?

Since it's built on PostEvent, it will do whatever happens with other
stuff that is PostEvent-ed, I suppose. Your description is probably
accurate...

-Peter