One thread freezes my whole application

[Michael Zhang]

My project uses Python-2.3.4 + Tkinter + PIL-1.1.4 to retrieve images
from UDP server and display those images on a RedHat 9 box.

On my GUI there are two buttons: "connection button" is to create a
connection to server, listen to any upcoming image packets, and write
those data into a buffer; "display button" is to read the data from the
buffer and display them in a infinite loop.

It's a natual approach to create two threads for each tasks (connection
and display) from the base thread (GUI application).

The callback for the display button implements a Tkinter canvas.

The callback for the connection button is imported from a module created
from a C function using swig-1.3.22.

For test purpose, I tried to invoke the display thread to display a
image from a local file system and then invoke the connection thread to
see if two threads can work separately.

The problem is when I click the connection button and invoke that
connection thread, the whole application (including base thread and
display thread) was frozen. What I expected was when one thread is
listerning the socket, the display thread should be able to continue its
running, and I should be able to invoke other (if any) threads from the
GUI base thread. Afterwards, I had to use ps/kill to clean them up.

I'm quite new to Python. I started to learn Python and those libraries
two months ago just after we chosed this project.

Here is my simplified code segments:

=============== Code segments Start ==============

class displayThread(threading.Thread):
# the thread for displaying images
def __init__(self, master):
threading.Thread.__init__(self)
self.master = master

def run(self):
UI(self.master) # create a Toplevel window to display

class connectionThread(threading.Thread):
# the thread for connecting to server
def __init__(self):
threading.Thread.__init__(self)

def run(self):
Cmodule.create_connection() # Cmodule is created from C


class ControlPanel(Frame):
# base GUI application from which all those threads are invoked
.....
def connect(self):
# callback for connection button
connectionThread().start()

def display(self):
# callback for display button
top = Toplevel()
displayThread(top).start()

=============== Code segments End ==============

What I'm thinking now is above threads code are quite standard and
simple, there must be some other subtle issues beyond my knowledge.

Could someone help me out about those thread stuff? Thanks!

Michael

 

[Anders J. Munch]

The problem is when I click the connection button and invoke that
connection thread, the whole application (including base thread and
display thread) was frozen. What I expected was when one thread is
listerning the socket, the display thread should be able to continue its
running, and I should be able to invoke other (if any) threads from the
GUI base thread. Afterwards, I had to use ps/kill to clean them up.
[...]

def run(self):
Cmodule.create_connection() # Cmodule is created from C

Do I understand you correctly that Cmodule is a C extension?

If create_connection includes C code that does blocking I/O without
releasing the global interpreter lock, your whole application will
block. Perhaps that's what you're seeing.

- Anders

 

[Michael Zhang]

The problem is when I click the connection button and invoke that
connection thread, the whole application (including base thread and
display thread) was frozen. What I expected was when one thread is
listerning the socket, the display thread should be able to continue its
running, and I should be able to invoke other (if any) threads from the
GUI base thread. Afterwards, I had to use ps/kill to clean them up.

[...]

def run(self):
Cmodule.create_connection() # Cmodule is created from C

Do I understand you correctly that Cmodule is a C extension?

If create_connection includes C code that does blocking I/O without
releasing the global interpreter lock, your whole application will
block. Perhaps that's what you're seeing.

Thank you, Anders

Your hint on GIL leads me quite lots of searching and reading on Google.
After several hours confusion (due to those different opinions), I
finally come to a quite simple and amazing solution:

Py_BEGIN_ALLOW_THREADS; # add this line
some_blocking_function_call();
Py_BEGIN_ALLOW_THREADS; # add this line as well

Now I've got what I really expected.

However, there is a new question (rather than a problem). Since my
Cmodule is created using SWIG, if there is any change in the original C
code, I have to run swig again to get new "Cmodule_wrap.c" file, and add
above two lines into that new wrapper file, then build a new module.
It's quite uncomfortable during debugging. Is there any way I can use to
achieve the same goal on Python side code, rather that on C extension
side? Can I just modify the Python code to release that GIL? or Do I
have any misunderstading about GIL, say, the concept about GIL only
applies to C extension rather than Python threading itself?

Michael

 

[Aahz]

Your hint on GIL leads me quite lots of searching and reading on Google.
After several hours confusion (due to those different opinions), I
finally come to a quite simple and amazing solution:

Py_BEGIN_ALLOW_THREADS; # add this line
some_blocking_function_call();
Py_BEGIN_ALLOW_THREADS; # add this line as well

I'm assuming you meant this:

Py_END_ALLOW_THREADS; # add this line as well

However, there is a new question (rather than a problem). Since my
Cmodule is created using SWIG, if there is any change in the original
C code, I have to run swig again to get new "Cmodule_wrap.c" file,
and add above two lines into that new wrapper file, then build a new
module. It's quite uncomfortable during debugging. Is there any way I
can use to achieve the same goal on Python side code, rather that on C
extension side? Can I just modify the Python code to release that GIL?
or Do I have any misunderstading about GIL, say, the concept about GIL
only applies to C extension rather than Python threading itself?

It's a bit more complicated than that. The concept of the GIL applies
to both Python and C extensions; however, only C extensions can release
the GIL.

In other words, only one thread of Python code can execute at any time
because of the GIL -- try to break that and your application *WILL* blow
up. C code that does not use *ANY* Python API may release the GIL to
allow other threads to run. Before starting to use the Python API, the
C code *MUST* re-acquire the GIL.

Traditionally, I/O code is the prime candidate for releasing the GIL.

 

[David Bolen]

However, there is a new question (rather than a problem). Since my
Cmodule is created using SWIG, if there is any change in the original
C code, I have to run swig again to get new "Cmodule_wrap.c" file, and
add above two lines into that new wrapper file, then build a new
module. It's quite uncomfortable during debugging.

One thought - if your wrapping is simple enough, you could make use of
%exception in your .i file to set up a general wrapper that would
release and then re-acquire the GIL around each of the wrapped
functions automatically. So no need to post-process the SWIG output.

Note that this assumes your C routines aren't going to be calling into
the Python core routines, or back into Python. In either of those
cases they'd need to re-acquire the GIL before doing so (except for a
very limited number of Python core functions that can be called
without owning the GIL).

Is there any way I
can use to achieve the same goal on Python side code, rather that on C
extension side? Can I just modify the Python code to release that GIL?
or Do I have any misunderstading about GIL, say, the concept about GIL
only applies to C extension rather than Python threading itself?

No, you can't touch the GIL from the Python side. The GIL is purely a
C-level interpreter construct. If you have Python code executing it
already by definition owns the GIL and can't release it or else it
wouldn't be allowed to be executing Any GIL management needs to be
done at C level.

-- David