urllib hanging in thread

[Russell E. Owen]

I am having trouble with some ftp code. On some platforms it works fine
and on others it reliably hangs.

The code uses urllib to retrieve a file in a background thread, which
runs the following code. The diagnostic message before urllib.urlopen is
printed, but the one after urllib.urlopen is not:

def _getTask(self):
try:
self._toFile = file(self.toPath, 'wb')
print "opening URL"
self._fromFile = urllib.urlopen(self.fromURL)
print "URL opened" # when hung this does not print
self._totBytes =
self._fromFile.info().getheader("Content-Length")
if self._totBytes:
self._totBytes = int(self._totBytes)

self._state = Running

while True:
nextData = self._fromFile.read(8192)
if not nextData:
break
elif self._state == Aborting:
self._cleanup(Aborted)
return
self._readBytes += len(nextData)
self._toFile.write(nextData)
self._cleanup(Done)
except Exception, e:
self._cleanup(Failed, exception = e)

However, I can only make it hang when run as part of a fairly large
Tkinter application. The same code runs fine standalone or as part of a
minimal Tkinter test application. The large application has a few other
threads for communicating via TCP. In all other respects the large app
runs fine; the ftp hangs while everything else keeps cooking along.
Closing the TCP connection (which should stop the other threads) has no
effect on the stalled ftp transfer.

It works fine on my MacOS X 10.3.3 box, including both the built in
python (a variant of 2.3 with aqua Tk) and a python 2.3.3 that I built
as a plain unix python with X11 Tk.

The code hangs on our RedHat linux boxes (python 2.3.3, dunno the linux
version). It also hangs as a standalone MacOS X executable, which is
especially weird since the code runs fine on the same computer before it
gets packaged up as an application.

Any suggestions would be most appreciated.

-- Russell

 

[Russell E. Owen]

I am having trouble with some ftp code. On some platforms it works fine
and on others it reliably hangs....

After some experimentation, I found the hanging was occurring in ftplib.
It delays importing re and compiling two regular expressions until they
are needed, and the thread hangs both on the import and (if I modify
ftplib to import re in advance) on the compilation. Modifying ftplib to
do both jobs in advance (when first imported) fixes the problem.

It seems strange to me that a thread could hang indefinitely trying to
perform these two tasks when the rest of the app seems to be running
fine. Perhaps the thread is simply being starved of cpu cycles, but then
it seems odd that the thread has enough cycles to print diagnostics and
do network communication.

Anyway, I can work around the problem by using ftplib directly instead
of using urllib, and importing my own patched ftplib. But I'm wondering
if anyone has any better suggestions.

-- Russell

 

[Russell E. Owen]

I am having trouble with some ftp code. On some platforms it works fine
and on others it reliably hangs.

[/QUOTE]
....and later followed up with:

After some experimentation, I found the hanging was occurring in ftplib.
It delays importing re and compiling two regular expressions until they
are needed, and the thread hangs both on the import and (if I modify
ftplib to import re in advance) on the compilation. Modifying ftplib to
do both jobs in advance (when first imported) fixes the problem.

Michael Hudson (via a separate forum) suggested the issue was contention
for the "import lock" (which I'd never heard of). He was right.

I originally wrote my app to be run as follows:
% python <tui_root>/TUI/Main.py
Later, I added a script to simplify execution (no need to mess with
PYTHONPATH) and bundling as a double-clickable app:
<tui_root>runtui.py
whose sole contents were:
import TUI.Main

I knew it was ugly, but didn't realize it was dangerous. Apparently when
one executes an app by importing it, the main thread holds the "import
lock", which prevents other threads from importing anything. ftplib
imports re when first used, and was hanging at that point.

What made it tricky to debug was I had forgotten to convert my own
command-line environment to running the app via the new troublesome
script. So I was seeing the problem on some systems but not others.
Anyway, fixing <tui_root>/TUI/Main.py to be run via a function call
fixed the problem, i.e. runtui.py now reads:
import TUI.Main
TUI.Main.runTUI()

So...a big thank you to Michael for his help and to Jack Jansen as well
for useful advice with debugging multi-threaded applications.

-- Russell