urllib hanging in thread

Discussion in 'Python' started by Russell E. Owen, May 4, 2004.

  1. I am having trouble with some ftp code. On some platforms it works fine
    and on others it reliably hangs.

    The code uses urllib to retrieve a file in a background thread, which
    runs the following code. The diagnostic message before urllib.urlopen is
    printed, but the one after urllib.urlopen is not:

    def _getTask(self):
    try:
    self._toFile = file(self.toPath, 'wb')
    print "opening URL"
    self._fromFile = urllib.urlopen(self.fromURL)
    print "URL opened" # when hung this does not print
    self._totBytes =
    self._fromFile.info().getheader("Content-Length")
    if self._totBytes:
    self._totBytes = int(self._totBytes)

    self._state = Running

    while True:
    nextData = self._fromFile.read(8192)
    if not nextData:
    break
    elif self._state == Aborting:
    self._cleanup(Aborted)
    return
    self._readBytes += len(nextData)
    self._toFile.write(nextData)
    self._cleanup(Done)
    except Exception, e:
    self._cleanup(Failed, exception = e)

    However, I can only make it hang when run as part of a fairly large
    Tkinter application. The same code runs fine standalone or as part of a
    minimal Tkinter test application. The large application has a few other
    threads for communicating via TCP. In all other respects the large app
    runs fine; the ftp hangs while everything else keeps cooking along.
    Closing the TCP connection (which should stop the other threads) has no
    effect on the stalled ftp transfer.

    It works fine on my MacOS X 10.3.3 box, including both the built in
    python (a variant of 2.3 with aqua Tk) and a python 2.3.3 that I built
    as a plain unix python with X11 Tk.

    The code hangs on our RedHat linux boxes (python 2.3.3, dunno the linux
    version). It also hangs as a standalone MacOS X executable, which is
    especially weird since the code runs fine on the same computer before it
    gets packaged up as an application.

    Any suggestions would be most appreciated.

    -- Russell
    Russell E. Owen, May 4, 2004
    #1
    1. Advertising

  2. In article <>,
    "Russell E. Owen" <> wrote:

    >I am having trouble with some ftp code. On some platforms it works fine
    >and on others it reliably hangs....


    After some experimentation, I found the hanging was occurring in ftplib.
    It delays importing re and compiling two regular expressions until they
    are needed, and the thread hangs both on the import and (if I modify
    ftplib to import re in advance) on the compilation. Modifying ftplib to
    do both jobs in advance (when first imported) fixes the problem.

    It seems strange to me that a thread could hang indefinitely trying to
    perform these two tasks when the rest of the app seems to be running
    fine. Perhaps the thread is simply being starved of cpu cycles, but then
    it seems odd that the thread has enough cycles to print diagnostics and
    do network communication.

    Anyway, I can work around the problem by using ftplib directly instead
    of using urllib, and importing my own patched ftplib. But I'm wondering
    if anyone has any better suggestions.

    -- Russell
    Russell E. Owen, May 14, 2004
    #2
    1. Advertising

  3. Re: urllib hanging in thread:solved!

    In article <>,
    "Russell E. Owen" <> wrote:

    >>I am having trouble with some ftp code. On some platforms it works fine
    >>and on others it reliably hangs.

    ....and later followed up with:
    >After some experimentation, I found the hanging was occurring in ftplib.
    >It delays importing re and compiling two regular expressions until they
    >are needed, and the thread hangs both on the import and (if I modify
    >ftplib to import re in advance) on the compilation. Modifying ftplib to
    >do both jobs in advance (when first imported) fixes the problem.


    Michael Hudson (via a separate forum) suggested the issue was contention
    for the "import lock" (which I'd never heard of). He was right.

    I originally wrote my app to be run as follows:
    % python <tui_root>/TUI/Main.py
    Later, I added a script to simplify execution (no need to mess with
    PYTHONPATH) and bundling as a double-clickable app:
    <tui_root>runtui.py
    whose sole contents were:
    import TUI.Main

    I knew it was ugly, but didn't realize it was dangerous. Apparently when
    one executes an app by importing it, the main thread holds the "import
    lock", which prevents other threads from importing anything. ftplib
    imports re when first used, and was hanging at that point.

    What made it tricky to debug was I had forgotten to convert my own
    command-line environment to running the app via the new troublesome
    script. So I was seeing the problem on some systems but not others.
    Anyway, fixing <tui_root>/TUI/Main.py to be run via a function call
    fixed the problem, i.e. runtui.py now reads:
    import TUI.Main
    TUI.Main.runTUI()

    So...a big thank you to Michael for his help and to Jack Jansen as well
    for useful advice with debugging multi-threaded applications.

    -- Russell
    Russell E. Owen, May 19, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Brian Hann

    IPC::Run and hanging pipes

    Brian Hann, Dec 8, 2003, in forum: Perl
    Replies:
    1
    Views:
    769
    Brian Hann
    Dec 11, 2003
  2. Kingsley

    urllib (in thread) never returns

    Kingsley, Jul 17, 2006, in forum: Python
    Replies:
    1
    Views:
    304
    Rene Pijlman
    Jul 18, 2006
  3. Replies:
    4
    Views:
    592
    Paul Dubuc
    Apr 12, 2006
  4. Jonathan Gardner

    Asynchronous urllib (urllib+asyncore)?

    Jonathan Gardner, Feb 26, 2008, in forum: Python
    Replies:
    1
    Views:
    462
    Terry Jones
    Feb 27, 2008
  5. Chris McDonald
    Replies:
    0
    Views:
    276
    Chris McDonald
    Nov 1, 2010
Loading...

Share This Page