Towards thread groups?

Discussion in 'Python' started by John Goerzen, Jul 24, 2003.

  1. John Goerzen

    John Goerzen Guest


    I am the author of OfflineIMAP[1], a program that currently is a heavy
    user of threads. For various reasons (see below), thread groups are
    something that is highly desirable for me in Python. Is anyone aware
    of an implementation of thread groups for Python? Is anyone else
    working on this? Is anyone else experiencing this sort of pain?

    Here's why I need this:

    OfflimeIMAP is a bi-directional IMAP/Maildir synchronization system.
    It can synchronize multiple accounts at once. Within a given account,
    it can synchronize multiple folders at once. Within a given folder,
    it can synchronize multiple messages at once. All these simultaneous
    actions are handled by threads. (In case you're wondering, there are
    bounds in the system to prevent this all from running away.)

    Right now, error handling is not very good. Let's say that one of the
    5 connections to an IMAP server for Account A dies on us and we get a
    network exception. Ideally, with this condition, I would shut down
    all the other threads processing Account A, terminate its open
    connections, and restart it. (An unexpected error can leave
    OfflineIMAP with an incorrect idea of what is on the server, so the
    safest thing to do is reset the algorithm and thus get updates from
    the server on what exactly it knows about.)

    However, I can't do that with the current system. The best thing I
    can do is catch the exception and transmit it to an exception handling
    thread. (By overriding methods of the Thread object.) That's close,

    The exception handling thread can do things like print out the error.
    But the recourses available to it are: 1) terminate the entire
    application, and 2) ignore the error. Option #2 is not satisfactory,
    to the only remaining option is #1.

    Here are some other things that won't work:

    * Putting all network code within a try block.

    Several problems with that: the exceptions thrown are different for
    different types of connections (plain socket, ssl, etc), and may
    not be known in advance.

    Moreover, actions such as "download message" are done in individual
    threads, and the parent thread -- which really needs to know about
    this -- has no way to get the message until it does a join, which
    it may or may not ever have cause to do. Even then, it would
    usually be better for it to get the message immediately.

    * Save some state on thread exit and retrieve it on join().

    Possible, yes, and might be a usable workaround. However, there
    are a number of places in the code that don't do a join(), and this
    seems an error-prone way of doing things.

    It basically involves manually passing data back up the "thread
    chain", like passing error codes back up through the call chain in
    C. Essentially, many of the benefits of exceptions in the first
    place disappear.

    Thoughts anyone?

    [1] OfflineIMAP is at

    -- John Goerzen
    John Goerzen, Jul 24, 2003
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.