Towards thread groups?

Discussion in 'Python' started by John Goerzen, Jul 24, 2003.

  1. John Goerzen

    John Goerzen Guest

    Hello,

    I am the author of OfflineIMAP[1], a program that currently is a heavy
    user of threads. For various reasons (see below), thread groups are
    something that is highly desirable for me in Python. Is anyone aware
    of an implementation of thread groups for Python? Is anyone else
    working on this? Is anyone else experiencing this sort of pain?

    Here's why I need this:

    OfflimeIMAP is a bi-directional IMAP/Maildir synchronization system.
    It can synchronize multiple accounts at once. Within a given account,
    it can synchronize multiple folders at once. Within a given folder,
    it can synchronize multiple messages at once. All these simultaneous
    actions are handled by threads. (In case you're wondering, there are
    bounds in the system to prevent this all from running away.)

    Right now, error handling is not very good. Let's say that one of the
    5 connections to an IMAP server for Account A dies on us and we get a
    network exception. Ideally, with this condition, I would shut down
    all the other threads processing Account A, terminate its open
    connections, and restart it. (An unexpected error can leave
    OfflineIMAP with an incorrect idea of what is on the server, so the
    safest thing to do is reset the algorithm and thus get updates from
    the server on what exactly it knows about.)

    However, I can't do that with the current system. The best thing I
    can do is catch the exception and transmit it to an exception handling
    thread. (By overriding methods of the Thread object.) That's close,
    but...

    The exception handling thread can do things like print out the error.
    But the recourses available to it are: 1) terminate the entire
    application, and 2) ignore the error. Option #2 is not satisfactory,
    to the only remaining option is #1.

    Here are some other things that won't work:

    * Putting all network code within a try block.

    Several problems with that: the exceptions thrown are different for
    different types of connections (plain socket, ssl, etc), and may
    not be known in advance.

    Moreover, actions such as "download message" are done in individual
    threads, and the parent thread -- which really needs to know about
    this -- has no way to get the message until it does a join, which
    it may or may not ever have cause to do. Even then, it would
    usually be better for it to get the message immediately.

    * Save some state on thread exit and retrieve it on join().

    Possible, yes, and might be a usable workaround. However, there
    are a number of places in the code that don't do a join(), and this
    seems an error-prone way of doing things.

    It basically involves manually passing data back up the "thread
    chain", like passing error codes back up through the call chain in
    C. Essentially, many of the benefits of exceptions in the first
    place disappear.

    Thoughts anyone?

    [1] OfflineIMAP is at http://quux.org/devel/offlineimap

    -- John Goerzen
    John Goerzen, Jul 24, 2003
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Petra Hübner
    Replies:
    0
    Views:
    425
    Petra Hübner
    Feb 16, 2004
  2. anonymous
    Replies:
    1
    Views:
    4,512
    Francisco Padron
    May 8, 2005
  3. Jeff
    Replies:
    1
    Views:
    351
    Robbe Morris [C# MVP]
    Dec 20, 2005
  4. Ken Loh
    Replies:
    0
    Views:
    363
    Ken Loh
    Feb 16, 2005
  5. Replies:
    3
    Views:
    344
    Peter Hansen
    Jun 10, 2005
Loading...

Share This Page