thread safe SMTP module

Discussion in 'Python' started by Gordon Messmer, Mar 2, 2007.

  1. I believe that I've seen this discussed previously, so maybe there's
    some interest in it. I wrote a threaded mail filtering framework a
    while ago, and one of the modules does address verification via SMTP.
    Since smtplib.SMTP uses blocking IO, it can block the whole
    interpreter. Sometimes the whole thing would stop working indefinitely.

    I'm now aware that Twisted offers a non-blocking SMTP class, but I
    didn't really want to make that a dependency of the address
    verification. I ended up subclassing the smtplib.SMTP class and
    rewriting the functions that do I/O. Perhaps someone who doesn't want
    to install Twisted will find this class useful, someday. It doesn't
    support TLS, but it is otherwise a thread-safe SMTP class.
    Gordon Messmer, Mar 2, 2007
    #1
    1. Advertising

  2. Gordon Messmer

    Aahz Guest

    In article <>,
    Gordon Messmer <> wrote:
    >
    >I believe that I've seen this discussed previously, so maybe there's
    >some interest in it. I wrote a threaded mail filtering framework
    >a while ago, and one of the modules does address verification
    >via SMTP. Since smtplib.SMTP uses blocking IO, it can block the
    >whole interpreter. Sometimes the whole thing would stop working
    >indefinitely.


    That doesn't make any sense. Blocking I/O generally releases the GIL,
    which is the whole reason Python doesn't totally suck for threading.
    There may be other thread problems, but I doubt that you have correctly
    analyzed their source. You can prove this for yourself by trying out my
    threaded spider at
    http://www.pythoncraft.com/OSCON2001/ThreadPoolSpider.py
    --
    Aahz () <*> http://www.pythoncraft.com/

    "I disrespectfully agree." --SJM
    Aahz, Mar 4, 2007
    #2
    1. Advertising

  3. Aahz wrote:
    >
    > That doesn't make any sense. Blocking I/O generally releases the GIL,
    > which is the whole reason Python doesn't totally suck for threading.


    Nevertheless, among the caveats listed at
    http://docs.python.org/lib/module-thread.html is:

    "Not all built-in functions that may block waiting for I/O allow other
    threads to run. (The most popular ones (time.sleep(), file.read(),
    select.select()) work as expected.)"

    > There may be other thread problems, but I doubt that you have correctly
    > analyzed their source.


    I subclassed smtplib.SMTP and replaced only the lines of code that had
    to do with blocking IO (connect, send and receive operations).
    Beforehand, python would occasionally lock up. Having made those
    changes, python stopped locking up. I think the problem was pretty well
    apparent. I can't pin it down to which one of those three operations
    was at fault, and it may be that only one was. However, when I use
    non-blocking IO, the application works. When I used smtplib.SMTP, it
    didn't.

    I'm open to other explanations.
    Gordon Messmer, Mar 4, 2007
    #3
  4. Gordon Messmer

    Aahz Guest

    In article <>,
    Gordon Messmer <> wrote:
    >Aahz wrote:
    >>
    >> That doesn't make any sense. Blocking I/O generally releases the GIL,
    >> which is the whole reason Python doesn't totally suck for threading.

    >
    >Nevertheless, among the caveats listed at
    >http://docs.python.org/lib/module-thread.html is:
    >
    >"Not all built-in functions that may block waiting for I/O allow other
    >threads to run. (The most popular ones (time.sleep(), file.read(),
    >select.select()) work as expected.)"


    That's why I said "generally".

    >> There may be other thread problems, but I doubt that you have correctly
    >> analyzed their source.

    >
    >I subclassed smtplib.SMTP and replaced only the lines of code that had
    >to do with blocking IO (connect, send and receive operations).
    >Beforehand, python would occasionally lock up. Having made those
    >changes, python stopped locking up. I think the problem was pretty well
    >apparent. I can't pin it down to which one of those three operations
    >was at fault, and it may be that only one was. However, when I use
    >non-blocking IO, the application works. When I used smtplib.SMTP, it
    >didn't.
    >
    >I'm open to other explanations.


    Assuming you have correctly tracked down the problem area, I would call
    that a thread bug in Python. But my experience is that you simply have
    run into a problem with the socket. I would suggest that using
    socket.setdefaulttimeout() would work just as well.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "I disrespectfully agree." --SJM
    Aahz, Mar 4, 2007
    #4
  5. Aahz wrote:
    >
    > Assuming you have correctly tracked down the problem area, I would call
    > that a thread bug in Python. But my experience is that you simply have
    > run into a problem with the socket. I would suggest that using
    > socket.setdefaulttimeout() would work just as well.


    I believe that solution, also would not work. This note is included in
    the socket documentation, regarding "timeout mode":

    http://docs.python.org/lib/socket-objects.html
    "A consequence of this is that file objects returned by the makefile()
    method must only be used when the socket is in blocking mode; in timeout
    or non-blocking mode file operations that cannot be completed
    immediately will fail."

    smtplib.SMTP uses file objects when reading SMTP responses. If I used
    setdefaulttimeout(), then the socket would be in timeout mode and the
    above note would be applicable.

    I am not at all above calling python's behavior a bug, except that it
    seemed like a known behavior given the note in the thread documentation
    regarding built-in functions that block on I/O.
    Gordon Messmer, Mar 4, 2007
    #5
  6. Gordon Messmer

    Aahz Guest

    In article <>,
    Gordon Messmer <> wrote:
    >Aahz wrote:
    >>
    >> Assuming you have correctly tracked down the problem area, I would call
    >> that a thread bug in Python. But my experience is that you simply have
    >> run into a problem with the socket. I would suggest that using
    >> socket.setdefaulttimeout() would work just as well.

    >
    >I believe that solution, also would not work. This note is included in
    >the socket documentation, regarding "timeout mode":
    >
    >http://docs.python.org/lib/socket-objects.html
    >"A consequence of this is that file objects returned by the makefile()
    >method must only be used when the socket is in blocking mode; in timeout
    >or non-blocking mode file operations that cannot be completed
    >immediately will fail."
    >
    >smtplib.SMTP uses file objects when reading SMTP responses. If I used
    >setdefaulttimeout(), then the socket would be in timeout mode and the
    >above note would be applicable.


    Hrm. At this point, I would suggest taking discussion to python-dev; it
    has been too long since I looked closely at thread/socket behavior.

    >I am not at all above calling python's behavior a bug, except that it
    >seemed like a known behavior given the note in the thread documentation
    >regarding built-in functions that block on I/O.


    No, at this point I think this is neither bug nor about thread blocking
    on I/O. I think it's about sockets dying and the inability of sockets in
    blocking mode to recover. I have seen this kind of behavior in
    single-threaded systems. But it really needs someone who knows more than
    I do, and I think the first step here is to go ahead and file a bug
    report for tracking purposes.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "I disrespectfully agree." --SJM
    Aahz, Mar 8, 2007
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jane Austine

    Is re module thread-safe?

    Jane Austine, Jul 11, 2003, in forum: Python
    Replies:
    1
    Views:
    290
    Skip Montanaro
    Jul 11, 2003
  2. nicolas_riesch

    how to write thread-safe module ? and pytz

    nicolas_riesch, Aug 10, 2005, in forum: Python
    Replies:
    5
    Views:
    408
    nicolas_riesch
    Aug 15, 2005
  3. John Nagle

    Is the subprocess module thread-safe?

    John Nagle, Oct 27, 2007, in forum: Python
    Replies:
    1
    Views:
    1,074
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Oct 28, 2007
  4. Gabriel Rossetti
    Replies:
    0
    Views:
    1,310
    Gabriel Rossetti
    Aug 29, 2008
  5. John Nagle
    Replies:
    5
    Views:
    463
    John Nagle
    Mar 12, 2012
Loading...

Share This Page