thread safe SMTP module

[Gordon Messmer]

I believe that I've seen this discussed previously, so maybe there's
some interest in it. I wrote a threaded mail filtering framework a
while ago, and one of the modules does address verification via SMTP.
Since smtplib.SMTP uses blocking IO, it can block the whole
interpreter. Sometimes the whole thing would stop working indefinitely.

I'm now aware that Twisted offers a non-blocking SMTP class, but I
didn't really want to make that a dependency of the address
verification. I ended up subclassing the smtplib.SMTP class and
rewriting the functions that do I/O. Perhaps someone who doesn't want
to install Twisted will find this class useful, someday. It doesn't
support TLS, but it is otherwise a thread-safe SMTP class.

 

[Aahz]

I believe that I've seen this discussed previously, so maybe there's
some interest in it. I wrote a threaded mail filtering framework
a while ago, and one of the modules does address verification
via SMTP. Since smtplib.SMTP uses blocking IO, it can block the
whole interpreter. Sometimes the whole thing would stop working
indefinitely.

That doesn't make any sense. Blocking I/O generally releases the GIL,
which is the whole reason Python doesn't totally suck for threading.
There may be other thread problems, but I doubt that you have correctly
analyzed their source. You can prove this for yourself by trying out my
threaded spider at
http://www.pythoncraft.com/OSCON2001/ThreadPoolSpider.py

 

[Gordon Messmer]

That doesn't make any sense. Blocking I/O generally releases the GIL,
which is the whole reason Python doesn't totally suck for threading.

Nevertheless, among the caveats listed at
http://docs.python.org/lib/module-thread.html is:

"Not all built-in functions that may block waiting for I/O allow other
threads to run. (The most popular ones (time.sleep(), file.read(),
select.select()) work as expected.)"

There may be other thread problems, but I doubt that you have correctly
analyzed their source.

I subclassed smtplib.SMTP and replaced only the lines of code that had
to do with blocking IO (connect, send and receive operations).
Beforehand, python would occasionally lock up. Having made those
changes, python stopped locking up. I think the problem was pretty well
apparent. I can't pin it down to which one of those three operations
was at fault, and it may be that only one was. However, when I use
non-blocking IO, the application works. When I used smtplib.SMTP, it
didn't.

I'm open to other explanations.

 

[Aahz]

Nevertheless, among the caveats listed at
http://docs.python.org/lib/module-thread.html is:

"Not all built-in functions that may block waiting for I/O allow other
threads to run. (The most popular ones (time.sleep(), file.read(),
select.select()) work as expected.)"

That's why I said "generally".

I subclassed smtplib.SMTP and replaced only the lines of code that had
to do with blocking IO (connect, send and receive operations).
Beforehand, python would occasionally lock up. Having made those
changes, python stopped locking up. I think the problem was pretty well
apparent. I can't pin it down to which one of those three operations
was at fault, and it may be that only one was. However, when I use
non-blocking IO, the application works. When I used smtplib.SMTP, it
didn't.

I'm open to other explanations.

Assuming you have correctly tracked down the problem area, I would call
that a thread bug in Python. But my experience is that you simply have
run into a problem with the socket. I would suggest that using
socket.setdefaulttimeout() would work just as well.

 

[Gordon Messmer]

Assuming you have correctly tracked down the problem area, I would call
that a thread bug in Python. But my experience is that you simply have
run into a problem with the socket. I would suggest that using
socket.setdefaulttimeout() would work just as well.

I believe that solution, also would not work. This note is included in
the socket documentation, regarding "timeout mode":

http://docs.python.org/lib/socket-objects.html
"A consequence of this is that file objects returned by the makefile()
method must only be used when the socket is in blocking mode; in timeout
or non-blocking mode file operations that cannot be completed
immediately will fail."

smtplib.SMTP uses file objects when reading SMTP responses. If I used
setdefaulttimeout(), then the socket would be in timeout mode and the
above note would be applicable.

I am not at all above calling python's behavior a bug, except that it
seemed like a known behavior given the note in the thread documentation
regarding built-in functions that block on I/O.

 

[Aahz]

I believe that solution, also would not work. This note is included in
the socket documentation, regarding "timeout mode":

http://docs.python.org/lib/socket-objects.html
"A consequence of this is that file objects returned by the makefile()
method must only be used when the socket is in blocking mode; in timeout
or non-blocking mode file operations that cannot be completed
immediately will fail."

smtplib.SMTP uses file objects when reading SMTP responses. If I used
setdefaulttimeout(), then the socket would be in timeout mode and the
above note would be applicable.

Hrm. At this point, I would suggest taking discussion to python-dev; it
has been too long since I looked closely at thread/socket behavior.

I am not at all above calling python's behavior a bug, except that it
seemed like a known behavior given the note in the thread documentation
regarding built-in functions that block on I/O.

No, at this point I think this is neither bug nor about thread blocking
on I/O. I think it's about sockets dying and the inability of sockets in
blocking mode to recover. I have seen this kind of behavior in
single-threaded systems. But it really needs someone who knows more than
I do, and I think the first step here is to go ahead and file a bug
report for tracking purposes.