Simple Python script as SMTP server for outgoing e-mails?

[Chris Angelico]

Yes setting SPF records will help your mail be accepted by other
servers, but I disagree with your appeal to make mail server SPF
handling as strict as your server does. SPF has problems in a number of
situations which could cause legitimate mail to be rejected. In my last
job I could only use SPF as one spam factor, not as a basis for rejection.

If legit mail is rejected for failing an SPF check, it's the sending
admin's problem, not yours. You should never have problems with it if
it's set up correctly. And since rejected mail gets reported to the
transmitting MTA, you don't need to drop it in a spambox or anything.
It's not spam, it's simply invalid mail (equivalent to something sent
to a dud address).

ChrisA

 

[Michael Torrie]

Incidently, how do ISP MTAs find whether the remote MTA is legit or
running on some regular user's computer?

1. Query Reverse DNS for IP
2. Find domain
3. Query DNS for MX
4. ?

My mail server did a number of things:
1. ensure IP address of sending server has a reverse name (domain didn't
particularly matter)
2. ensure the HELO address in SMTP matches IP address of sending server
3. check sender IP address against spam blacklists, which includes
netblocks of home ISPs, some entire countries, flagged subnets
4. greylist sender IP if the recipient requested it. First connection
always fails with a nonfatal server error, next connection must wait at
least 5 minutes. If a reconnection happened too quickly, the IP was
temporarily black listed. After success, IP address is whitelisted for
a time. A commandline MTA will not be able to get through greylisting;
only a mail server with queuing could. Spambots tend to give up on the
first error, even now. Cheaper targets I guess.
5. spamassassin checked SPF (DNS) and domainkeys (message itself) and
weighted the spam factor accordingly

I think there were other basic rules that sendmail applied to the
sender, but I can't remember all of what they are. This is well and
truly off topic now for the python list, though.

 

[Chris Angelico]

My mail server did a number of things:
1. ensure IP address of sending server has a reverse name (domain didn't
particularly matter)
2. ensure the HELO address in SMTP matches IP address of sending server
3. check sender IP address against spam blacklists, which includes
netblocks of home ISPs, some entire countries, flagged subnets
4. greylist sender IP if the recipient requested it. First connection
always fails with a nonfatal server error, next connection must wait at
least 5 minutes. If a reconnection happened too quickly, the IP was
temporarily black listed. After success, IP address is whitelisted for
a time. A commandline MTA will not be able to get through greylisting;
only a mail server with queuing could. Spambots tend to give up on the
first error, even now. Cheaper targets I guess.
5. spamassassin checked SPF (DNS) and domainkeys (message itself) and
weighted the spam factor accordingly

I think there were other basic rules that sendmail applied to the
sender, but I can't remember all of what they are. This is well and
truly off topic now for the python list, though.

And yet off-topic does happen... For what it's worth, here's how my
server is set up:
1. A variety of protocol-level checks. If you don't say HELO, for
instance, you get rejected. Surprisingly, these simple checks actually
keep out a lot of spam - but I've yet to see any legiit mail blocked
by them. (Not that I keep logs of these any more. I stopped watching
after it looked clean for a while.) And if legit mail is rejected,
it'll be resent or bounced by the sending MTA anyway.
2. SPF checks on the MAIL FROM:<> address. Again, if legit mail gets
rejected (which would be the fault of the sending domain owner), the
server at the previous hop will deal with it. Only hard failures get
thrown out; anything else just gets marked (which we usually ignore)
and delivered as normal, not even spam-scored.
3. Bayesian spam filter, set very conservatively so we get false
negatives but (almost) no false positives.

Any spam that gets through these three checks gets delivered, and then
the users will drop it in their junk folder. Every week I do a
train-and-wipe run across all junk folders, which logs spam counts
from our primary mailboxes. Last week's run was 228 spam across the
six logged accounts (some of those accounts collect from many
addresses), or an average of five false negatives per account per day,
and false positives are almost completely unheard-of. Considering how
much spam assaults the outside of my fortress's walls, that's a fairly
good ratio, I think. SPF for the win.

ChrisA

 

[Michael Torrie]

If legit mail is rejected for failing an SPF check, it's the sending
admin's problem, not yours. You should never have problems with it if
it's set up correctly. And since rejected mail gets reported to the
transmitting MTA, you don't need to drop it in a spambox or anything.
It's not spam, it's simply invalid mail (equivalent to something sent
to a dud address).

Sure. Tell that to the people you work for who depend on e-mail. When I
was a sysadmin (quite recently), I'd have gotten fired for enforcing
such an arbitrary policy. Indeed when mail wasn't coming through that
someone in the organization was expecting and wanting, regardless of
SPF, it was indeed *my* problem and my job was on the line. BOFH
attitudes simply aren't going to change that reality.

SPF is just one more of the many things that are contributing overall to
absolutely breaking and demise of SMTP. I'm afraid when it does finally
cease to work, it's going to be replaced with less open,
centrally-controlled messaging systems like facebook. Which is unfortunate.

 

[Chris Angelico]

Sure. Tell that to the people you work for who depend on e-mail. When I
was a sysadmin (quite recently), I'd have gotten fired for enforcing
such an arbitrary policy. Indeed when mail wasn't coming through that
someone in the organization was expecting and wanting, regardless of
SPF, it was indeed *my* problem and my job was on the line. BOFH
attitudes simply aren't going to change that reality.

Is your job on the line if the sender of that email got the
recipient's address right? Is your job on the line if the sender
mucked up his SMTP settings and the message didn't even get to your
server? Is your job on the line if the email never even got sent? Then
why should your job be on the line if the sender violates his own
declared protocol? Remember, if you don't publish an SPF record, your
emails will be accepted regardless. It's only if you explicitly create
that DNS record that ends with "-all" that any of this will happen -
which means you *asked* for that mail to be rejected. If you do that
and then send mail from a different IP, then I *will* reject it.
Accepting mail and just giving it a spam score is *worse*, because the
sender won't even know why it didn't get through (what if most of his
mail gets accepted, but that one email when he sent a blank body,
subject "RE: your invoice", and a zip file attachment, managed to trip
the spam cutoff and get dumped?), whereas rejecting will result in a
quick and easy bounce, probably within seconds (minutes maybe).

I stand by SPF checking. It has never been a problem. If you don't
stand by protocols, you weaken those protocols.

And speaking of protocols, I'm now going to have to follow the "I'm on
an airliner and mobile phones have to be turned off" protocol, as the
flight's due to depart shortly. Ah, protocols... some you love, some
not so much.

ChrisA

 

[Nobody]

Incidently, how do ISP MTAs find whether the remote MTA is legit or
running on some regular user's computer?

Look up the IP address in a database. If they don't have a database,
perform a reverse DNS lookup and reject anything which looks like a
typical auto-generated name for a consumer DSL/cable connection.

FWIW, I've been running sendmail on my home system (ADSL with static IP)
for years, and have had very few problems with mail being rejected.

 

[Chris Angelico]

If you want your emails to have the best chance of arriving your SPF should
list servers you use but not deny that there might be others.

That usually makes the SPF record completely useless. The whole point
is to say that random addresses on the internet _will not_ send mail
from you.

I have a very common situation where an overly strict SPF may cause
problems:

Like many people I have multiple email addresses which all end up in the
same inbox. The one I most commonly give out to businesses bounces the
email unchanged to the gmail inbox that I use. That means all emails I
receive through that email address appear to Google to have originated from
the forwarding servers. An SPF record from the original sender that claims
to have a complete list of originating servers will therefore fail
validation.

Ah, there's a solution to this one. You simply use your own
envelope-from address; SPF shouldn't be being checked for the From:
header. Forwarding and using the original sender's address in the SMTP
'MAIL FROM' command is forging mail from them, so it is correct for
that to be thrown out. The mail is coming from your own account, so
you put your address in it, and you might even be able to put an
uber-strict SPF record like "v=spf1 ip4:1.2.3.4 -all" which is quick
to process and guarantees that nobody can pretend to forward mail on
your behalf. The checks are for the *current connection*, not anything
earlier.

ChrisA

 

[Chris Angelico]

Ah, there's a solution to this one. You simply use your own
envelope-from address; SPF shouldn't be being checked for the From:
header.

There's an example, by the way, of this exact technique right here -
(e-mail address removed) sends mail to me with an envelope-from of
"[email protected]" - which passes SPF,
since python.org has a TXT record designating the sending IP as one of
theirs. It doesn't matter that invalid.invalid (your supposed domain)
doesn't have an SPF record, nor would it be a problem if it had one
that said "v=spf1 -all", because that domain wasn't checked. Mailing
lists are doing the same sort of forwarding that you're doing.

(Apologies to those who read this as a newsgroup, for whom this won't
be as parallel an example. But it's still the case, just not for the
posts you receive.)

ChrisA

 

[Vincent Vande Vyvre]

On Windows a script where de endline are the system line sep, the files
are open with a double line in Eric4, Notepad++ or Gedit but they are
correctly displayed in the MS Bloc-Notes.

Example with this code:
----------------------------------------------
# -*- coding: utf-8 -*-

import os
L_SEP = os.linesep

def write():
strings = ['# -*- coding: utf-8 -*-\n',
'import os\n',
'import sys\n']
with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s.replace('\n', L_SEP))

write()
----------------------------------------------

The syntax `s.replace('\n', L_SEP)`is required for portability.

Regards
-
Vincent V.V

 

[Jason Swails]

On Tue, Jul 23, 2013 at 7:42 AM, Vincent Vande Vyvre <

On Windows a script where de endline are the system line sep, the files
are open with a double line in Eric4, Notepad++ or Gedit but they are
correctly displayed in the MS Bloc-Notes.

Example with this code:
------------------------------**----------------
# -*- coding: utf-8 -*-

import os
L_SEP = os.linesep

def write():
strings = ['# -*- coding: utf-8 -*-\n',
'import os\n',
'import sys\n']
with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s.replace('\n', L_SEP))

I must ask why you are setting strings with a newline line ending only to
replace them later with os.linesep. This seems convoluted compared to
doing something like

def write():
strings = ['#-*- coding: utf-8 -*-', 'import os', 'import sys']
with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s)
outf.write(L_SEP)

Or something equivalent.

If, however, the source strings come from a file you've created somewhere
(and are loaded by reading in that file line by line), then I can see a
problem. DOS line endings are carriage returns ('\r\n'), whereas standard
UNIX files use just newlines ('\n'). Therefore, if you are using the code:

s.replace('\n', L_SEP)

in Windows, using a Windows-generated file, then what you are likely doing
is converting the string sequence '\r\n' into '\r\r\n', which is not what
you want to do. I can imagine some text editors interpreting that as two
endlines (since there are 2 \r's). Indeed, when I execute the code:

on UNIX and open the resulting file in gedit, it is double-spaced, but if I
just dump it to the screen using 'cat', it is single-spaced.

If you want to make your code a bit more cross-platform, you should strip
out all types of end line characters from the strings before you write
them. So something like this:

with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s.rstrip('\r\n'))
outf.write(L_SEP)

Hope this helps,
Jason

 

[Vincent Vande Vyvre]

Le 23/07/2013 14:39, Jason Swails a écrit :

On Tue, Jul 23, 2013 at 7:42 AM, Vincent Vande Vyvre

On Windows a script where de endline are the system line sep, the
files are open with a double line in Eric4, Notepad++ or Gedit but
they are correctly displayed in the MS Bloc-Notes.

Example with this code:
----------------------------------------------
# -*- coding: utf-8 -*-

import os
L_SEP = os.linesep

def write():
strings = ['# -*- coding: utf-8 -*-\n',
'import os\n',
'import sys\n']
with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s.replace('\n', L_SEP))


I must ask why you are setting strings with a newline line ending only
to replace them later with os.linesep. This seems convoluted compared
to doing something like

def write():
strings = ['#-*- coding: utf-8 -*-', 'import os', 'import sys']
with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s)
outf.write(L_SEP)

Or something equivalent.

If, however, the source strings come from a file you've created
somewhere (and are loaded by reading in that file line by line), then
I can see a problem. DOS line endings are carriage returns ('\r\n'),
whereas standard UNIX files use just newlines ('\n'). Therefore, if
you are using the code:

s.replace('\n', L_SEP)

in Windows, using a Windows-generated file, then what you are likely
doing is converting the string sequence '\r\n' into '\r\r\n', which is
not what you want to do. I can imagine some text editors interpreting
that as two endlines (since there are 2 \r's). Indeed, when I execute
the code:

on UNIX and open the resulting file in gedit, it is double-spaced, but
if I just dump it to the screen using 'cat', it is single-spaced.

If you want to make your code a bit more cross-platform, you should
strip out all types of end line characters from the strings before you
write them. So something like this:

with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s.rstrip('\r\n'))
outf.write(L_SEP)

Hope this helps,
Jason

The '\n' are in the original file.

I've tested these other versions:

-------------------------------
def write():
strings = ['# -*- coding: utf-8 -*-\n',
'import os\n',
'import sys\n']
with open('writetest.py', 'w') as outf:
txt = L_SEP.join([s.rstip() for s in strings]):
outf.write(txt)
------------------------------

-------------------------------
def write():
strings = ['# -*- coding: utf-8 -*-',
'import os',
'import sys']
with open('writetest.py', 'w') as outf:
txt = L_SEP.join( strings):
outf.write(txt)

 

[Vincent Vande Vyvre]

Le 23/07/2013 15:10, Vincent Vande Vyvre a écrit :

The '\n' are in the original file.

I've tested these other versions:

-------------------------------
def write():
strings = ['# -*- coding: utf-8 -*-\n',
'import os\n',
'import sys\n']
with open('writetest.py', 'w') as outf:
txt = L_SEP.join([s.rstip() for s in strings]):
outf.write(txt)
------------------------------

-------------------------------
def write():
strings = ['# -*- coding: utf-8 -*-',
'import os',
'import sys']
with open('writetest.py', 'w') as outf:
txt = L_SEP.join( strings):
outf.write(txt)

Also with:

----------------------------------------
def count():
with open('c:\\Users\\Vincent\\writetest.py', 'r') as inf:
lines = inf.readlines()
for l in lines:
print(l, len(l))

count()

 

[Jason Swails]

On Tue, Jul 23, 2013 at 9:26 AM, Vincent Vande Vyvre <

Le 23/07/2013 15:10, Vincent Vande Vyvre a écrit :

The '\n' are in the original file.

I've tested these other versions:

------------------------------**-
def write():
strings = ['# -*- coding: utf-8 -*-\n',
'import os\n',
'import sys\n']
with open('writetest.py', 'w') as outf:
txt = L_SEP.join([s.rstip() for s in strings]):
outf.write(txt)
------------------------------

------------------------------**-
def write():
strings = ['# -*- coding: utf-8 -*-',
'import os',
'import sys']
with open('writetest.py', 'w') as outf:
txt = L_SEP.join( strings):
outf.write(txt)

Also with:

------------------------------**----------
def count():
with open('c:\\Users\\Vincent\\**writetest.py', 'r') as inf:
lines = inf.readlines()
for l in lines:
print(l, len(l))

Unrelated comment, but in general it's (much) more efficient to iterate
through a file rather than iterate through a list of strings generated by
readlines():

def count():
with open('c:\\Users\\Vincent\\writetest.py', 'r') as inf:
for l in lines:
print(l, len(l))

It's also fewer lines of code.

('# -*- coding: utf-8 -*-\r\n', 25)

('import os\r\n', 11)
('import sys', 10)

Then it seems like there is an issue with your text editors that do not
play nicely with DOS-style line endings. Gedit on my linux machine
displays the line endings correctly (that is, '\r\n' is rendered as a
single line).

Good luck,
Jason

 

[Michael Torrie]

There's an example, by the way, of this exact technique right here -
(e-mail address removed) sends mail to me with an envelope-from of
"[email protected]" - which passes SPF,
since python.org has a TXT record designating the sending IP as one of
theirs. It doesn't matter that invalid.invalid (your supposed domain)
doesn't have an SPF record, nor would it be a problem if it had one
that said "v=spf1 -all", because that domain wasn't checked. Mailing
lists are doing the same sort of forwarding that you're doing.

This is good and all, and I think I will modify my local postfix mail
server I use for personal stuff, just for correctness' sake.

I hadn't spent much time studying SPF in depth before, but after reading
your comments (which were insightful) I'm now more convinced that SPF is
worthless than ever, at least as a spam prevention mechanism. Spammers
can use throwaway domains that publish very non-strict SPF records, and
spam to their hearts content with random forged from addresses and SPF
checks pass. The only way around that is to enforce SPF on the From:
header in the e-mail itself, which we all agree is broken. I've been
reading this:

http://www.openspf.org/FAQ/SPF_is_not_about_spam

Not very encouraging. When the other expensive options for going after
spammers who have valid SPF records, they propose domain blacklists as a
solution.

 

[Steven D'Aprano]

On Windows a script where de endline are the system line sep, the files
are open with a double line in Eric4, Notepad++ or Gedit but they are
correctly displayed in the MS Bloc-Notes.

I suspect the problem lies with Eric4, Notepad++ and Gedit. Do you
perhaps have to manually tell them that the file uses Windows line
separators?

I recommend opening the file in a hex editor and seeing for yourself what
line separators are used.

Example with this code:
----------------------------------------------
# -*- coding: utf-8 -*-

import os
L_SEP = os.linesep

def write():
strings = ['# -*- coding: utf-8 -*-\n',
'import os\n',
'import sys\n']
with open('writetest.py', 'w') as outf:
for s in strings:
outf.write(s.replace('\n', L_SEP))

write()

I don't think it is. Behaviour is a little different between Python 2 and
3, but by default, Python uses "Universal Newlines". When you open a file
in text mode, arbitrary line separators should be automatically
translated to \n when reading, and \n will be automatically translated to
os.line_sep when writing.


http://docs.python.org/3/library/functions.html#open
http://docs.python.org/2/library/functions.html#open

Some further discussion here:

http://stackoverflow.com/questions/12193047/is-universal-newlines-mode-
supposed-to-be-default-behaviour-for-open-in-python

 

[Chris Angelico]

Excellent idea, I'll tell the email forwarding service to rewrite their
system immediately.

Yes. If they are using your domain in the MAIL FROM command and not
using your mail servers, then yes, you should tell them, and use a
different service until they fix that. It is not SPF's fault. It is a
fundamental error of protocol, which SPF checks are highlighting.

ChrisA

 

[Gilles]

try http://emailrelay.sourceforge.net/

Thanks. I did find it, but it says it's not a full MTA:

"E-MailRelay is not a routing MTA. It forwards e-mail to a
pre-configured SMTP server, regardless of any message addressing or
DNS redirects."
http://emailrelay.sourceforge.net/userguide.html#SH_1_2

IOW, it'll just send outbound e-mails to my ISP's MTA, so I'm back at
square one.

 

[Gilles]

Where did you look? Here's one I found. It's not the real sendmail
program, but it implements the interface which is all you need:

http://glob.com.au/sendmail/

I just googled for sendmail win32

Thanks, but I need an MTA, not just a command-line app, so I can send
e-mails from my e-mail client and just change the SMTP line in the
configuration.

 

[Gilles]

http://www.hmailserver.com

Thanks. hMailServer was one of the apps I checked, and I was just
making sure there weren't something simpler, considering my needs,
ideally something like Mongoose MTA.

Regardless, because of the SPAM anti-measures mentioned above, it
seems like I was over-optimistic about running an MTA and sending
e-mails from my home computer :-/

 

[Dennis Lee Bieber]

I suspect the problem lies with Eric4, Notepad++ and Gedit. Do you
perhaps have to manually tell them that the file uses Windows line
separators?

Don't know about those, but SciTE I know has both an menu option for
line ending (<cr><lf>, <lf>, <cr>), and one for "convert line endings"

I also know Windows Notepad (at least on WinXP) would have the opposite
problem: <lf> termination files were displayed as single lines (Wordpad,
OTOH, correctly parsed those).