Add "Received:" header to email msg in correct position?

Discussion in 'Python' started by Grant Edwards, May 5, 2014.

  1. I'm working on a Python app that receives an e-mail message via SMTP,
    does some trivial processing on it, and forwards it to another SMTP
    server.

    I'd like to do the polite thing and add a "Received:" header, but I
    can't figure out how to get Python's email module to add it in the
    correct place. It always ends up at the "bottom" of the headers below
    From: To: etc. It's supposed to go at the above all the Received:
    headers that where there when I received it.
     
    Grant Edwards, May 5, 2014
    #1
    1. Advertisements

  2. Grant Edwards

    Ethan Furman Guest

    I don't know that it matters, but which Python version?
     
    Ethan Furman, May 5, 2014
    #2
    1. Advertisements

  3. Sorry, should have mentioned it: 2.7.5
     
    Grant Edwards, May 5, 2014
    #3
  4. Grant Edwards

    Tim Chase Guest

    Looking at the stdlib source, it doesn't look like there's an easy
    way to specify where it gets inserted. However, the source to
    email.message.Message.add_header() is all of 9 lines of code, so it
    wouldn't be too hard to subclass Message and twiddle self._headers as
    you would any other list (i.e., using .insert() to specify an index).

    It might look something like

    class MyMessage(email.message.Message):
    def insert_header(self, index, _name, _value, **_params):
    parts = []
    for k, v in _params.items():
    if v is None:
    parts.append(k.replace('_', '-'))
    else:
    parts.append(_formatparam(k.replace('_', '-'), v))
    if _value is not None:
    parts.insert(0, _value)
    self._headers.insert(index, (_name, SEMISPACE.join(parts)))

    You might still need to search for *where* you want to insert it, but
    I'll leave that as an exercise to the reader. :)

    -tkc






    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.12 (GNU/Linux)

    iQEcBAEBCgAGBQJTZ/4KAAoJEN385Nh6n9aQmagIAKkeLRANxfbsl4kDkCkuCWn/
    uYothgT82ihtFwmfSdPeMbrjIUAPnPVl3jwujU9nYcFRwT7bXBbiCTVapBSeeCD0
    uInv8lPhEOLQgvATy4oABQnTqw5aQwylTi9RSgwozNu2E89xi0sYuVF+36WniOlk
    ACZ0HPh5lhYWKMB0SSmNo9TmpVWDyhOXTe7W8PJjxjXRatBcgV4WwGUb2QiAXH4f
    Z/eMAXhcABcJDJ6SL12OaAMGGF3ooBDN+Wm7GTzu6ka98hwM7qcxXjilBtXiMxV3
    L2nIZXzaTp9ChtxaZuh0AR2OfEW0l8ditSFJgExss0edbgEaq/JI+4/im2fKHCQ=
    =swTy
    -----END PGP SIGNATURE-----
     
    Tim Chase, May 5, 2014
    #4
  5. Thanks.

    There's is a somewhat messy way to do it by calling msg.items() to
    retrieve all the current headers, removing all of them, and then
    adding them all back (includeing the new one) in the order I want
    them.
    [ insert_header() method example that lets you specify an index ]

    A couple other options I was thinking about:

    * Just override __set_item__ so it treats 'Received' as a special
    case and inserts it before any existing 'Received' headers -- or
    prepends it if there aren't any.

    * Add a prepend_header() method that just sticks it at the top -- in
    most cases, I think that will be correct enough.

    I think I like your suggestion better. Since the keys() method
    preserves the order of header keys, that makes figuring out the
    insertion point trivial.
     
    Grant Edwards, May 6, 2014
    #5
  6. Does msg.items() return the actual message header list, or a copy?
    Because if it returns the list itself, you can just insert directly into
    the list using the insert method.
     
    Steven D'Aprano, May 6, 2014
    #6
  7. Yes, it's required. RFC 2821 [1] section 3.8.2 says "prepend".

    ChrisA

    [1] http://www.ietf.org/rfc/rfc2821.txt
     
    Chris Angelico, May 6, 2014
    #7
  8. Even if it's special-cased as a dedicated "prepend received header"
    method, that would probably do. I can't think of any other headers
    where you need to specifically order them rather than appending.

    ChrisA
     
    Chris Angelico, May 6, 2014
    #8
  9. A shallow copy:

    def items(self):
    """Get all the message's header fields and values.

    These will be sorted in the order they appeared in the original
    message, or were added to the message, and may contain duplicates.
    Any fields deleted and re-inserted are always appended to the header
    list.
    """
    return self._headers[:]
     
    Grant Edwards, May 6, 2014
    #9
  10. I couldn't find it in an RFC. But every reference I could find that
    mentioned Received: headers did say that you read them from the bottom
    up (most recent is on top).
    I'd like to do it the right way whether it's required by the letter of
    the law or not.
     
    Grant Edwards, May 6, 2014
    #10
  11. I was looking in the message format RFCs, and as has already been
    pointed out, it's in the SMTP spec (RFC2821). Since there _is_ an RFC
    that specifically states you MUST prepend a header line to a message,
    this seems to be a hole in the email module...
     
    Grant Edwards, May 6, 2014
    #11
  12. Yeah, I went looking for it in 2822 but didn't find much. After some
    digging, found it in 2821, section 3.8.2. It's not dwelled on, but the
    word "prepend" is used, which to my mind is a clear indication of
    necessary ordering.

    ChrisA
     
    Chris Angelico, May 6, 2014
    #12
  13. Grant Edwards

    Burak Arslan Guest

    The rationale for "prepend" is to make it possible for MTAs to add their
    "Received:" headers to messages without having to parse them.

    So you're supposed to do the same: Just write your Received header,
    followed by '\r\n', followed by the rest of the message to the socket
    and you should be fine.

    Best,
    Burak
     
    Burak Arslan, May 6, 2014
    #13
  14. I need to check and manipulate other headers for other reasons, so I'm
    using the email module for that. In order to keep things consistent
    and easy to understand, I'd like to use the email module to prepend
    the Received header as well. That keeps my application from having to
    have any knowledge about e-mail message formatting.
     
    Grant Edwards, May 6, 2014
    #14
  15. Grant Edwards

    Terry Reedy Guest

    I usually does these days.
    email has been improved with successive 3.x versions.
    https://docs.python.org/3/library/email.html#package-history

    There might be an email5 backport for 2.7 on pypi.

    If the prepend requirement is covered by

    "The email package is a library for managing email messages, including
    MIME and other RFC 2822-based message documents. It is specifically not
    designed to do any sending of email messages to SMTP (RFC 2821), NNTP,
    or other servers; those are functions of modules such as smtplib and
    nntplib. The email package attempts to be as RFC-compliant as possible,
    supporting in addition to RFC 2822, such MIME-related RFCs as RFC 2045,
    RFC 2046, RFC 2047, and RFC 2231."

    and the current 3.4/5 version does not prepend and there is no existing
    tracker issue, then a new issue would seem to be appropriate.
     
    Terry Reedy, May 6, 2014
    #15
  16. That's a bit tricky. RFC 2822 section 3.6.7 says the Received: headers
    are "strictly informational, and any formal interpretation of them is
    outside of the scope of this document", but it does reference RFC
    2821. Should the addition of another Received header be part of
    building an RFC 2822 compliant message, or should you build up a
    message without that, and have one added at transport time?

    I would say that, even if this isn't considered a bug (as in, failure
    to comply with standards it claims to comply with), it would still be
    a viable feature addition.

    ChrisA
     
    Chris Angelico, May 7, 2014
    #16
  17. I thought that was the job of the SMTP servers, not of email
    applications. So I'm not sure
    that what you want to do is the polite thing to do.
     
    Antoon Pardon, May 7, 2014
    #17
  18. This application seems like an SMTP server.
     
    Emre Hasegeli, May 7, 2014
    #18
  19. It doesn't to me. As far as I can see what he wants to do can be done
    by a mail program like procmail in combination with some mail
    filtering/processing.
     
    Antoon Pardon, May 7, 2014
    #19
  20. Exactly. And the SMTP server I'm writing uses the email module to
    manipulate headers as messages are processed.
    Why not?
     
    Grant Edwards, May 7, 2014
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.