Correct use of SMTP "Sender" header?

Question

Our web application sends email messages to people when someone posts new content. Both sender and recipient have opted into receiving email messages from our application. When preparing such a message, we set the following SMTP headers:

FROM: author@example.com
TO: recipient@example.com
SENDER: webapp@mycompany.com

We chose to use the author's email address in the FROM header in an attempt to provide the best experience for the recipient; when they see the message in their mail client, the author is clear. To avoid the appearance of spoofing, we added the SENDER header (with our own company email address) to make it clear that we sent the message on the author's behalf. After reading RFCs 822 and 2822, this seems to be an intended use of the sender header.

Most receiving mail servers seem to handle this well; the email message is delivered normally (assuming the recipient mailbox exists, is not over quota, etc). However, when sending a message FROM an address in a domain TO an address in the same domain, some receiving domains reject the messages with a response like:

571 incorrect IP - psmtp (in reply to RCPT TO command)

I think this means the receiving server only saw that the FROM header address was in its own domain, and that the message originated from a server it didn't consider authorized to send messages for that domain. In other words, the receiving server ignored the SENDER header.

We have a workaround in place: the webapp keeps a list of such domains that seem to ignore the SENDER header, and when the FROM and TO headers are both in such a domain, it sets the FROM header to our own email address instead. But this list requires maintenance.

Is there a better way to achieve the desired experience? We'd like to be a "good citizen" of the net, and all parties involved -- senders and recipients -- want to participate and receive these messages. One alternative is to always use our company email address in the FROM header, and prepend the author's name/address to the subject, but this seems a little clumsy.

Answer 1

You're looking at the wrong things. Those are the message headers. You should be looking at the SMTP envelope. (How the envelope is specified depends from how, exactly, your application is submitting mail to the mail system. On many systems the envelope is specified by command-line arguments to the mail submission utility program.) Depending from exactly when in the protocol transaction it decides to issue that 571 response, the SMTP Relay server may not have even seen the message headers at all.

The response text is saying that the administrator of that particular SMTP Relay server you are talking to has restricted what you can put in the SMTP envelope. It appears to be complaining about the recipient part of the envelope. But it may be deferring validation of the envelope sender until specification of the first recipient, so it may be complaining about the sender.

Note that the envelope sender is where delivery status messages are sent, and you'll not want to have those directed to random people around the world. (Aside from the fact that many people don't like this, it makes no sense for delivery status messages for your mail to be returned to anyone but you.) Specify yourself as the envelope sender.

It is wrong to require MX resource records, by the way. An SMTP Relay server can be located by A and AAAA resource records in the absence of any MX resource records. See RFC 5321 § 5.1.

Answer 2

I might be wrong, but the most likely cause of the above error, especially in the case of Postini, is that the domains where you are getting rejected have a strict SPF policy. Most mail servers with SPF checking will be checking just the From: header, they won't care about the Sender header.

To check if this is the case run "dig +short TXT domain.com" where domain.com is what's giving you the error message. You should get back something like:

"v=spf1 mx -all"

The important part is the -all. This means that the domain owner has stated that they'll only ever send email from the servers that act as their mail servers, all other mail will be rejected.

Fortunately, if this is the case, you can actively check before sending out the email! Get the WebApp to do a SPF check when the user puts in their email address. If there is a strict policy in place, add the domain to your list. There's no shortage of libraries for all languages that can do SPF checks.