There are several problems with the transporting email a secure way and this questions might be better asked at security.stackexchange.com. The mail can be intercepted at various stages:
- Explicit TLS (STARTTLS) and implicit TLS (SMTPS) provide hop-by-hop and not end-to-end security, that is each mail server in between has access to the unencrypted mail.
- The next hop for a mail is determined by looking up the MX record via DNS. This can be spoofed unless DNSSec is used, which is mostly not the case. If it is spoofed even proper checking of the certificates within the TLS handshake will not help, because it will check that the connection to the wrong MTA is the correct one.
- MX is only defined to use the server at port 25, that is with explicit TLS (STARTTLS). Since there is no way for the sending MTA to know if the receiving MTA even supports STARTTLS the connection can be reduced to plain by a man-in-the-middle attacker by modifying the list of supported extension in the reply to the EHLO command or by denying the STARTTLS command. And even hacker with no way to intercept the connection directly might be able to inject TCP RST packets at the beginning of the TLS handshake so that the client thinks that there are problems using TLS and will fall back to plain text. Since there are actually enough real-world problems with the TLS handshake the client might not get suspicious.
- And within the TLS handshake often no proper validation is done. Sometimes no certificates gets checked at all, sometimes only the chain but not if the hostname matches etc. But again, if MX is already spoofed validation might be done against the wrong server anyway :(
Which means to make at least hop-by-hop encryption safe enough you have to add non-trivial fixes to multiple places and most of them are not in control of a sending MTA.
To get in detail to questions from the comments:
1.) Is STARTTLS normally implemented between servers or does it start from client-server and proceed from hop to hop?
STARTTLS must be implemented on each hop (MTA). It's can be used only if the receiving MTA does support STARTTLS and the sending MTA can do it. It does not depend on how the mail was delivered to the initial hop (MTA) from the client. Generally TLS with SMTP only encrypts the connection between hops, so it can be read in clear on each hop if it is not encrypted end-to-end too (with PGP or S/MIME).
2.) Can mail encrypted by PGP or S/ MIME still be redirected by MX spoofing?
Yes, it can still be redirected. But the mail itself can still only be decrypted by the real recipient, i.e. the owner of the target key.
3.) Does DNSSEC have to be adopted on a wide scale to be useful or is it something an individual can benefit from by configuring on his own email server?
Every bit helps, but only a bit. And note that it not only needs the DNSSec records but the forwarding MTA must actually use DNSSec. Most MTA probably do only DNS and don't realize if they get a spoofed MX record.
4.) Do you have to worry about TCP RST injection for implicit tls or just explicit?
Implicit TLS is only used from client to MTA, i.e. to the initial hop. This is the most secure step because it is actually in control of the client and the client uses usually a fixed SMTP server with fixed TLS settings, so no problem with MX spoofing and connection downgrades. If the client is instead configured to use TLS only if available the attack might work against implicit TLS too.
5.) Can't you configure the server/MTA to drop connection (bounce email) if other MTA isn't TLS compatible i.e. switch from opportunistic to required, or is that something you can only do on the client (like Thunderbird) for client-server connection?
Most servers can be configured this way but most servers have to talk to a variety of servers and they don't know up front if the server will support TLS. But sendmail for example can be configured to talk only TLS to specific servers or not to talk TLS to specific other servers. So this configuration can be hardened to known good servers, but this must be done manually.
6.) If #5 is possible and email bounces, will sender receive a 'failed to send' notice?
Yes, usually this is the case, like with all delivery errors. But like most of the other delivery errors only technically fit people might understand these error messages.
7.) As far as certificates not getting checked, is that something that can be fixed with proper configuration/implementation. I read that one of the issues was that a lot of certificates are self signed and most MTA's accept them by default so they can stay compatible with more MTA's. Is that part of what you were referring to?
It depends on the server. Lat time I looked postfix was fine but sendmail could not properly check the hostname against the certificate. And yes, self-signed is one problem and the other main problem are missing intermediate certificates. But since mail delivery works anyway (since certificate errors are ignored by the sender) most admins don't realize the bad configuration or don't care.
8.) Lastly, say all non-trivial fixes were put in place, and i understand the other MTA/client is out of my control. Can these MITM and injection attacks target specific connections going to and from my email address? In this situation they haven't broken into my server and they have no previous experience with my contacts.
Again, this is only hop-by-hop encryption and the mail is available in clear for reading and modifying on each of the hops. Thus an attacker on any of the hops involved in the transfer (usually at least two, one at the sender and one at the recipients site) can intercept and also modify the mails and can of course restrict itself to deal only with selected mails. The only protection is end-to-end encryption using PGP or S/MIME.
