You may also want to look at this question, which also touches on the subject.
I understand that an SSL / TLS certificate, verified by a public CA,
will enable transport security on SMTP between servers. However, I
don't see how this could help with transport security between the
server and a client. At best, it seems like the client's outgoing mail
would be encrypted, but incoming mail would be in the clear. Am I
understanding this wrong or is that a separate solution for that?
All of the client mail access methods - basically POP3, IMAP, and webmail - support secure alternatives. POP3 and IMAP can either be run on a dedicated SSL port (just like an HTTPS server, traditionally POP3S is on port 995 and IMAPS is on port 993) or they can support STARTTLS which allows a client to connect to the default plain-text port and then negotiate with the server to switch to encryption first thing in the session.
With mail servers like POP/IMAP/Webmail you can insist upon encryption - if you configure the server correctly, it will refuse to authenticate or share mail before encryption is in place with the client.
I also understand that the SSL certificate would not secure
transmission between the far-end server and any of its clients. For
that S/MIME or PGP is required (I'm not at all interested in PGP.) but
my clients do need to be able to communicate publicly, so I'm resigned
to that part.
You are correct. Because SMTP encryption is opportunistic rather than required, and because mail may pass through multiple servers not controlled by you, the only way to be sure the mail is encrypted through the entire path is client-based encryption like PGP or S/MIME.
That being said, I am attracted to the idea of running S/MIME strictly
between my server and my clients. Is that possible? Can the server
encrypt run S/MIME for all mail to/from internal users, but not run
S/MIME for outbound public mail? Simultaneously can it accept
unencrypted public mail and apply S/MIME encryption for the final
delivery to the internal user? I know that seems convoluted, but I'm
asking as much out of curiosity as for practical reasons.
It is possible to do this but it has little advantage over simply using an encrypted client access protocol, and would be harder to set up (e.g., it's not a config switch on an existing mail server).
You could have your mail server S/MIME-encrypt your mail as it arrives and before it stuffs it in your mailbox. Then the mail would need to be decrypted on the client in order to read it. However, if you did this and still used a plaintext mail access protocol, then your email credentials would be sent in plaintext* over the network, which could allow someone to copy your (encrypted) email, to delete your email, to send mail as if they were you... in short, you want to encrypt that client access protocol, even if you're thinking of enforcing S/MIME on top of your email.
The sole advantage that I can see of encrypting your email with S/MIME as it arrives is that the mail will sit encrypted on the server, and not be vulnerable to being read by an attacker who gains control of your server. (That only protects your back traffic; any new mail arriving could be snooped before your server manages to S/MIME it if the attacker is on the system).
Is it a viable combination to use SSL / TLS certificates and S/MIME,
or does one invalidate the other?
You could layer server-imposed S/MIME on top of encrypted client access protocols; neither invalidates the other. You will want encrypted client access either way to protect credentials. S/MIME will provide additional on-server (in-storage) protection which encrypted POP/IMAP/Webmail alone will not.
Can anybody suggest a best-practice
for a start-up business situation? I would like to be able to deliver
a solution in which it is secure to transmit trade secrets over this
email server between internal users, at the very least.
Encrypt your client access protocol, and don't worry about S/MIME. Here's why:
- You absolutely need to encrypt client access, for credentials* if not for email.
- S/MIME that you introduce at your server introduces key management issues. When Dave in Sales leaves, and you don't have a copy of his key, then all his email to your customers is suddenly unreachable. Are you going to do key escrow? Encrypt with dual keys? You're going to introduce usability and availability headaches if you add S/MIME, and you won't even be protecting your end-to-end like you should get out of S/MIME.
- Server-imposed S/MIME is only useful if your mail server gets compromised. In all likelihood, if your mail server gets compromised, having your mail be encrypted is only one of a large number of problems in your lap. It's kind of like having the deck chairs properly arranged on the Titanic. Bear in mind that most compromises are not discovered for months - an attacker could watch your new email as it comes and goes; not having access to the email that was there before the compromise is not a game-changer for him.
Now, some of that may differ based on the business you're in. If all your business proposition is tied up in mail messages, well, maybe that extra layer becomes worth it. But standard business, best practices, probably not worth it.
* There are, in fact, some client login methods which effectively encrypt the credentials. Generally they will require the server have a plaintext copy of the user's password, which is bad. And support for the different methods may vary while every client/server pair will support LOGIN or PLAIN. As a result, the vast majority of email client access servers encrypt the session and use the plaintext LOGIN or PLAIN password methods.
