assessing the security concerns for running an own email server

Question

I'm working at a small research center that has to manage a lot of things independently on a tight budget. One of these is all things IT infrastructure, including web and email hosting. Though I've not been formed as a system administrator, I've so far succeeded in finding an affordable root server at a hosting company, setting up our websites and web applications, and keeping them running for a couple of years.

So far, our email has been hosted externally, which means that we pay little for little value: 875Mb email storage space for max. 35 POP-mailboxes. After a couple of data-losses due to local inbox corruptions in email clients, I've decided to at least investigate the option of using the postfix / qmail mail server that's present on the root server we're hiring. At first sight, there are many advantages: we'd be able to switch to virtually unlimited IMAP accounts, include the email in the email accounts in the server backups, use more storage per mailbox, etc. On top of that, the mail server is already included in the fee we're paying for the webserver, so we could cut the external email hosting cost and even get much more for less.

The technical part has been fun enough: I've been able to set it all up (discovering the virtues of the Plesk panel) and in principle we could just switch to the new email server right away. Yet, I'm not confident I can properly estimate the risks involved in managing an email server, security wise. Of course, I have SpamAssassin and antivirus (Plesk Premium antivirus) enabled on all email accounts, set up an SSH certificate and added SPF, DMARC, and DKIM records to our DNS. My major concern is: does this suffice, and what are the chances of being attacked and having the entire server compromised?

For example, I've noticed how -even in this premature testing phase-, the QMail logs are full of messages such as:

Dec 15 17:07:00 server4545 postfix/smtpd[23838]: connect from unknown[91.200.13.5]
Dec 15 17:07:00 server4545 plesk_saslauthd[24512]: Invalid mail address 'albert@'
Dec 15 17:07:00 server4545 postfix/smtpd[23838]: warning: unknown[91.200.13.5]: SASL LOGIN authentication failed: authentication failure
Dec 15 17:07:00 server4545 postfix/smtpd[23838]: lost connection after AUTH from unknown[91.200.13.5]
Dec 15 17:07:00 server4545 postfix/smtpd[23838]: disconnect from unknown[91.200.13.5]
Dec 15 17:07:04 server4545 postfix/smtpd[23838]: connect from 24-35-233-74.fidnet.com[24.35.233.74]
Dec 15 17:07:04 server4545 postfix/smtpd[23838]: lost connection after CONNECT from 24-35-233-74.fidnet.com[24.35.233.74]
Dec 15 17:07:04 server4545 postfix/smtpd[23838]: disconnect from 24-35-233-74.fidnet.com[24.35.233.74]
Dec 15 17:07:20 server4545 postfix/smtpd[23838]: warning: hostname ip-address-pool-xxx.fpt.vn does not resolve to address 118.71.172.216: Name or service not known
Dec 15 17:07:20 server4545 postfix/smtpd[23838]: connect from unknown[118.71.172.216]
Dec 15 17:07:22 server4545 postfix/smtpd[23838]: NOQUEUE: reject: RCPT from unknown[118.71.172.216]: 554 5.7.1 <name@ourdomain.com>: Relay access denied; from=<voicemailandfax@jgnluxwatch.com> to=<name@ourdomain.com> proto=ESMTP helo=<[100.74.205.159]>
Dec 15 17:07:22 server4545 postfix/smtpd[23838]: disconnect from unknown[118.71.172.216]
Dec 15 17:07:22 server4545 /usr/lib/plesk-9.0/psa-pc-remote[8577]: Message aborted.
Dec 15 17:07:22 server4545 /usr/lib/plesk-9.0/psa-pc-remote[8577]: Message aborted.

This seems to suggest a lot of interest from totally unrelated IP addresses, even though I'm currently not even using the thing for testing. The good part seems to be that those attempts are being recognized and rejected, but still I'm wondering where they come from and how much I should worry. Even if they are innocent, I can imagine processing all those requests could cause serious overhead for the server. For testing purposes, I have added an MX record with low priority in our DNS; I wonder if this already is 'inviting' so many requests to our mail server?

In other words: I'm looking for reasonable advice on the security risks involved in running an own mail server. After all, if things break, they might break everything and I'll have to deal with it myself.

Any advice would be much appreciated!

Ron

Answer 1

As an ex-admin who then became a penetration tester, I feel your pain. However it turns out that because, like you seem to, I cared enough about the security of my system to find out what things like DKIM and SPF are, and implement them, and use SSH keys etc. it turned out that I was doing most things right. As it sounds like you are ;-)

From your logs, it doesn't look like you are, but one of the most important things is to ensure you are not running an open relay. The following link should allow you to test that: http://mxtoolbox.com/diagnostic.aspx

Also you need to prevent user enumeration. You don't want an attacker to be able to log in to your SMTP server and do: "VRFY bob" or "EXPN joe" and have the mail server reply to tell you whether joe exists on the system or not. It is possible to configure mail servers to not support those features. You also don't want the mail server to respond to "MAIL TO: tom" to immediately tell you that tom exists. You get the idea. Look into smtp/pop3/imap user enumeration and mitigate against it.

Another thing - don't use pop3 or imap without encryption. The password will go over the internet in clear text - Wifi is often easy to hack as well, and this will give away the credentials to unencrypted mail services in a moment. You need transport layer security (TLS).

Similar with SMTP, you want to require authentication for remote users sending mail, but this must also be sent over a TLS connection. Also, don't forget to require authentication for local users sending mail, and prevent open-relaying internally. I recently did a pen-test at a law firm where I could telnet to their mailserver on port 25 and send a mail from the head of the company to an employee, telling him he had a raise or was fired. Obviously I didn't do either, but you don't want to allow the possibility. It is best to have non-repudiation if possible.

As it is clear you are good at finding stuff out for yourself, I won't go into great detail, but suffice to say - you need to know about the relative merits of things like POP3S and POP3 with STARTTLS, the difference between ports 25, 465, 587, etc. IMAP and IMAPS, etc. Requiring client TLS certificates for authentication is probably way over the top, but don't use self-signed server TLS certificates, you should get bona-fide ones. You can get them cheap from go-daddy or similar. Don't use SSL2, SSL3, or TLSv1.0. TLSv1.2 should be all that is enabled if possible.

Also, configure your mail servers not to announce their software version (or any product details if possible) when you connect to the port. A simple generic banner should suffice.

Require strong passwords from your users AND admins!

One more thing, to avoid spam, grey-listing can work very well if configured correctly. I used it for a long time and never had issues with undelivered mail.

Machines on the internet are always being scanned and poked at. If you have configured it securely then I wouldn't worry to much about that (although you could use iptables rate limiting if you are concerned about DoS - or speak to your ISP), BUT on the PLESK etc. control panel side of things and SSH, use iptables to whitelist the IPs that REQUIRE access to those ports, so that attackers cannot even see that they are there. Use nmap on your server to see what attackers will see. If all ports appear filtered to the general public except your mail ports, then great. Only use HTTPS, not HTTP to connect to the web administration stuff and again, only TLSv1.2. Or better still, get it only to listen on localhost so it is inaccessible from the outside, and then use SSH port forwarding to connect to it instead!

Google for "server hardening PRODUCTNAME" for each software item you are using, and also "penetration testing PRODUCTNAME", "PRODUCTNAME exploits" could help.

Very important - Keep all software and the linux OS as patched and up to date as possible.

An often overlooked risk is the risk of your own workstation being compromised. As an administrator you are a prime target. If this happens then an attacker may be able to steal your passwords or SSH keys and find your files that will tell them everything thing they need to know about your config. Once they have that, then they can get into your mail system no matter how well you have secured it, even if it is a hosted Office 365 solution. In fact, in some ways it will be easier for them if it is hosted elsewhere and a more familiar configuration.

Don't log into your machine as root or administrator unless you need to. Don't give out more access to your colleagues than they need. etc. Go very careful with opening e-mail attachments etc - the usual stuff.