Dealing with spam scripts as a shared hosting operator, with users who host mail externally

Question

Assume the following setup;

1. A shared hosting server, sharing a single IP with many small websites.
2. A subset of the set of websites is sending e-mails. Some of those e-mails may be to their own domain.
3. Another subset of the set of websites is hosting their mail on another platform.
4. A third subset who host is vulnerable to a remote code execution vunerability.
5. In the Venn diagram for subsets (2), (3), (4), assume any part can be nonempty.

There are issues with spammers becoming more and more competent at circumventing the installed security measures. With full code execution access, a hacker can circumvent the usual Linux mailing stack and instead implement their own E-mail server, directly communicating via socket code to external mail servers and sending a massive amount of spam.

A server is a pretty powerful machine, so such free-roaming code can send significant enough amounts of spam to get on the radar of the Big Boys, such as Microsoft and Google. These companies have their own home-rolled obfuscated proprietary systems which tend to block based on IP. I know that this is a stupid thing to do: IP does not equal a unique machine. Obviously a small hosting provider can do little about these Big corporate policies, and you cannot ignore them either; lots of consumers (e.g. users of the websites on the platform) will have e-mail addresses on the platforms of the big providers.

There's also an element of secrecy involved; the big companies are actively trying to fight the spammer botnets, so they provide no or minimal further information about any blocking so the spammers can't use this information. By sending spam that's 'signatured' they may have identified the particular botnet and marked the hosting server as a suspected cybercriminal tool. It's not entirely unreasonable, because truthfully that's what the server is being used for at that point.

IPv4 address exhaustion means that non-shared hosting is just a non-option for these small websites. It's financially just not achievable.

Note; assume there will always be some vulnerable sites, that's an intrinsic weakness of a shared platform, not every customer will have their code equally secure.

Upon being hacked, the customer is of course notified. Standard procedure is to take the site offline and reinstate a clean version, requiring the developers to fix the security issues before that.

Still, there remains a pretty big problem; One tiny mistake for one out of many websites will effectively shut down the e-mail for the entire server, and getting everything back up again can take days by curing rather than preventing this kind of issue. As sites get older, more sites start using the same IP, and spammers get ever more competent, the situation will get worse and worse over time.

While I can think of some partial solutions, so far they've all been unable to keep the full feature set we're providing. For example, I've tried blocking outgoing traffic over the standard E-mail port (25) through the server's firewall for all users except root and a trusted e-mail user. This will fully prevent our spam problems, because this means the websites are required to use the standard stack, and it's simple to configure things like rate limiting and spam filters for outgoing mail on this standard stack.

The tiny amount of spam that can still get through would not put us on these secret ip-blocking lists.

This works for most of the websites; but not all. Subset (3) is throwing a wrench into the works if combined with (2). Let's illustrate:

Our host (IP = H) is supposed to send a message to a microsoft host (IP = M), through its mail server. Say we send from 'noreply@example.com' to 'contact@example.com'. If this message is sent directly, everything works. However, when sent via the mail server, it needs DNS records. But then it tries to send to localhost (H), this fails, because (H) does not have the mailbox 'contact', that should only exist on the microsoft host...

Furthermore, plain blocking everything like that may not work for all the websites; some may have their own built-in mail engine, although if with some fiddling we can overcome the problem above I'm not impartial to telling these people to fix their software to use the correct mail program.

I'm interested in opinions about smart(er) ways of going about this conundrum.

Answer 1

This is a long and complicated topic, but I will try to answer as concisely as I can.

Basically, you are mixing three kinds of problems here:

• Abuse of legitimate email sending facilities which your server provides;
• Execution of malicious code on your webserver which may send emails to any outside recipients on their own, i.e. without using the legitimate local facilities;
• Proper configuration of DNS records.

I will leave out the latter one as this would make the answer far too long, and focus on the first two kinds of problems.

You are not saying anything about how many machines and IP addresses you have at hand in order to separate things, but with reasonable resources, my advice from years of practice would be as a first line of defense:

1. Do not allow your webserver to sent email anywhere except to an SMTP relay server which you own and control and which is separate from the webserver in case of compromise. Put an external firewall to the web server machine which will prevent any IP connection originating from your webserver to port 25 on any server except your SMTP relay server. (Some people may tell you to prevent not only connections originating from your webserver to port 25 but to all possible ports as web servers should theoretically reply to requests and not start any requests themselves. Unfortunately, this will give your customers trouble if they for example run any CMS which wants to talk back to its extension repo, for example. I will leave this subject to a separate discussion.)

2. In the connection between your web server and your SMTP relay server make sure you use some form of authentication. If necessary, create a user for each website on your SMTP relay server and require the webserver to authenticate to the SMTP relay server in order to send mails. This will make it extremely simple for you to switch off email sending for just one specific website.

3. On your SMTP relay server, implement reasonable techniques to prevent spam. Usually some combination of rate limiting and possibly content scanning (think Spamassassin) where you would just filter out very high scores should help.

One could imagine some scripting magic then when an emerging spam outbreak is detected on the SMTP relay server the user belonging to the sending website will be blocked automatically to prevent damage.

After all, keep a 2nd and 3rd SMTP relay server in cold standby with a different public IP address and if possible in a different ASN. Should your countermeasures fail and the IP address of your first SMTP relay server get "burned" you can immediately switch over to one of your "yet unknown" servers to ensure proper outbound email delivery.

Note: When implementing this, make sure you properly understand things like DKIM and SPF and set the proper records. Then that will work. If you need to switch off a server which made it onto some blocking lists (most of them not being that secret by the way) you will usually get de-listed after some time (think weeks, not hours) and that IP would be available again for some kind of round robin.

Answer 2

ThorstenS's answer is spot on (+1 from me); in order to control the email these websites are sending, you need to run them over your own server.

However, I disagree with some of the details - when talking about those small websites on shared hosting, the customer typically is a small entitiy with no technical knowledge at all, who had some friend, an intern, or a small web design company put up a web site a few years ago, and for whom 'configure your software to use this, and only this, mailserver' or 'make sure your software authenticates to the mailserver' is way beyond their ability.

So what I'd do isn't block ports, I'd redirect them to my own mail server. This is quite easy using iptables if you're using a Linux host; an example of this can be found on ServerFault. This is transparent to your users, so no-one needs to reconfigure anything, and you can modify your destination, again without telling anyone, if the need arises.

Next, rate limiting by user - there's two (well, three) approaches to this without requiring authentication. Either, put a user-dependent tag on your SYN packets, and use iptables's rate limiter. Or, use that tag to redirect to different ports on your mails erver, have your mail server software listen to each of them, and use the port to distinguish between users. Or, the solution that's probably the most robust (and the least performant), when a tcp connection comes in on the mail server, get the remote port number, call netstat -ntp or lsof or similar on the web server to get the owner, and use that to do your rate-limiting.

If a user insists they have to use a specific mail server (for example, I'm using mailtrap.io on my test sites), either open the port to that address (which can be a pain with addresses being moving targets), or just redirect them to another config on your mail server that does the rate limiting thing and has the real target as its forwarder.