Compromised DigitalOcean Droplet, doing DDoS attack, what is correct process to investigate cause?

Question

I was informed by DigitalOcean today that my Droplet on there has been disconnected because it was doing a DDoS attack.

They've asked me to investigate and find out what was causing it.

This was Ubuntu 14, and I had 6 Apache VirtualHosts on there. All were live.

One of my sites was a WordPress installation with a couple of plugins.

Another site had some Google Maps API code in it.

The rest had only my original code in them.

I haven't entered the server yet. Once I do, what would be the process to properly find the software causing this?

I suspect this happened because I wasn't using an SSH key with my password.

Answer 1

First, my condolences on having to deal with something like this this. But you can clean this up. First, I just need to address this:

I suspect this happened because I wasn't using an SSH key with my password.

99% sure that is not the case. Pretty much every web server compromise I have personally dealt with—and cleaned up—over my 20+ years of experience came from application level flaws and not anything connected to SSH or SFTP. In fact, most web developers/administrators will never deal with SSH/SFTP level compromises; flaws in front-facing code is the main entry point for many pieces of malware and unwarranted incursions on public web systems.

And in your case you state the following:

This was Ubuntu 14, and I had 6 Apache VirtualHosts on there. All were live.

One of my sites was a WordPress installation with a couple of plugins.

Another site had some Google Maps API code in it.

My guess is that—unless you keep up-to-date on WordPress updates/patches—your WordPress installs were vulnerable. And not just core WordPress but the plugins as well.

Backup existing codebase, database and configs.

The very first thing I would do is neuter the Apache virtual hosts by perhaps setting an index.php that says, “Down for Maintenance” in each of the root indexes of those sites. I would also make a copy of each virtual host install via a TAR/Gzip archive and download that for potential forensics. This should be done for all Apache virtual servers including dumps of the MySQL databases as well and relevant config files.

Check the /tmp/ directory for any signs of infection; blow it away if need be.

The next thing I would recommend you do is dump and recreate the /tmp/ directory on the server. The reason being is many infections of malware on Linux web servers place a good chunk of their payload in the /tmp/ directory. I go into more depth in this answer here but it boils down to doing the following.

So first, look in the /tmp/ directory and see of there is anything there that shouldn’t be there. 9 times out of 10 malware on Linux systems will be able to install themselves in /tmp/.

If you are unsure what should/shouldn’t be in /tmp/ there is an easy—but extreme—thing you can do to clear out the bad stuff. Just run this online in the command line:

rm -rf /tmp && mkdir /tmp && chown root:root /tmp && chmod 1777 /tmp

Or run each command individually like this:

sudo rm -rf /tmp 
sudo mkdir /tmp
sudo chown root:root /tmp
sudo chmod 1777 /tmp

Then reboot the server to see if that clears things up. If it does, congrats! But you are not out of the woods yet since it whatever caused the original system can still penetrate your system, it’s only a matter of time before they reinfect you again. Meaning, this cleans up the mess caused by a weakness in your system, but you need to find out what that weak-point might be and harden it.

Bash “shellshock” vulnerability checks.

And in that other answer I provide tips on how to check for bash “shellshock” vulnerability. This site provides good testing tools to see if your server is vulnerable to bash “shellshock” exploits. Honestly this has been one of the more common security holes I have had to patch since it was discovered. So there is a good chance this could be a weak point on the server as well. As for how to fix bash vulnerabilities if found, see the section below on how to upgrade/patch all OS level components. Do that and bash should be upgraded/patched as well.

Upgrade/patch all OS level components.

Now that might sound daunting, but the reality is security patches are released all the time for Linux systems. And if you are using a standardized Linux install like Ubuntu or CentOS, then you can simply run an update/upgrade through your package installer like this. On Ubuntu just do this:

sudo apt-get update

sudo apt-get upgrade

Now if you haven’t updated the system in a while you might see some huge number of patches and updates that need to be dealt with. Don’t panic! That’s normal. Just run the update and upgrade and wait. Might have to reboot afterwards, but when that is done your core OS should be full patched and up to date.

Assess the codebase of your Apache virtual server code systems; WordPress and Google API stuff.

Brace yourself: This is the uglier part. You should download and assess the codebase that was potentially infected. Hopefully only one or two sites were compromised. How to go about this is idiosyncratic to each setup, but generally what you should do is load each site into a development environment, upgrade the WordPress core, upgrade the plug-ins, check to see if all works fine and then consider that code clean/stable.

Now I have vastly oversimplified this step. But the reality is even after doing patches and upgrades you could still have malware code in your WordPress core. So the other thing you can do is to rebuild each WordPress site from the ground up. Keep the databases, the templates and basically rebuild the site from a new, clean WordPress core.

Yes, that sounds like a lot of work, but if you have that many virtual servers with different codebases on them there is no choice.

Post-mortem advice.

This doesn’t work for all, but I will say this: Backups and maybe even a GitHub repository for the clean codebase is the way to go to clean up the mess without the messy last “assessment” step I have above.

What I do is run regular MySQL dumps on any database-drive site I work on, and I make sure a clean core codebase is in GitHub. Then if a malware infection happens, I can dig through the MySQL backups and even redeploy clean code from GitHub to assure that the codebase is clean. And if the server itself is just hosed past belief? Well, just dump the infected system, rebuild a new Linux system, deploy the code, import the databases, adjust the configs and call it a day.

Remember, 99% of all websites are just a database, codebase and a set of configs. If you have some clean way to “freeze” or backup non-infected code and a backup of MySQL databases, then all you have to do is deal with the configuration stuff. Which is a minor pain compared cleaning up code from scratch.