Brutalized VPS recovery data now available. Considerations?

Question

Backstory
My sites and VPS were stolen from me. The hosting company and I were locked out and unable to access it. They weren't able to create a temp password for access because the attacker blocked it. The last time I was logged into WHM, root control was taken and all HDDs were no longer bootable. I believe the same person used a worm to monitor my desktop remotely. 5 pcs and 5 mobile devices, bricking my ddwrt r7000 router with PIA VPN killswitched. A dozen or so mint/ubuntu vms were taken over. Many usb drives were made to be write-only.It was relentless. I stopped trying to figure out what was going on and reformatted all devices.

I am now waiting on the server image and a memory snapshot, as well as an rsync copy. Upon transaction ill get a fresh server image...certainly of a different IP, unless they aren't to be convinced.

Here is the email I received today:

Blockquote This ticket was just assigned to me. I have made a backup of the account >out of the way of the backup processes here.

[2018-03-25] pkgacct completed

I was reading over some of the conversations and would like to just make sure we are on the same page.

We can not dd ( bit for bit copy ) the current state because the account does not have a storage medium that can handle it. If you wanted to add keys that we can rsync over ssh to a remote destination just give us the destination of the output file and we will be happy to help with that.

We normally do not keep hacked operating systems around unless there is some specific interest. What I find interesting is the openVPN software:

uperior.hosting.com [home]# ifconfig as0t0 Link encap:Ethernet HWaddr inet6 addr: Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:436367538 errors:0 dropped:504 overruns:0 carrier:0 collisions:0 txqueuelen:200 RX bytes:0 (0.0 b) TX bytes:26310498062 (24.5 GiB)

asbr0 Link encap:Ethernet HWaddr inet addr: Bcast: Mask: inet6 addr: Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:431222324 errors:0 dropped:0 overruns:0 frame:0 TX packets:1492069 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:20595591150 (19.1 GiB) TX bytes:634373123 (604.9 MiB)

If you are using this server as a production WebServer I would recommend using CentOS 7 and installing something like 'Cockpit' instead of using a VPN through a cPanel server on CentOS 6.

What is in the scope of our support is to insure you have a path to and from the server while you salvage the barebones data. I will conclude with some options and questions. (The following are my answers)

1. An inquiry about the openvpn activitu

2. Letting them know, again, I want a copy of the server image, a memory snapshot, and all logs available

3. Fresh installation of cPanel/WHM

4. Fresh IPs for sites and VPS

5. SFTP info

Blockquote

Is having a VPN, IDS, firewall, honeypot enough? Have I left anything out?

VPS is a from Bluehost, running CentOS 7 with any alternative to WordPress...

Answer 1

I'll start with what to do with your current system:

1. Get in and make a backup of everything.
2. Unless you can demonstrate major losses ($10k+), I wouldn't even begin to think about involving law enforcement. They have their hands full, and given the current patterns on the internet, it's highly likely that your culprit is in a different country than you are. Nobody is going to do an extradition process for hacked Wordpress sites. (Sorry, I know it's hard to hear, but it's the reality.)
3. Burn the current server to the ground.
4. Consider every password on your old server compromised.

Now, how do you build a new server to avoid this happening again? I'm going to make a few assumptions based on what you wrote in your post:

• Multiple Wordpress installs.
• mod_php on Apache
• CPanel/WHM

Here's a few recommendations:

1. Get your new server.
2. Do a fresh installation of your applications, and setup strong passwords/credentials that have nothing to do with your old ones.
3. Configure SELinux to limit the exposure of each site as much as possible.
4. Be careful what wordpress plugins you install. They have a much worse security track record than wordpress core.
5. Ensure that directories that are writable by the webserver are never interpreting files as PHP.
6. Use 2-factor authentication for everything you can.
   • CPanel
   • SSH

Without being able to do digital forensics on your system, it's hard to know for sure, but I'm going to go out on a limb and guess what happened:

1. Attacker runs scanner looking for vulnerable Wordpress installs.
2. Attacker finds vulnerable Wordpress on your server. Gets RCE as webserver.
3. Dumps password hashes for wordpress databases.
4. Cracks password hashes, one of them matches a WHM/CPanel password.
5. Gets into CPanel. Maybe as an admin, or maybe the version of CPanel had a bug that allows privilege escalation to admin.

You talk about fear of retaliation and the attacker coming after you again and again. Unless this is personal (a vendetta of some sort), I wouldn't worry about that. Attackers like this will just move on to their next compromised host. Just don't give them another chance.

Answer 2

David's answer gave some excellent recommendations (which I highly recommend you follow). I will focus my answer instead on your specific fears in hope to alleviate them.

What I wish to accomplish is to acquire as much information about the hacker as possible. I want to download my legacy backups. I want to get in and out as soon as possible.

You say you want to get as much information about this person as possible. This may not be practical if they used any form of anonymity. Post-break in analysis is still important, of course. The first step would be to do a complete backup of the disk. If you have not already shut the system down since it was compromised, you may also be able to take a snapshot of the memory on the system, making it possible to do later forensic analysis on it. Memory will contain far more information about the attacker as they cannot reliably control what stays in memory, whereas it is very easy to prevent valuable information from being saved to persistent storage. Because of this, it's likely that you will not be able to obtain the disabled logs unless they already existed and were subsequently deleted.

My precautions are to stay behind a killswitched vpn and spend as little time there as possible.

There are a number of methods hackers can use to get the original IP behind a VPN, due to the architecture of VPNs. Using Tor or a similar anonymity network would be preferable. Spending as little time as possible may not be advisable as you may miss something important. Staying on a little bit longer does not increase the chance that you are detected. The hacker may even think you are another hacker who has broke in using the same vulnerability. If they notice you, chances are regardless of who they think you are, they'll simply disconnect you from the server.

The server-side SSH logs among others likely contain your home address anyway.

Is there any way to somehow recover these disabled logs? What else should I be looking at if the goal is to compile enough data to turn this guy into the authorities?

If the logs were disabled, you likely cannot retrieve them. If they were still written but merely deleted, you may be able to recover them from unallocated filesystem space. Unless the system has not shut down since the compromise and you are able to take a full snapshot of the system's memory, it is unlikely that you will be able to retrieve the information that was given in the logs.

As David already mentioned, you will not likely get much from involving law enforcement. The best you can get from obtaining this information is knowing more about how the hacker broke in, making it possible for you to close the hole they exploited and avoid it in the future.

My fear is that, even with a re-imaged server and all the hardening I'm capable of, this guy knows my site domains and will likely attack again. I suppose this should be a separate question, but should I prepare myself for having to altogether give up my domains due to his skill level? Id hate to do it, but I fear he will do all he can to wreck the server and sites again, then proceed as before to destroying my home network and all devices.

Honestly, you don't have to worry about retaliation. You are just random collateral to their activities, not a carefully selected victim. They likely won't even remember your domain in a week. Of course these are generalizations, and if you have an actual enemy with a personal vendetta against you, things are a little different. There are a few possible classifications that may fit your attacker:

• Curious - A hacker who breaks into a site because they are curious or want to improve their hacking skills will do so often with little regard for the results of their actions, but they are not out to get you. If you lock them out, they may want to find a way to get back in, but they will likely not be angry. They could even see it as a challenge.

• Criminal - A purely criminal hacker has something to gain by attacking servers, such as money or digital resources. They won't waste time with revenge. These attackers are like water. They seek the path of least resistance.

• Automated - It is very possible that no person attacked your site. A criminal enterprise wants to maximize profits, so to be more efficient, they often automate attacks. A script may scan the internet for vulnerable services and attack them. After breaking in, it will do their dirty work and attempt to establish persistence. You cannot make a script angry.

• Script kiddie - A script kiddie is a junior hacker who thinks they know everything. They are more likely to seek revenge because they may take the act of you recovering your server as personal. They do not have the necessary skills to do any real harm to your home network. The worst they can do is attempt to DDoS you. Don't worry about them.

• Personal enemy - If you have an actual enemy with a personal vendetta against you, you have more to worry about. Since you gave no indication that this is the case, I'll assume it isn't.

The chances are his skill levels are not as high as you think. Breaking into a Wordpress site and adding a backdoor does not take a professional. As such, I would not worry about getting rid of your domains. It is absolutely possible to protect from this hacker using the advice given in another answer. As soon as someone else's site is more vulnerable than yours, a hacker moves on.

Answer 3

In this case you don't know how the attacker was able to enter, so you are right to be concerned that the exact same thing will happen again even if you do a complete reinstall. To deal with this problem, you should configure your system for remote logging. A dedicated separate computer saves system logs which cannot be tampered with. You should configure your web server access log to send a real time copy over to the logging server as well as anything security related, in addition to the regular system log. The logs should be separate so that if one gets too big and needs to be trimmed down that it won't affect the others.

You don't know how they got in. Your own computer could be compromised and the password could have been intercepted. Someone at the VPS provider could have accessed it, or someone renting another VPS along side yours could have exploited something. Assuming that those things didn't happen, the next step may be to ensure that everything vulnerable is updated on the new install, and then configure security software such as SELinux. Be sure SELinux policy violations are sent to the remote logging server.

Answer 4

My precautions are to stay behind a killswitched vpn and spend as little time there as possible.

That is infosec voodoo. Assuming you're behind a router with PAT, the attacker is going to have no way to connect back to you. In fact, the VPN creates a tunnel through the PAT layer, so you're actually increasing your exposure by using it! (Althought the VPN policy may prevent inbound connections.)

The much greater risk is that if the attacker still has access to the compromised system, then typing in your password is dangerous, i.e. SSH interactive login or sudo password could be stolen by a root user on the compromised system. If you have public key authentication configured, then that would be safe to use as long as you don't use sudo, however see my note below about backups...

The best practice is to remove the hard drive from the compromised machine and forensically analyze it on a separate machine (a professional would use a forensic write blocker) so that the attacker has no ability to interfere.

As a VPS customer, you have no ability to do this yourself. Unless your provider is willing to do it for you (unlikely since it would disrupt other tenants on the same hardware), you are better off deleting the VPS and starting from scratch.

Hopefully I can get what I need from backups and [snip]

Backups should always be stored on a separate system. If your only backups are stored on the compromised service, then you have put yourself in a very bad position. If you have to type in a password to get to the backups, then the attacker can steal your password.

Even if you don't need to type in a password (e.g. public key authentication), you would be restoring data that has already been breached. If the attacker installed some backdoor previously, then that backdoor would be saved in the backups, and you would end up restoring that backdoor onto your new server! Or if the attacker used some other means to compromise the system, then the attacker could have inserted a backdoor into your backups in order to regain access after you re-image.

Is there any way to somehow recover these disabled logs?

Yes, it is possible to recover deleted log data using forensic analysis, but for the reasons stated above, this is unlikely to be an available option for you.

As an alternative, your provider may have logs about what IP addresses are connecting to their systems, and they may be willing to share that data with you if you ask nicely.

My fear is that, even with a re-imaged server and all the hardening I'm capable of, this guy knows my site domains and will likely attack again.

I'm not really sure what you mean by "domains". You can keep your existing hosting account and domain names, assuming that your password has been reset to something secure.

The most likely scenario is that you were compromised due to an unpatched vulnerability in WordPress or one of your plugins (or custom code if you've written any of it yourself). To prevent a future compromise, you need to start from scratch (don't use a backup of data) and make sure that all of your applications and plugins are the latest version. Disable any plugins that you don't absolutely need.

It sounds like you might be a bit over your head. You are managing complex software by yourself, and you don't know how the previous compromise occurred. Maybe you should move your site to a provider that offers better patching and security, such as hosting through WordPress.com, so that the responsibility isn't 100% on you.

Answer 5

Most hosting providers make available a small OS image (the purpose of which is to facilitate initial OS install when a client wants some custom OS, or to fix errors preventing the OS from booting; typically called “rescue system” or something) which you can boot and gain access to the compromised VPS without ever launching any malware the attackers could have put there. You just mount your volume(s), copy all the necessary data and examine it at your leisure.

If you can’t do this, your only concern is not to grant the attackers any further attack vectors. Don’t SSH to other machines from the compromised one, make sure you aren’t using SSH agent forwarding, X11 forwarding and the like. Just tar the interesting files and then scp the tar file from your local machine. Or do something like ssh -C root@compromised-host cat /dev/the-root-device >compromised-root-dev.img if you know what to do with the image then.

Or maybe just ask the hosting provider to make a filesystem image and send it to you in some way (which would amount to the same procedure I started with).

Answer 6

Is there any way to somehow recover these disabled logs?

Yes.

If the entire virtual machine is yours (and you aren't sharing it with someone else), get a backup of your entire virtual machine. Do that immediately. That will include whatever logs that your machine has.

Also ask Bluehost for whatever relevant logs they can provide. Do that immediately so that they don't lose the logs. (Many places will cycle out logs over time, so make sure that someone knows to preserve logs that are believed to be potentially interesting.)

Then focus on getting a safe machine operational. (As BlueHost mentions, this likely involves an OS reload.) Ensure that your data is safe. (e.g., if your data includes information about accounts, and includes an account that the attacker uses to gain access, that is not safe.) Have that safe machine use the safe data that you value. Do all that as quickly as possible.

What else should I be looking at if the goal is to compile enough data to turn this guy into the authorities?

Where is this attacker from? Who is this attacker?

As forest notes, "authorities" are unlikely to care much. Chances are that you may have been attacked by some other compromised machine. Therefore, the person who owns that machine is also a victim, and trying to legally attack that person will probably not benefit anybody very much. What you need to do is to get after the person who really initiated the attack. Unfortunately, with several methods of redirection being possible, many attackers are not feasibly traceable.

My fear is that, even with a re-imaged server and all the hardening I'm capable of, this guy knows my site domains and will likely attack again.

Yup. That's a concern. Until the person is stopped, there is such potential. Hopefully you are more successful at preventing the next attacks. Ongoing education and efforts should get you to that point, hopefully sooner rather than later. In the mean time, ensure you have good, accessible backups, and sleep well at night knowing that you can restore from backup.

I suppose this should be a separate question, but should I prepare myself for having to altogether give up my domains due to his skill level?

Nope. Don't give in.

Id hate to do it, but I fear he will do all he can to wreck the server and sites again,

That could happen if you don't secure the server more successfully. Try to spend time doing that. Also, make sure you have good backups. Don't just trust Bluehost. I'm not trying to say anything bad about Bluehost. I'm saying, don't just trust one backup solution, especially if trouble is expected.

then proceed as before to destroying my home network and all devices.

Oh, now that's just terrible. Why would damage to a remote virtual machine, run on a professional hosting site, affect your home network? There's some major security problems if that occurs.

Is this even worth it? He's capable of far worse than what he did and I am in fear of retaliation.

Get backups of your web site.
Get backups of important data on your home network.

If you do a good job of that, then even if thievery ends up being possible (a copy of confidential information may be stolen), you can still recover.

Answer 7

Just get an image of the web server drive. Mounting it on a fresh virtual machine is unlikely to cause significant hazard. Code from the mounted drive should not run, you can peruse the data at your leisure.

It is most certainly not safe to connect to a compromised machine with a remote access protocol. While servers are more constantly available to be compromised, client software connecting to a malicious server can be exploited (if an exploit exists).

Attempting to restore your web content from the content on the mounted drive is probably ill advised. Modifications to code files can be quite hidden and difficult to find. Using PHP, they can easily provide a backdoor to your next VPS machine...

Stuff in the SQL database might be modified to contain code to attack visitors to your website.

If you really want to harden your site get rid of WordPress. Wonderful things are being done with static site generators (Gatsby, Jekyll, etc...)

Answer 8

As some information alluded to here - if you used these compromised credentials anywhere else you must change your passwords. Preferably, you should use a good tool like Keepass or Lastpass that allows you to protect any number of strong, generated, random passwords, and you only have to remember the one main passphrase.