1

A few days ago, my company server was attacked by hackers who wiped the whole database in /var/lib/mysql/ folder.

Fortunately, I was able to restore everything from backups, but I'm not able to find the location of a security breach.

I found a bunch of .exe files in /var/lib/mysql/ folder which in my opinion definitely should not be there. The hack took place on 29 Nov, so I think the files responsible are ending with "exp.exe" and "401hk.exe" which when opened contained following line:

http://104.223.17.73:4568/401hk.exe

contents of /var/lib/mysql/

Is it possible that hackers wrote executables into the /tmp folder and moved those files to /var/lib/mysql where they executed the attack remotely?

It is critical to me to find the vulnerability to prevent such attacks from happening.

So far i checked all the access logs, run LYNIS security audit - found nothing critical, ran an rootkit check - found nothing.

Maybe someone has experience with a similar attack?

Marko T
  • 11
  • 2
  • It will be difficult for us to outline all the ways you could use to figure out where to find the security hole. – schroeder Dec 06 '16 at 07:30
  • @schroeder i'm sure it would but, im interested how those infected files got into /var/lib/mysql/ in the first place. – Marko T Dec 06 '16 at 07:39
  • I get that, so am I, but I'm not sure it's possible for us to tell you that with so little information. – schroeder Dec 06 '16 at 07:40
  • @schroeder I'm more than happy with providing more information, if you could tell me what you need. – Marko T Dec 06 '16 at 07:47
  • More information than you could show here: The full architecture of your site, logs, and analysis of the uploaded files. It could be a SQL injection attack, a misconfigured service, a poor password, or a vulnerability in some portion of either your site code or your infrastructure. I go back to my first comment. What you need to do is to narrow down the options. – schroeder Dec 06 '16 at 07:52
  • I assume that you have reset all the passwords on the server and service? I assume you've patched everything? I assume you've increased logging temporarily to see if you can gather more data? – schroeder Dec 06 '16 at 07:53
  • Unless you can narrow down the question, I'm closing this as having been answered by another question. In its current form, it is too broad. – schroeder Dec 06 '16 at 08:04

1 Answers1

-1

I'd strongly recommend imaging your disk and then wiping it securely and re-building the server. Ensure the compromised server is disconnected from the network immediately to stop any compromise spreading. Work on the disk image not the compromised server if you want to check the possible source and extent of the compromise.

Don't rely on restoring data to overwrite a virus or backdoor - you need to securely wipe the disk and rebuild the server. Scan and check backups as well before using them.

Presumably you have anti-virus running on the server? If you have then either it's not updating regularly, it's a zero day virus, or a direct hack that gained access to some level of interactive login. If you haven't install one and scan the image/disk - it may tell you what is going on. Scan all backups too.

If someone malicious has gained access to your server then, with sufficient permissions, they could move files wherever they wanted to. A full forensic investigation is needed to find out the cause and extent of your compromise.

You can start by checking access logs for interactive logins - tie up recent login access with specific accounts and check with the account owners to discount those logins - or not!

You can't trust timestamps on files - they can be changed easily. Do you have any intrusion detection system with file integrity checks? What does that system say about changed files and dates?

There are so many steps to take to try to track this down but if you're not prepared for a breach then it's very likely you'll not find the source of the issue easily if at all.

Also consider what data was in the database in terms of sensitivity and confidentiality. It's possible that the compromise has leaked some, or all, of the data from your database - if someone can wipe it they can almost certainly transfer the data out of the network (depending on your firewall egress rules).

Finally don't forget to check all servers and devices that this server has access too i.e. can connect to from services or accounts on this server - the compromise may have spread.

There's plenty of information on line on how to investigate breaches to find possible causes and how to clear up - don't just restore some data and hope for the best though if you don't know precisely how the compromise occurred and what damage/changes has taken place.

David Scholefield
  • 1,824
  • 12
  • 21
  • I think the whole thrust of the question is the "full forensic investigation" part that you don't actually get into. If he restores everything like you say, then the vulnerability remains. – schroeder Dec 06 '16 at 08:00
  • The question isn't "what to do after an attack" (we have a canonical question for that), but how to find the vulnerability. He's asking about how to conduct forensics. – schroeder Dec 06 '16 at 08:02
  • After attack, i turned off the server then immeditely made snapshot of attacked machine. I set up a fresh virtual machine at google cloud computing with fresh LAMP setup, restored the database, examined any changes to /var/www/ files and copied the contents to new server to make service accessible again. – Marko T Dec 06 '16 at 08:07
  • I agree with the thrust of the question. If he restores everything like I say he won't necessarily be vulnerable though, the attack vector may have been historic that isn't restored from a clean rebuild or backup. But I agree with the main point. Is there a canonical answer for how to do forensics in this scenario or is that too wide? – David Scholefield Dec 06 '16 at 08:58