MySQL hacked on AWS AMI: 'Pay to get data back' - how could this be possible and how to avoid it next time?

Question

Today morning I noticed that some of the websites I host on a EC2 instance aren't working. When I verified the MySql database, it was wiped out! :( The only thing I have found was only a record telling me I was hacked and to pay if I want my data back :D ... anyway.

How did they manage to get into my DB? What steps should I do now to secure my instance/DB?

ports open:

This is my MySql log, I would really appreciate if someone could have a look and tell me some about:

2017-03-18 15:27:19 14056 [Note] InnoDB: Shutdown completed; log sequence number 5692547
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'BLACKHOLE'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'CSV'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'MEMORY'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'MyISAM'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'MRG_MYISAM'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'sha256_password'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'mysql_old_password'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'mysql_native_password'
2017-03-18 15:27:19 14056 [Note] Shutting down plugin 'binlog'
2017-03-18 15:27:19 14056 [Note] /usr/libexec/mysql56/mysqld: Shutdown complete

2017-03-18 15:27:20 12178 [Note] Plugin 'FEDERATED' is disabled.
2017-03-18 15:27:20 12178 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-03-18 15:27:20 12178 [Note] InnoDB: The InnoDB memory heap is disabled
2017-03-18 15:27:20 12178 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-03-18 15:27:20 12178 [Note] InnoDB: Memory barrier is not used
2017-03-18 15:27:20 12178 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-03-18 15:27:20 12178 [Note] InnoDB: Using Linux native AIO
2017-03-18 15:27:20 12178 [Note] InnoDB: Using CPU crc32 instructions
2017-03-18 15:27:20 12178 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-03-18 15:27:20 12178 [Note] InnoDB: Completed initialization of buffer pool
2017-03-18 15:27:20 12178 [Note] InnoDB: Highest supported file format is Barracuda.
2017-03-18 15:27:20 12178 [Note] InnoDB: 128 rollback segment(s) are active.
2017-03-18 15:27:20 12178 [Note] InnoDB: Waiting for purge to start
2017-03-18 15:27:20 12178 [Note] InnoDB: 5.6.35 started; log sequence number 5692547
2017-03-18 15:27:20 12178 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2017-03-18 15:27:20 12178 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2017-03-18 15:27:20 12178 [Note] Server hostname (bind-address): '*'; port: 3306
2017-03-18 15:27:20 12178 [Note] IPv6 is available.
2017-03-18 15:27:20 12178 [Note]  - '::' resolves to '::';
2017-03-18 15:27:20 12178 [Note] Server socket created on IP: '::'.
2017-03-18 15:27:20 12178 [Note] Event Scheduler: Loaded 0 events
2017-03-18 15:27:20 12178 [Note] /usr/libexec/mysql56/mysqld: ready for connections.
Version: '5.6.35'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2017-03-18 16:06:17 12178 [Warning] IP address '27.18.88.215' could not be resolved: Name or service not known
2017-03-18 18:29:03 12178 [Warning] Hostname 'thinkdream.com' does not resolve to '14.192.9.41'.
2017-03-18 18:29:03 12178 [Note] Hostname 'thinkdream.com' has the following IP addresses:
2017-03-18 18:29:03 12178 [Note]  - 103.206.122.114
2017-03-18 18:38:36 12178 [Warning] IP address '117.44.26.66' could not be resolved: Name or service not known
2017-03-18 19:37:22 12178 [Warning] IP address '49.4.143.152' could not be resolved: Name or service not known
2017-03-18 21:24:57 12178 [Warning] IP address '49.4.135.14' could not be resolved: Name or service not known
2017-03-18 22:03:15 12178 [Warning] IP address '171.221.233.50' could not be resolved: Name or service not known
2017-03-18 22:36:58 12178 [Warning] IP address '182.18.72.116' could not be resolved: Name or service not known
2017-03-18 23:05:57 12178 [Warning] IP address '146.0.72.199' could not be resolved: Name or service not known
2017-03-18 23:05:57 12178 [Warning] IP address '146.0.72.199' could not be resolved: Name or service not known
2017-03-18 23:51:04 12178 [Warning] IP address '49.4.142.104' could not be resolved: Name or service not known
2017-03-19 00:18:55 12178 [Warning] IP address '222.187.224.190' could not be resolved: Name or service not known
2017-03-19 00:22:02 12178 [Warning] IP address '49.4.135.189' could not be resolved: Name or service not known
2017-03-19 01:26:56 12178 [Warning] IP address '182.18.72.82' could not be resolved: Name or service not known
2017-03-19 01:49:36 12178 [Warning] IP address '118.193.165.12' could not be resolved: Name or service not known
2017-03-19 01:52:47 12178 [Warning] IP address '107.179.126.47' could not be resolved: Name or service not known
2017-03-19 01:55:14 12178 [Warning] IP address '49.4.142.189' could not be resolved: Name or service not known
2017-03-19 04:27:45 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:27:54 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:28:06 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:28:15 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:28:15 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:28:26 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:28:38 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:28:56 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:29:15 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:29:33 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:30:13 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:30:44 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:31:17 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:32:05 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:32:22 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:32:58 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 04:32:59 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 05:23:02 12178 [Warning] IP address '113.108.21.16' could not be resolved: Name or service not known
2017-03-19 07:18:40 12178 [Warning] IP address '61.177.139.252' could not be resolved: Name or service not known
2017-03-19 07:18:40 12178 [Warning] IP address '61.177.139.252' could not be resolved: Name or service not known
2017-03-19 08:59:45 12178 [Warning] IP address '49.4.142.178' could not be resolved: Name or service not known
2017-03-19 12:28:36 12178 [Warning] IP address '107.179.45.19' could not be resolved: Name or service not known
2017-03-19 15:47:23 12178 [Warning] IP address '103.37.45.166' could not be resolved: Name or service not known
2017-03-19 16:33:18 12178 [Warning] IP address '61.160.194.88' could not be resolved: Name or service not known
2017-03-19 18:09:59 12178 [Warning] IP address '139.196.18.68' could not be resolved: Name or service not known
2017-03-19 18:10:44 12178 [Warning] IP address '117.41.229.53' could not be resolved: Name or service not known
2017-03-19 21:00:33 12178 [Warning] IP address '182.18.72.81' could not be resolved: Name or service not known
2017-03-19 21:31:10 12178 [Warning] IP address '123.249.45.172' could not be resolved: Name or service not known
2017-03-19 21:40:05 12178 [Warning] IP address '123.249.27.92' could not be resolved: Temporary failure in name resolution
2017-03-19 21:52:52 12178 [Warning] Host name 'hostby.chnet.se' could not be resolved: Name or service not known
2017-03-20 00:33:24 12178 [Warning] IP address '122.114.224.10' could not be resolved: Temporary failure in name resolution
2017-03-20 00:41:00 12178 [Warning] IP address '106.111.128.184' could not be resolved: Name or service not known
2017-03-20 02:44:32 12178 [Warning] IP address '49.4.142.177' could not be resolved: Name or service not known

Impossible to say. Could be an SQL injection, a compromised root account or just the fact that your MySQL server appears to be globally open (this is a very bad idea for this very reason - why is the server configured this way?) or countless other things. — Sven, Mar 20 '17 at 09:12
I was about to finish some migrations, reason I have opened remote access to my DB but just for a few hours. On the PHP side I use Laravel 5.4, which I guess is quite secure. Hmmm, I am speechless. — Edmond Tamas, Mar 20 '17 at 09:16

score 4 · Accepted Answer · answered Mar 20 '17 at 09:17

4

Security group rules show that you opened 3306 for everyone and it is dangerous.

Don't allow traffic to 3306 from everywhere.
Restrict 3306 access to known ip,s and better option is to restrict it access through VPN.
Add log monitoring tools to inform you in case of any malicious traffic.
If you have small setup then use Monit to monitor logs.
Strict user policies in MySQL.

There are tons of other things which can be used to secure MySQL. But it is good to start with these.

answered Mar 20 '17 at 09:17

xs2rashid

184
5

Thanks, I will start with the above suggestions. By the way, while checking the users on my MySql server, there are 3 root accounts created. Is this normal? Forgive me for my noobness. – Edmond Tamas Mar 20 '17 at 09:21
1

It is normal if you know why you have these accounts.Those three root accounts might have different network access rules. – xs2rashid Mar 20 '17 at 09:30

score 3 · Answer 2 · answered Mar 20 '17 at 09:53

The first thing you should do to prevent this happening again is replacing every single instance of MySQL you have.

While I would recommend you not consider paying for your data, if you must, then keep one instance around that will allow you to get that data back, then dump it ASAP, check and recheck that dump, and then import it into a clean install.

If you can afford to not retrieve your data, burn everything to the ground and start again.

@xs2rashid's suggestions are definitely good ones. Certainly consider not allowing any access you don't need - i.e. whitelist everything, rather than using a blacklist.

I would also suggest you make a point of ensuring you run mysql_secure_installation on your nodes, and use a password manager (e.g. KeePass) to generate strong passwords. Better still might be to use a CA/PKI - cfssl makes it easy to generate the certs you would need for that.

You might want to use fail2ban to help block anything suspicious as well (How do I setup monitoring of MySQL with Fail2ban?), as a guard against mistakes in your network protections.

You also expose SSH to the world, which means you almost certainly want to ensure you are using public-key authentication, disallowing root logins, and restricting access/login to SSH as much as possible (e.g. limit network access, and limit which users/groups are allowed to login).

I would tend to think you might gain from reading through the appropriate CIS benchmarks for your distro, and consider applying at least some of their recommendations.

I am really thankful for the above suggestions, I will take in consideration all of it. One question, when you say: 'replacing every single instance of MySQL' do you refer to make a clean MySql install or to drop the EC2 instance itself? Respectively: 'burn everything to the ground and start again' - meaning to launch a clean EC2? Thanks! — Edmond Tamas, Mar 20 '17 at 13:04
Yes, I would, just to be safe, start from fresh instances.All else aside, it means you can focus on recovering, with (at least some) expectation your machines are clean. Otherwise, you would need to spend time to recover, and also need to work out how best to detect further signs of compromise, etc. Although you only mention the damage to your DBs, the actual compromise vector isn't clear, and so I would have concerns about how exactly all this happened. It is almost certainly a better use of your time to focus on hardening your base config, and restoring what you can. — iwaseatenbyagrue, Mar 20 '17 at 13:11

MySQL hacked on AWS AMI: 'Pay to get data back' - how could this be possible and how to avoid it next time?

2 Answers2