10

My EC2 instance was hacked recently. It doesn't really matter as I'm just starting my website and there was no sensitive information on my server yet, but I do plan for there to be in the future. I am going to terminate the compromised server and set up another one in order to secure my website.

My issue is that I want to make sure this never happens again, and the only way I can convince myself that my new server is secure is by knowing how the first server was compromised and making sure that the new server cannot be compromised by the same method. However I am having trouble identifying/understanding how the first server was hacked.

The stack used is Rails 4.2.4 + Postgresql 9.3 + Puma + Nginx 1.4.6. The server I'm running is Ubuntu 14.04.2 LTS on an AWS EC2 t2.micro instance. My AWS access keys have not been published anywhere, and I am the only one who should have SSH access to my server.

Now I'm no hacker, or even a good sysadmin for that matter (though I'm learning) so I have to keep an open mind about how the server was infiltrated, but since the only way I access my server is by SSH, I'm assuming that is how hackers got in too. I had erroneously left an inbound security group rule for the SSH port 22 to have a source of 0.0.0.0/0, which also makes me think it is likely this is how hackers got in.

I access my server by ssh using a private key with command similar to

ssh -vi "/home/me/.ssh/my_private_key.pem" ubuntu@my_aws_instance.com

I was notified of the compromise from an ec2-abuse@amazon.com email that my server had been making DDOS attacks. Although I couldn't verify this by looking around my logs, I did find some hidden php scripts in /var/tmp which looked as if they were designed to send out phishing emails which convinced me that the server had indeed been hacked.

There is no password set up for ssh which was confirmed by hydra when I tried to attack my own server. This implies to me that the attacker must have guessed my RSA private key... except I understand this is not realistically possible.

I don't know of any other possible ways that the server could have been compromised. Does anyone have any ideas? Are any of my assumptions or steps wrong?

Inbound Security Group Rules (at the time of being hacked):

HTTP               TCP    80      0.0.0.0/0
PostgreSQL         TCP    5432    0.0.0.0/0
SSH                TCP    22      0.0.0.0/0
Custom TCP Rule    TCP    9200    91.216.55.150/32

Thanks in advance for your help. I'm sure that I have left out a lot of useful information so if there is anything more you'd like to know just ask.

EDIT:

The website does contain some fairly standard Rails data input functionality. Mainly through html forms and there I also allow for image upload using a gem called carrierwave.

Community
  • 1
Dennis
  • 101
  • 1
  • 4
  • 4
    You are assuming the hacker came in via ssh, this may not be the method the hacker used. Did your PostgreSQL database have proper protections in place against the hackers? – Ferrybig Mar 09 '16 at 14:52
  • @Ferrybig Good point. I don't know how an attacker might change files on my server through postgresql, but I can access my postgresql database remotely using `psql` by password (which is 12 chars long)... does this make it vulnerable? – Dennis Mar 09 '16 at 14:59
  • 2
    As @Ferrybig states SSH may not have been the vector used. Could you update your question to tell us about the web site/applications which this server hosts? e.g. any cms type areas, file uploads etc, or is it all static (flat) content? – GreatSeaSpider Mar 09 '16 at 15:03
  • Also, you should see what ports your server is answering on. And you should'nt tell people the length of your PW. =) – Mindwin Mar 09 '16 at 15:07
  • @Mindwin haha ok point taken. `nmap` reports that ports 22 (SSH), 80 (HTTP) and 5432 (postgres) are open. Checking through my postgresql logs it does look like postgresql access was attacked too – Dennis Mar 09 '16 at 15:13
  • @Ferrybig my postgresql username is also non standard. How could an attacker gain the necessary access to perform a DDOS by hacking postgresql access? – Dennis Mar 09 '16 at 16:08
  • There are a couple of ways to get a shell from postgres. The ways I'm familiar with do require access to the superuser account for postgres, but it is possible. – h4ckNinja Mar 10 '16 at 01:26
  • @Micheal thanks for that, very useful to have that confirmed – Dennis Mar 10 '16 at 16:37

2 Answers2

15

The stack used is Rails 4.2.4 + Postgresql 9.3 + Puma + Nginx 1.4.6. The server I'm running is Ubuntu 14.04.2 LTS on an AWS EC2 t2.micro instance.


I access my server by ssh using a private key...

Unless you have edited the sshd config to allow password login as well, this is sufficient in ensuring a remote attacker cannot bruteforce your SSH login. If the computer you use as an SSH client is ever compromised however, you should consider the keys compromised as well.


How did hackers compromise my EC2 instance?

Difficult to say without more information. A list of running services, firewall rules, and a stack of log files would help.

My issue is that I want to make sure this never happens again...

Tough cookies. As long as you have a publicly facing web server, it will get attacked. The better approach to take is accepting that this can happen again in the future, and prepare for it. Some realistic things you can do are keep software fully patched, tune your firewall rules (do you really need to give access to the whole world?), implement an IDS/IPS on the AWS local network and route all traffic through it, and finally log all traffic and forward to a hardened logging server.

This is perhaps the most important step in understanding how you were hacked, and how to prevent it in the future. Grepping through log files is an option, but having a clean visual representation of your server's logs makes it much easier to detect anomalies, and to later search when a server is attacked. Having a separate, hardened log server is important because if your web server is compromised, the attacker can erase traces of the attack locally.

If you wanted to go one step further, you can automate the process of detecting anomalies, and define reactions based on that.

cremefraiche
  • 2,173
  • 12
  • 24
  • Does setting AWS SG Inbound Rule to single IP is enough or setting non-standard SSH port is essential too? For me, obfuscating SSH is redundant here, if only I can connect to instance. – Suncatcher May 09 '18 at 09:36
  • With AWS security policies in place there is no reason to obfuscate your SSH port. Any attempts will be dropped before reaching your instance. You should still be using SSH keys. – cremefraiche Jul 01 '20 at 11:35
1

The best solution is to implement a second EC2 instance with Linux. Then install OpenVPN to this instance. You can actually get OpenVPN Access Server with a free license (2 concurrent users) since you are the only one who SSH into the web server. Having rules that allow SSH from 0.0.0.0/0 is bad news. Also, your database ports should not be open to the internet (0.0.0.0/0).

How does this work? In order to SSH to the web server for example, you must first connect to the VPN network. Then SSH to the web server using its private IP. Your web server's security group rule will be something like this: "Allow SSH from 10.0.8.0/24" How to install OpenVPN Server on AWS? Follow this link: Install OpenVPN Access Server on AWS

Community
  • 1
Bruce Malaudzi
  • 111
  • 3
  • So, with this solution, openvpn would be open to 0.0.0.0/0. It would be interesting to add a bit to this answer, explaining why opening openvpn to 0.0.0.0/0 is more secure than opening openssh to 0.0.0.0/0 (with a hardened openssh configuration). – mti2935 Jul 02 '20 at 13:47