I've had an EC2 instance for about a year running a LEMP stack. This week I tried adding a server block to NGINX and weirdness happened. For reference I ssh in as ubuntu, and have never to my knowledge logged in as any other user.
First: After configuring the server block for the new site (site2.com), testing the conf and restarting NGINX, site2.com showed a file I know I made quite some time ago, but I couldn't find it anywhere. I grepped the heck out of the instance, no sign of the file, and it was definitely not in the site2 root directory or /var/www/html. During this time site1 was working fine.
Second: In the process of trying to figure this out, I stopped NGINX, confirmed site1 went down, but site2 was still pointing at the file I couldn't find. I tried cacheless refreshes, firefox -> forget site, tried chrome, etc, fairly certain it was not a cached thing. I listed the running services, but didn't see anything the would act as a web server.
Third: While trying to hunt down the issue (starting to get a little desperate over the ghost file) I looked at iptables, which I never altered (I use UFW), but it was suggested on a blog of forum or whatnaught. I flushed them, and shortly after that puTTY froze up and kicked me off. I tried aws instance -> stop and then start, and then the real fun began.
Fourth: After the instance was running again I tried to ssh in with puTTY to no avail, the system wouldn't accept the key file. I ended up having to remake the ppk file, and then was able to get back in, but everything was different. All the configuration, all the installed programs, all the relevant files are now missing like the ghost file site2 was serving, but in a horrible twist, the ghost file is now very happily sitting in /var/www/html as if it had never not been there. df -h shows the drive is holding 4.5G of data, but trying to find that data didn't get me anywhere. I'm also being told there is no "No space left on device" when I try an apt-get update.
Any suggestions at this point are welcome.
