All my servers have just been hacked and now I'm ready to start locking down my web servers. They are currently running Ubuntu 12.04 with
I have no idea how they were hacked. What are the recommendations to reduce the chance of losing my servers again?
When you have so many technologies, it's hard to say specific issues that you should look out for with each tool. In general, for each of those, do a google search of lock down <technology> ubuntu, and read through the first five to ten results. Taking these actions for each of your services should dramatically reduce how vulnerable your system is.
Below are some blanket things that you can think about/do. Since your question is pretty broad, I think this will be plenty of work already. Keeping tabs on ServerFault and http://security.stackexchange.com/ will also increase your awareness.
Be totally aware of everything that is exposed to the outside world. I noticed that you didn't mention ssh, which might suggest that there are other services that you are running but didn't mention. Run netstat -lntp and see if there are any services that you did not realize you are running, and remove/kill them if they are not necessary.
Learn iptables so that you can control network traffic in/out of your machines. For instance, you can whitelist only ports 80 and 443 if you expect to only have web traffic. You get more control than if you just killed off every service running using ports, because you may need to use ports and your loopback interface to have services on the same machine connect to each other.
For instance, php may need to need to be in communication with redis, and you may do this over the loopback interface. You could could allow incoming connections to redis only over the loopback interface without allowing outside machines connect to redis' port.
This pertains to defense in depth. Set up your services such that exploitation of one service has a low chance of affecting other services or the rest of your machine. One action that you could take is to create a different user account to run every service. If an intruder breaks that service such that he can acquire a shell, he may be only able to acquire the privileges of one not very privileged user. Those lockdown guides for each service will discuss how to do this.
Every interaction with all of the services that you're running on your machine should have logging enabled. When anything happens to you, you need to go through your logs and look for anything out of the ordinary. You can react and patch up any issues that arise.
For instance, you notice that someone has broken in, and files have been modified that shouldn't have been modified. Going through your auth.log, you notice that there were several tens of thousands of attempts to log into your machine through ssh, eventually concluding with a successful login. That's when you realize that your password is only five characters long and you should probably change it.
Similar what logging will help you with, but more automated. Do some research on IDS and IPS softwares, like http://www.snort.org/ or http://www.tripwire.com/. It's nice when you have tools that scream to you that you are under attack.
Hope these general tips help, and be aware that there are many other things that you can do to protect yourself (that you will become aware of over time if you visit serverfault and itsecurity often enough :) ).
