Given SSH access, what should I do to protect a server and monitor (dangerous) user actions?

Question

Context:

• Small business, mostly a software house for web applications but also some desktop software.
• Many external collaborators, so a variety of outside users with access to the servers.
• Single physical server, with Linux and Xen as a virtualization solution - each VM has specific uses and controlled access. External users can access two of them
• The (virtual) servers provide a variety of services: LAMP stack, email, DNS, etc. This question pertains to my concerns with local user access
• Users can SSH into two of the VMs

Requirements

• Give users access to common development tools, both for web (PHP, Ruby on Rails, etc) and standalone applications (gcc, g++, etc); this includes not only compilers and the like but also editors. Right now, they have full shell access.
• Users must be able to use source control on the server: svn and git
• Some have access to MySQL databases

Question

What steps should I take in order to:

1. Provide a full shell to the users if possible, or equivalent solution that meets the requirements above in a safe manner
2. Automate monitoring of dangerous activities where possible (as in sudo/su notifications)
3. Minimize effects if a user gets his hands on a 0day privilege escalation exploit before someone patches it on the server - is this even possible?

Answer 1

If users must have SSH to your server, a useful tool to protect your server root is chroot - this will let you give them the apparent functionality of server root, without actually giving them the crown jewels.

Alternatively, as you use virtual machines anyway, why not provide them with virtual server instances?

Both of these will allow you to run monitoring tools (such as tripwire) in order to check for changes to files. Logging should be done outside the users' environment.

With entries in /etc/sudoers you can limit what activities users are allowed to carry out under su.

Answer 2

You can also restrict users to using bash 4 (removing other shells) and record all commands and send copies to a separate log server.

Bash: History to Syslog (http://blog.rootshell.be/2009/02/28/bash-history-to-syslog/)

This can help you audit user activities and, more importantly, perform forensics when someone does something bad. I would also do the full legal disclosure thing - all activities are recorded and audited, violators will be prosecuted, etc.

Deploying OSSEC (Host-based Intrusion Detection System) would help as well; it has many built-in rules to detect and alert you to bad things. OSSEC can alert you to all sudo/su activity.

Answer 3

Do not forget about tunneling, port forwarding. By default AllowTcpForwarding is yes in sshd_config file and it allows users to forward arbitrary ports. It may cause much more trouble than you can guess. (This is not an ideal stackexchange style answer but I could not find a better answer, it's hard to summarize port forwarding with a few sentences) You should learn more about port forwarding before allowing access.

Port forwarding in a nutshell

SSH permits multiplexing data streams over a single SSH connection. One or more of these streams can be a port forward, such that traffic to a port on the attacker's client computer is sent over the SSH connection to a specified port on a specified host from the server. This means that if your shell server has access to, for instance, a MySQL server, then anyone who connects to the SSH box can access the MySQL server directly. They can then use their own client tools rather than being limited to the software you've supplied on the SSH box.