Securing administration network

Question

We would like to secure our administration network from advanced persistant threats and also from rogue administrators. I have already read the best practices from SANS Institute, NSA or other guides.

We have separate physical networks for administration and production and admin workstation don't have internet connection, usb ports, etc... We also have encrypted databases.

Actually, this is not preventing some rogue admin to backdoor the database server to get the key (as decryption is done on database server). We could not implement application side encryption as it's proprietary software.

I want to implement zero-access administration for sensible production servers. I would like to do so by having the servers themselves pulling administration scripts and execute them only if they are signed (pgp) by at least 2 admins (with an agent and something like a git repository). (like code review but enhanced with some crypto). With that, I think that we will obtain an administration network without any single point of failure. Someone will need at least 2 administrators accounts involved to execute code on sensible servers.

I would like to know if you have some advices for protecting our administration network from rogue administrators and APT ?

I would also like to know your opinion on this kind of administration (all scripted) and if you have already implemented something like that (or have some examples)? (ie: How NSA is doing after Snowden ?)

Is there any other good practice to protect from leakage by administrators (with the constraint of not being able to encrypt the data application side) ?

Answer 1

As of late, there has been a best practice in the DevOps Community:

If it's not automated, it's not secure.

Automation provides benefits for security (e.g. limited, audited access via configuration management, e.g. Puppet, Chef, SaltStack), but also for availability and ease of support. As far as availability goes, it should be fast to roll back changes as well as QA them to detect issues. For ease of support, it no longer relies on one-off ssh commands etc run by system administrators, instead a fix is automated and can be easily reproduced.

In these environments, remote access via SSH etc to individual servers is generally disabled. Instead, one must use the configuration management system to push any changes -- or simply deploy a new server, depending how the DevOps stack is designed.

SELinux can also be leveraged to make it so that system administrators do not have access to actual data, only sysadmin functions. However, this is very cumbersome to maintain and is not commonly practiced, to the best of my knowledge, outside NSA and other high-security state-backed entities.

To detect and potentially prevent data loss/leaks, a DLP solution can be invoked. This can be installed on servers (via configuration management) and will be triggered if a sysadmin attempts to download data, etc. These solutions are often expensive, but offer a last-line of defense against rogue staff.

Corporate policy can also address remediation efforts -- if it is made clear to the sysadmins what activities might violate corporate policy, and sanctions are made clear in the policy; this may have a strong deterrent effect against potential leakers.

In the end however, a sufficiently clever sysadmin can likely circumvent most of these protections eventually. To cite the old aphorism, "Locks keep out honest people." You can add as many locks as you like, if you have staff dedicated to nefarious activities it will be difficult to stop them.