How can two-man control be implemented efficiently?

Question

Two-man control can be a strong defence, but the overhead is a concern, as discussed previously.

How can two-man control be implemented with minimal overhead?

I would be most interested in a concrete example: an online banking platform. The environment includes a load balancer, web servers, database servers, and a link to the core banking platform (which is out of scope).

If the two men collude then they will be able to perform malicious actions on the system. That is an inherent limitation with two-man control, and we should not attempt to solve it.

To avoid this question being too broad, lets only consider sys-admins - not developers, support agents, etc.

Answer 1

The key is to only implement two-man control for operations that really need it.

Routine tasks

Routine tasks, like deploying an application release, or applying security patches can be scripted. One sys-admin can create the script, but it cannot be run until a second sys-admin has reviewed and approved it. Common tasks like restarting the application server, or deploying the latest application release, can have a pre-approved script that is used repeatedly. Non-standard tasks will need a bespoke script.

To do this, there's a need for some kind of workflow management system. You may be able to use a ticketing system like Trac, build a bespoke system on something like JIRA, and configuration management tools like Puppet may have features that would help. The two sys-admins do NOT need to be physically local; they can work remotely.

The systems should be designed so that as much routine maintenance is automated as possible - scheduled log rotation, etc.

Diagnostics

Sys-admins can have read-only access to logs without two-man control, provided the logs mask sensitive information. They can also have a low-privilege account to view status pages, and run commands like top or ps. This allows a single sys-admin to diagnose issues - and they can work remotely. They may be able to do remediation by submitting a script as a routine task. However, in some cases that won't be possible, so there is a need for emergency access.

Emergency access

Sometimes there is a serious issue and you just need to SSH in as root. To maintain two-man control you need two people at this point. In practice, they need to be physically next to each other. A simple way to implement access control is to have an account for each pairing, where each sys-admin knows half the password. You could create a smarter system with a custom PAM module, perhaps even requiring a separate smart card from both sys-admins.

Once logged in, it is down to the sys-admins to maintain two-man control. They must inspect each others commands before issuing, and lock the terminal if one of them needs to step out.

The aim should be that emergency access is rarely needed.

Development environments

It is only necessary to use two-man control for the live environment. A lone sys-admin can have full access to a development environment - provided that live data is not present. They can try out a task in the development environment, perform some testing, iron out the bugs. Only when they have got a working script do they submit it to another sys-admin for approval.

Conclusion

If all this is done, I believe you can implement strong two-man control with about 25% overhead. Making it a success relies on various things (like automated maintenance) that are good practice anyway. If a system is so sensitive as to need two-man control, it is a good idea to get all the basics right as well!

I have never seen this implemented in a commercial or government environment. The closest I have seen was an online banking platform where most sys-admins did not routinely have root access. To performs a task, this would first be planned in detail, then approved on the change control system. To implement the change the sys-admin then got root access for a limited period. Also, there were a small number of senior sys-admins who had permanent root access, in case of emergencies. This arrangement is much better than I typically see - although it falls short of full two-man control.

Answer 2

How can two-man control be implemented with minimal overhead?

TL;DR This can be accomplished using a special sudo command that requires a second person to approve each command, combined with a nice whitelist to make safer commands more convenient. It is important that each command is approved by the two-man rule or else the protection is easily defeated.

---

Two-man rule implementation options:

Unconstrained root access

Enable root access temporarily

A small root script could require a second person (User A) to authorize before sudo becomes available for the acting sysadmin (User B)

1. User A: enable-sudo user-b
   (auto-adds an entry to /etc/sudoers with either a time-limit or scheduled removal via at job)

2. User B: sudo do stuff will start working at this point

Unfortunately, a back-door could easily be added by User B once User A has authorized the access. This could be as simple as re-adding user-b to /etc/sudoers or starting a separate service.

It is not possible to robustly enforce security rules once someone has gained root (as it is "unconstrained"), so you should covertly monitor the most likely bypass possibilities and alert another (more trusted) sysadmin if something begins looks odd after a two-man enabling of sudo.

• unexpected at or cron job
• unexpected background process still running
• unexpected startup script

And don't tell them exactly what safeguards you are using. This will help against the obvious back-doors, especially if user-b is not familiar with the protections in place, but will not thwart a user-b who is determined to cause trouble.

Command-basis two-man rule

Create your own version of sudo, which alerts the second person of any command suggested by the first person, with the ability to approve or deny.

I won't go into the implementation details here, but it would be fairly straight-forward to implement.

In this case, a back-door would have to be hidden within one of the provided files, for example a software update to be installed.

File editing with vim

Your custom version of sudo should not allow launch of direct file editing commands. If someone types oursudo vi, then oursudo will have to launch a special script for file editing.

1. Copy the file to a /tmp location (randomly generated filename to prevent symlink attacks)
2. Launch vim with reduced privileges to a non root account (so user-b cannot use the "Save As" function within vim to bypass this wrapper)
3. When saving is completed, present a diff of the file changes to user-a.
4. Once approved, copy the approved file from /tmp to the target location.

Physical barriers

Requiring physical access to gain root access could help bring visibility to what user-b is doing. Depending on the office environment, the team may have a clue to what user-b is up to, and/or user-b may be less comfortable causing trouble in that environment. It depends on how often such access is necessary though.

Shortcomings: root = unconstrained

As you can gather from the ideas I've brought up, there does not seem to be a sure-fire to protect against back-door installations. root access is designed not to allow this.

Whitelist commands approach (constrain the root access)

The best thing is to create a whitelist of available commands. (which sudo has the ability to offer) This does create a significant overhead at first, but eventually may become more practical as you learn the command that are commonly used.

Deploying application updates

Your applications should not run on root access. Those that do must go through a separated Code Review system and then updates from the approve VCS commit can be included.

Patching [packaged] software

This is cannot be constrained, so I would suggest using command-basis two-man rule outlined above.

Troubleshooting

Safe tools can be permitted via sudo, and then use command-basis two-man rule to roll the fix.

Answer 3

There are a lot of good answers here. In the end it is a personal decision and the organizational process is also a very important part of the whole process.

I would like to point to another, better managable solution. Disclaimer: This is a solution I am developing.

privacyIDEA is a two factor management system. Or to put it more broadly it is an authentication management system. One possibiliy is, you can also assign an abstract authentication token "4-eyes-token" to a user account. This way you can setup that a certain user account or certain commands (sudo) need authentication by two or more persons. These users could authenticate with passwords or with one time password hardware tokens like yubikeys.

The nice side effect is, that you get a digitially signed audit log within privacyIDEA, so that you can know, which two persons acted within your two-man-rule.

Answer 4

There's a cryptographic technique that is very relevant here, called secret sharing:

Secret sharing (also called secret splitting) refers to methods for distributing a secret amongst a group of participants, each of whom is allocated a share of the secret. The secret can be reconstructed only when a sufficient number, of possibly different types, of shares are combined together; individual shares are of no use on their own.

So one strategy that you could adopt is:

• All administrative actions require authentication with a cryptographic key.
• The key is split n ways across your n administrators, with a two-share threshold to recover the secret.
• Every administrator must keep their share secret from all the others.

I don't know of many implementations of secret sharing out there. See this question for references. I would add this:

• HashiCorp's Vault is a client/server distributed system for storing and retrieving secrets (e.g., passwords that must be accessed by applicaqtions). The contents of the Vault database are encrypted with a key that's not stored anywhere, but rather split into a set of secret shares with configurable number and threshold. To reconstitute the key they have a process they call unsealing where the owners of the requisite number of shares must each enter their share.

So it might be possible to cook something up with Vault to achieve this. The key ideas would be:

• Keep all the admin credentials inside a Vault instance that's only unsealed when they are needed, and sealed right away when they're not;
• Script all your tasks so that admins never see the actual credentials, only their Vault master key shares.

Answer 5

The only way two-man control can be implemented efficiently without any possibility to bypass it, is to physically secure the resource in question (computer, data media etc) so its inaccessible unless 2 persons, via biometrics, authenticates to the service.

I would NOT recommend a 2 man rule for login as root or similiar for a computer, instead, I would rather recommend restricting so that type of access is only available on-site on a physical terminal, and then that physical terminal is simply tucked into a locked room (server room or whatever) requiring 2 man to authenticate to unlock.

The reason, is that otherwise, either one man can maliciousy plant a backdoor to allow single man control, or the two mans can collaborate to reduce system security to single man control.

By physically restricting to two-man control, its not possible to bypass the system, even if the two mans collaborate to give access to the single individual because the other man thinks "he is trusted" or something like that.

(of course, two mans could collaborate on executing a malicious command or whatever, but the important factor here is that they cannot collaborate on giving permanent single man access)