We have two sysadmins who know the passwords to all our systems. If the "unthinkable" happened (AKA "they both went under a bus") there is currently no way for the remaining team members to obtain administrator access to the systems.
What steps or procedures should we follow to protect against this, while still keeping the systems secure?
Have them write the critical passwords down, seal them in an envelope, and store them in a safe, safety deposit box, or some other location where the physical security is high. You want to make sure only someone like the CEO or some other trustworthy person could access this, and make sure they can only access it a way that will leave physical evidence.
Have a procedure in place where every time they make a significant password change (like the recovery password on a DC, like the password for the original administrator account in the forest's first domain (which should be in the Enterprise Admins group), or the like that they record the account and the password twice. And those account/password combinations are immediately sealed up in an envelope and labeled appropriately. One envelope stays on-site in a secured location (like a safe). The other is secured wherever your offsite tapes/software is. In that case, you have recovery of all critical passwords.
Everyone else mentioned the necessity of physically writing the passwords down and storing them in a safe place. I will mention something equally important: Make sure the passwords are legible!!! Nothing is worse than looking at a disaster recovery password log and not being able to tell the difference between zeros, the letter O, ones, lower case 'l's and etc because your now deceased/disenfranchised/estranged SysAdmin has the handwriting of a tenured medical professor. Even "handsome" handwriting can carry ambiguities in various symbols. If the passwords are sufficiently complex, those ambiguities can lead to very long nights of password guessing OR give you a reason to learn how to set up Cain and Abel to brute force the system with the password on it using regex terms.
For example, I would write arrows pointing to the username and say "For all you Windows admins -- This username is actually case sensitive!!" or "This is the 12th letter of the English alphabet aka the letter "l" pronounced "ell"". Yes, I'm that paranoid.
Yes, I have OCD.
No I'm not on my meds. =(
In addition, if you are paranoid about security - split up the password. Put control of say a third of the password to the CEO and a manager, another third to the CTO and a different manager, and the last third to the CIO and a manager. That way, at least three people must agree before your precious password can be reconstituted and no one person has the ability to hose you (should they be fired or something.)
I store all password in PasswordSafe, with the program and database stored on a server with access granted to only those senior managers who should have access, as per company dictates. The contents of the database are also exported into a document, which is then printed using a font that makes it easy to distinguish those ambiguous characters and the printed copy is stored off-site in a secure location. Never rely on handwriting for this kind of thing.
In addition to having some sort of secured password list, document the process for gaining access to the system when no password is known...physical access, forcing a password change, "maintenance mode" etc.
Whatever the method might be for a given system, it's worth having that be a known process with understanding all around of what gaining that access takes (any downtime? impact of forcing a password change on a system?).
We type the passwords in and print them using a console or OCR font. After verifying, the document is closed without being saved. The passwords are then sealed in an envelope and stored in the CFO's safe of all places. Somewhat regularly, the CTO and CFO open the safe, verify it's still sealed, and have the admins make new envelopes.
Separately, the CTO maintains a backup admin accounts, with the passwords stored off-site with the backup tapes.
We haven't totally gotten the hang of what to do when a password is changed, because there are services that fail to start when an admin password is changed — but it happens rarely enough that it's not too big of a deal.
The envelope trick actually did come in pretty handy when the most senior sysadmin developed a terminal medical condition and passed before a full turnover of duties was complete. Unfortunately, with him knowledge of some of the cabling routes was irretrievably lost, and we have problems with this to the day — years and years later.
As a failsafe to be able to logon with the Domain Admin account in the event that your sysadmins disappear, create a special user account and delegate the ability to reset\change passwords to this account. Give this username and password to a trusted individual such as your boss, with instructions on how and when to use it. If your boss can change the domain admin password in the event of an emergency then you're half way to regaining control of your network.
You'll probably need to create a custom MMC console as well, but this can be stored in a secure network share.
Another option is to setup an account on the servers with root privileges and only allow access to this account via ssh key. Keep the private key only on a flash drive and securely stored.
If the unthinkable happens you can login via this account/key and change the passwords.
Passwords are not the problem. Any incoming replacement sysadmin will need to prove his/her skill by knowing how to recover a root enabled account on each server. Sure it requires access to the physical servers, but it is an unthinkable occurrence anyway. It happened with me that an IT professional wanted an admin login for a server that was faulty - I simply told him "you'll need to put a screen and keyboard onto the server and create a new account for yourself" - because I was not willing to give this person of unknown abilities a password.
I hate to think what mess an incompetent person can cause if given all the admin passwords.
It is more important to document the architecture of the system. Including URL's and IP addresses. And what regular procedures must be followed to maintain the network. This sort of documentation must be kept safely - but security is less important than for passwords.
