I don't want to set all of the root user passwords all the same do I?
no, you dont :) and you want to disable root-ssh-logins as well and use an admin-login that has sudo-rights for often.used tasks via ssh and a password-protected keyfile.
with a rack of 40 servers you should check a management-tools like puppet that creates all necessary users, logins, ssh-key-management, configs etc.
If not how do you manage and keep up with all of the passwords?
use strong passwords and stuff like keepass or the mentioned lastpass. dont use online-password-safes.
when you have ssh+keys enabled, you'll need passwords not that often, except for your ssh-keyfile and certain root-tasks, but the you have your pw-manager open and just c&p the passwords intro the console-app. if you use linux inserting a password into a terminal is just 3 clicks; quite comfortable for me.
Is an LDAP server a viable option to use with root logins?
i wouldnt allow root-logins except from a local console. ldap is nice for accounts, but i still would have 1 local account for login if ldap fails or is otherwise not available. and setup via pam'n'stuff is 8was at least 3 years ago) big PITA