Best way to go about changing a root password on 3000+ Solaris, AIX, and Linux servers?

Question

Long story short: Big old corporation, lots of UNIX/Linux servers.

I inherited responsibility for a bunch of scripts that that left a few years ago. One of them was a script that would be run every $X amount of months to globally update the root password across all of our servers.

The script is a mess of Shell Script and Expect, and it works on the SSH trust that is set up between all of our servers and a central command-and-control server.

The problem is, the script is a giant mess. The Expect commands are trying to account for every possible version of "passwd" that exists on any UNIX/Linux box out there - and they vary quite a bit.

As we're expanding and upgrading a lot of our infrastructure, the script is getting really unmanageable.

My question is: Is there a better way to do this? Assuming there's an already established SSH trust, what's the best way to change the root password on 3000+ servers at the same time?

Answer 1

Use Puppet.

Puppet is very flexible, easy to maintain and uses SSL. Might sound a bit overkill and you'll have to put some extra effort to build Puppet system up.

But. Most probably this is not the last mass-update you'll be doing to these machines. Puppet will and does save you a lot of time when actual whatever mass-update procedure begins and scripts are very readable/reusable.

At least this worked for me few years back and still I'm able to re-use some of those Puppet recipes (aka scripts). I have also used it in a bit smaller environments just make sure that every machine is actually having a known state.

I have proven it many times (at many companies) that all customised deployment scripts become pain in the butt after a while or when that next guy steps in. And as long as you carry a cellphone, old scripts will and do haunt you.

If you think that this actually sounds good, here's a great Puppet tutorial with virtual environment included to get you started.

Answer 2

I've used the Perl module Authen::PAM on Solaris with great success. Here's a sample script:

#!/usr/bin/perl

use Authen::PAM;

my $username = 'root';
my $password = '1234567';

die qq{Error: Unknown user\n} unless getpwnam($username);

die qq{Error: You must run this as root.\n} unless ($> == 0);

my $pamh;

sub my_conv_func
{
    my @res;
    while ( @_ )
    {
        my $code = shift;
        my $msg = shift;
        my $ans = "";

        if ($code == PAM_PROMPT_ECHO_OFF() )
        {
            if (($msg =~ /^New Password:/i) or ($msg =~ /^Re-enter new Password:/i))
            {
                $ans = $password;
            }
            else
            {
                die qq{Unknown message: $msg\n};
            }
        }
        else
        {
            print qq{$msg\n};
        }

        push @res, (PAM_SUCCESS(), $ans);
    }
    push @res, PAM_SUCCESS();

    return @res;
}

ref($pamh = new Authen::PAM("passwd", $username, \&my_conv_func)) || die "Error code $pamh during PAM init!";

my $res = $pamh->pam_chauthtok;

print $pamh->pam_strerror($res),"\n" unless $res == PAM_SUCCESS();

exit 0;

Answer 3

If you can write Perl, the module Net::OpenSSH::Parallel allows to write scripts that perform actions in parallel on remote hosts via SSH quite easyly.

It contains a sample script for changing passwords that you can use as a base. As it seems that you have an heterogeneous environment you would like to group the hosts by type and use a different dialogue handling sub for every one.

Answer 4

I don't know about "best", and whether it's possible for all of the non-Linux *nix machines in your mix, but have you looked at puppet or cfengine for this kind of activity?

There are also commercial (highly expensive) tools for identity management out there, two I've seen/used in the past are Oracle Identity Manager and the Novel equivalent.

Answer 5

After continuing to research this, I've learned a few things...

First and foremost, this is a really annoying task to automate, especially across many different environments. The most correct answer to this question is probably @tomi's: use Puppet.

Eventually I hope to get Puppet to manage out infrastructure, but deploying to the entire enterprise's UNIX servers for a root password change right now isn't that feasible of an option.

After reading many manpages and lots of Google-fu, I managed to come up with a script that loops through a list of target servers, opens an SSH connection, and runs one of the following:

# Solaris
# Generate new pass via crypt(newpass,salt) and insert it into /etc/shadow

# AIX
$ echo "root:newpass" | chpasswd -f NOCHECK

# Linux
$ echo "newpass" | passwd root --stdin

# IBM VIO (Virtual I/O)
$ echo "echo \"padmin:newpass\" | chpasswd -f NOCHECK" | oem_setup_env

# IBM HMCs (Hardware Management Consoles)
$ chhmcusr -u hscroot -t passwd -v "newpass"

It does quite a bit more than running just those commands, but those above are what work the magic.

I couldn't find any simple way to non-iteratively change the password on Solaris - so we resorted to modifying /etc/shadow on the fly.

Answer 6

Recently I have done this Using Bash Script..

#!/usr/bin/env bash

# Change Password of Remote Server Using SSH

#--------------------------------------------
# Define User which you want to
# Change Password in remote server
#--------------------------------------------
Uname="root"
# Create Password in Encrpyted Form Using below command,
# and store in this script
# perl -e'print crypt("YourPassword", "salt")' ; echo -e
# then copy and past in following varible,
# password should be single qouted*

Password='safv8d8ESMmWk'

Update_Pass() {
  ssh $ruser@$Server_ip  -p $port "echo ${Uname}:${Password} | chpasswd -e"
}

Show_Help(){
cat <<_EOF
Usage $0        [OPTION]..
Mandatory arguments to long options are mandatory for short options too.
  -i, --ip     <IP_ADDR_OF_SREVER> IP Address Of Remote Server
  -u, --user   <Username>          Username Of Remote Server    <Default User is root>
  -p, --port   <port>              Port Of Remote Server        <Default is 22>

Note:- For Security Reason Do Not Provide Password to the script, because
       it will get save in history, so do not provide it,
       script will prompt for password

Report $0 bugs to loginrahul90@gmail.com
_EOF
}



Main() {

        case $1 in
           -i|--ip) Server_ip=$2;
                    ruser="$4"; [[ -z $ruser ]] && ruser=root
                    port="$6";  [[ -z $port  ]]  && port=22
                    Update_Pass ;;
                *)  Show_Help ;;
        esac
}

Main $*

Answer 7

This is my solution so far. still need to see if it works on multiple systems

#!/usr/bin/env bash

ChangePassword()
{
    echo "changing password for server $ServerIp"
    ssh root@$ServerIp "echo root:${NewPassword} | chpasswd" < /dev/null
}

CreatePassword()
{
    while true;
    do
        echo "Please enter the new password :"
        read -s NewPassword <&0
        echo "Confirm the password :"
        read -s ConfirmPassword <&0 
        # /proc/${PPID}/fd/0

        if [ "$NewPassword" == "$ConfirmPassword" ]
        then
            break
        else
            echo "Passwords do not match"
            echo "Try again..."
        fi
    done
    ChangePassword
    echo "end of createpassword"
}

SetUpPasswordlessSSH()
{   
    echo "enter the old password from server $ServerIp when asked"
    ssh root@$ServerIp mkdir -p .ssh
    cat .ssh/id_rsa.pub | ssh root@$ServerIp 'cat >> .ssh/authorized_keys'

    echo "Passwordless SSH is now available"
    echo "Now you can change the password"
    CreatePassword
}

NoSSH()
{
    echo "Passwordless SSH for this server with ip $ServerIp is not yet set up."
    read -p "Do you want to set it up now? " -n 1 -r <&0
    echo "" 
    if [[ ! $REPLY =~ ^[Yy]$ ]]
    then
        break
    else
        SetUpPasswordlessSSH
    fi
}

AcceptHostKey()
{
    read -p "Do you trust the server? " -n 1 -r <&1 
    echo ""
    if [[ ! $REPLY =~ ^[Yy]$ ]]
    then
        break
    else
        SetUpPasswordlessSSH
    fi
}

Main()
{
    while read -r ServerIp <&9
    do
        echo  "Server $ServerIp ..."
        status=$(ssh -o BatchMode=yes -o ConnectTimeout=5 $ServerIp echo ok 2>&1)
        if [[ $status == ok ]]
        then
            echo "creating password"
            CreatePassword
            echo "password changed"
        elif [[ $status == "Permission denied"* ]]
        then
            NoSSH
        elif [[ $status == "Host key verification failed"* ]]
        then
            echo "Error: $status"
            AcceptHostKey
        else
            echo "ERROR OCCURED FOR SERVER WITH IP: $ServerIp"
            echo "Error: $status"
        fi
    done 9< servers.txt
    history -cw
    clear
}

Main $*

Answer 8

You could use pdsh to execute your command on multiple hosts at the same time.

Answer 9

In addition to puppet: SaltStack Another approach - automate execution using SSH libs either sequentially or in parallel using Fabric http://docs.fabfile.org/en/1.6/, Capistrano or similar that do not require much time/effort to deploy.

Answer 10

Two options:

Use puppet

Use run deck. Run deck is a server which allows you to execute commands on hundreds of machines simultaneously. You can group machines into groups, in order to execute commands on just a subset of machines.

http://rundeck.org

Answer 11

I think the format of the /etc/shadow file is pretty standard across linux distros. Can you just write a simple awk script to update the password.

cat /etc/shadow| awk -v pass='NEWPASSHASH' -v now=`date '+%s'` 'BEGIN{ OFS=FS=":"} /^root/ {$2=pass; $3=now} {print}' > /tmp/shadow && mv /tmp/shadow /etc/shadow

I would make sure to test it so you don't lock your self out ;)