How can I deploy rolling OS upgrades & reboots with Puppet or MCollective?

Question

I'm looking for the best way to perform regular rolling upgrades for my infrastructure.

Typically, this involves doing this on each host, one at a time:

sudo yum update -y && sudo reboot

But, I'm hitting limits of that being a scalable.

I want to only reboot one node at a time within each of my roles, so that, say, I don't take down all of my load balancers, or DB cluster members, at the same time.

Ideally, I'd wanna do something like:

for role in $(< roles_list.txt) ; do
    mco package update_all_and_reboot \
        --batch 1 --batch-sleep 90 \
        -C $role -F environment=test
done

But, that doesn't quite seem to exist. I'm not sure if using the "shell" agent is the best approach, either?

mco shell run 'yum update -y && reboot' \
    --batch 1 --batch-sleep 90

---

Am I just looking at the wrong sort of tool for this job, though? Is there something better for managing these sort of rolling reboots, but that I can somehow link up with my Puppet-assigned roles, so that I can be comfortable that I'm not taking down anything important all at once, but that I can still do some parallel updates & reboots?

Answer 1

Configuration

Deploy

cd /usr/share/ruby/vendor_ruby/mcollective/application
wget https://raw.githubusercontent.com/arnobroekhof/mcollective-plugin-power/master/application/power.rb

and

cd /usr/libexec/mcollective/mcollective/agent
wget https://raw.githubusercontent.com/arnobroekhof/mcollective-plugin-power/master/agent/power.ddl
wget https://raw.githubusercontent.com/arnobroekhof/mcollective-plugin-power/master/agent/power.rb

on both hosts, i.e. test-server1 and test-server2.

Services

Restart mcollective on both services:

[vagrant@test-server1 ~]# sudo service mcollective restart

and

[vagrant@test-server2 ~]# sudo service mcollective restart

Commands

Run the following commands on the mcollective server node:

The host test-server2 is listening:

[vagrant@test-server1 ~]$ mco ping
test-server2                             time=25.32 ms
test-server1                             time=62.51 ms


---- ping statistics ----
2 replies max: 62.51 min: 25.32 avg: 43.91

Reboot the test-server2:

[vagrant@test-server1 ~]$ mco power reboot -I test-server2

 * [ ============================================================> ] 1 / 1

test-server2                             Reboot initiated

Finished processing 1 / 1 hosts in 123.94 ms

The test-server2 is rebooting:

[vagrant@test-server1 ~]$ mco ping
test-server1                             time=13.87 ms


---- ping statistics ----
1 replies max: 13.87 min: 13.87 avg: 13.87

and it has been rebooted:

[vagrant@test-server1 ~]$ mco ping
test-server1                             time=22.88 ms
test-server2                             time=54.27 ms


---- ping statistics ----
2 replies max: 54.27 min: 22.88 avg: 38.57

Note that it is possible to shutdown a host as well:

[vagrant@test-server1 ~]$ mco power shutdown -I test-server2

 * [ ============================================================> ] 1 / 1

test-server2                             Shutdown initiated

Finished processing 1 / 1 hosts in 213.18 ms

---

Original code

/usr/libexec/mcollective/mcollective/agent/power.rb

module MCollective
  module Agent
    class Power<RPC::Agent

      action "shutdown" do
  out = ""
  run("/sbin/shutdown -h now", :stdout => out, :chomp => true )
  reply[:output] = "Shutdown initiated"
      end

      action "reboot" do
  out = ""
  run("/sbin/shutdown -r now", :stdout => out, :chomp => true )
  reply[:output] = "Reboot initiated"
      end

    end
  end
end

# vi:tabstop=2:expandtab:ai:filetype=ruby

/usr/libexec/mcollective/mcollective/agent/power.ddl

metadata    :name        => "power",
            :description => "An agent that can shutdown or reboot them system",
            :author      => "A.Broekhof",
            :license     => "Apache 2",
            :version     => "2.1",
            :url         => "http://github.com/arnobroekhof/mcollective-plugins/wiki",
            :timeout     => 5

action "reboot", :description => "Reboots the system" do
    display :always

    output :output,
           :description => "Reboot the system",
           :display_as => "Power"
end

action "shutdown", :description => "Shutdown the system" do
    display :always

    output :output,
           :description => "Shutdown the system",
           :display_as  => "Power"
end

/usr/share/ruby/vendor_ruby/mcollective/application/power.rb

class MCollective::Application::Power<MCollective::Application
  description "Linux Power broker"
  usage "power [reboot|shutdown]"

  def post_option_parser(configuration)
    if ARGV.size == 1
      configuration[:command] = ARGV.shift
    end
  end

  def validate_configuration(configuration)
    raise "Command should be one of reboot or shutdown" unless configuration[:command] =~ /^shutdown|reboot$/

  end

  def main
    mc = rpcclient("power")

    mc.discover :verbose => true
    mc.send(configuration[:command]).each do |node|
      case configuration[:command]
      when "reboot"
        printf("%-40s %s\n", node[:sender], node[:data][:output])
      when "shutdown"
        printf("%-40s %s\n", node[:sender], node[:data][:output])
      end 
    end

    printrpcstats

    mc.disconnect

  end

end

# vi:tabstop=2:expandtab:ai

---

Modified code

/usr/libexec/mcollective/mcollective/agent/power.ddl

metadata    :name        => "power",
            :description => "An agent that can shutdown or reboot them system",
            :author      => "A.Broekhof",
            :license     => "Apache 2",
            :version     => "2.1",
            :url         => "http://github.com/arnobroekhof/mcollective-plugins/wiki",
            :timeout     => 5

action "update-and-reboot", :description => "Reboots the system" do
    display :always

    output :output,
           :description => "Reboot the system",
           :display_as => "Power"
end

/usr/libexec/mcollective/mcollective/agent/power.rb

module MCollective
  module Agent
    class Power<RPC::Agent    
      action "update-and-reboot" do
        out = ""
        run("yum update -y && /sbin/shutdown -r now", :stdout => out, :chomp => true )
        reply[:output] = "Reboot initiated"
      end
    end
  end
end

# vi:tabstop=2:expandtab:ai:filetype=ruby

Command

[vagrant@test-server1 ~]$ mco power update-and-reboot -I test-server2

 * [ ============================================================> ] 1 / 1


Finished processing 1 / 1 hosts in 1001.22 ms