How to recover from master failure in kubernetes

Question

I have three nodes multi-master kubernetes(1.17.3) cluster(Stacked control plane and etcd nodes),

11.11.11.1 - master1
11.11.11.2 - master2
11.11.11.3 - master3

So before going to productions, I am trying possible failures and did below steps

Graceful Removal of Master Nodes

• Run kubectl drain 11.11.11.3 on master3
• kubeadm reset on master 3
• kubectl delete node 11.11.11.3 on master3

So by applying above steps all pods are running on master 1 and 2, it removes entries from kubeadm-conf configmap and also from etcd, infact i run above steps on master2 and still one master is up and running and i can run kubectl.

Non-Graceful Removal of Master Nodes

• I shutdown master3 but dont face any issue still two master accessible and i can run kubectl and do administrations.
• As soon as i shut master2 i have no access to kubectl and its saying apiserver is not accessible. How i can recover master1 in this situation?

This can happen that two nodes may have hardware issue in production, from my search it look like etcd issue but how i can access etcd and remove master2 and master3, i thought to do docker ps and docker exec <etcd container> but docker ps is not showing etcd container.

Answer 1

A topic near and dear to my heart.

The short version is:

1. create an etcd snapshot from the surviving etcd node
2. create a new, 3-node "disposable" etcd cluster, restoring from that snapshot
3. stop master1's etcd member
4. reset its state (rm -rf /var/lib/etcd, and delete the server and peer certs unless you have used the same CA for the disposable cluster -- something I highly recommend but may not be possible for a variety of reasons)
5. join it to the new, now healthy disposable etcd cluster
6. now you can either bring the other masters back online, joining them to that new cluster until you have a new quorum, and tear down the disposable cluster, or you can walk over the disposable etcd members, removing them from the cluster one at a time until only master1 remains in your "cluster of one"

I have had great success using etcdadm to automate all of those steps I just described, with the bad news being that you have to build the etcdadm binary yourself, because they don't -- as of this message -- attach built artifacts to their releases

In the future, you'll want to include etcdctl member remove $my_own_member_id from any orderly master teardown process, since if a member just disappears from an etcd cluster, that's damn near fatal to the cluster. There is an etcd issue speaking to the fact that etcd really is fragile, and you need a bigger team running it than you do kubernetes itself :-(

Answer 2

Keeping in view my findings and testing, To restore the full functionality of your k8s cluster, make a remaining etcd node a standalone etcd cluster. Follow below steps

As we lost two master nodes out of three and now etcd pod on remaining node is not able to complete quorum so this static pod is continuously failing and exiting and you cant do etcdctl member remove <node>

Stop the etcd pod:

• mv /etc/kubernetes/manifest/etcd.yaml /backup
• mv /var/lib/etcd/member /backup/

Temporarily force a new cluster on the etcd host:

• cp /backup/etcd.yaml etcd.yaml.bak
• edit /backup/etcd.yaml and change following values

etcd.yaml

Remove lost nodes(master2 and master3) from initial cluster
- --initial-cluster=master1=https://10.11.158.114:2380

Remove following line
- --initial-cluster-state=existing

Add Following line
- --force-new-cluster

Restart the etcd pod:

 cp /backup/etcd.yaml /etc/kubernetes/manifest/

Remove the --force-new-cluster command from /etc/kubernetes/manifest/etcd.yaml as it will force every time.

Restore to original state (if possible)

if master2 and master3 were not crash and lost due to other reason(mean disk and data is available) then as soon as these masters are available you can do following steps to go to original state

cd /etc/kubernetes/manifest
rm -rf etcd.yaml
cd /var/lib/etcd
rm -rf member
cp -R /backup/member /var/lib/etcd/
cp /backup/etcd.yaml /etc/kubernetes/manifest/

This is how things works for me, Please suggest if i can optimize these steps.