kube-apiserver exits while control plane joining the HA cluster

Question

Ubuntu 16.04.5
All operation are done as root user
software versions are as below:

ii  kubeadm                             1.20.4-00                                  amd64        Kubernetes Cluster Bootstrapping Tool
ii  kubectl                             1.20.4-00                                  amd64        Kubernetes Command Line Tool
ii  kubelet                             1.20.4-00                                  amd64        Kubernetes Node Agent
ii  kubernetes-cni                      0.8.7-00                                   amd64        Kubernetes CNI
ii  containerd.io                       1.2.6-3                                    amd64        An open and reliable container runtime

I'm Creating Highly Available clusters with kubeadm following the guide:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

Assumed that the control plane nodes are: A(devops1ar01n01 172.16.80.3) B(devops1ar01n02 172.16.80.4) C(devops1ar01n03 172.16.80.5).

I followed the link below to setup a load balancer using kube-vip. On both A and B I created file /etc/kube-vip/config.yaml and /etc/kubernetes/mainfest/kube-vip.yaml:

https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#kube-vip

The I ran command below to init the first control plane node A(kube-vip listens on port 16443):

kubeadm init --control-plane-endpoint kube-vip:16443 --upload-certs

The output are as follow:

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join kube-vip:16443 --token pa0bw2.gn6bqnyjlmh0o7xn \
    --discovery-token-ca-cert-hash sha256:fd7bb5afe0307b8694c218f07c1f3adbf270254d1f37bcec75ed292b7223cc8b \
    --control-plane --certificate-key 44995042d21c87ea5ed4f62443fe665cbfd7c71397485ca9f06d1483548c1883

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join kube-vip:16443 --token pa0bw2.gn6bqnyjlmh0o7xn \
    --discovery-token-ca-cert-hash sha256:fd7bb5afe0307b8694c218f07c1f3adbf270254d1f37bcec75ed292b7223cc8b

Then I followed the output to run commands:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

export KUBECONFIG=/etc/kubernetes/admin.conf

Then I installed the CNI plugin weave on node A by running command below:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=1.20.4"

Then check the pods:

root@devops1ar01n01:~# kubectl get pod -n kube-system
NAME                                     READY   STATUS    RESTARTS   AGE
coredns-74ff55c5b-s2bh5                  0/1     Running   0          36m
coredns-74ff55c5b-stm2l                  1/1     Running   0          36m
etcd-devops1ar01n01                      1/1     Running   0          36m
kube-apiserver-devops1ar01n01            1/1     Running   0          36m
kube-controller-manager-devops1ar01n01   1/1     Running   0          36m
kube-proxy-bnzpd                         1/1     Running   0          36m
kube-scheduler-devops1ar01n01            1/1     Running   0          36m
kube-vip-devops1ar01n01                  1/1     Running   0          36m
weave-net-8fmf9                          2/2     Running   0          14s

Til then, everyting goes well, but something goes wrong while node B joining the cluster.(--v=8 to see the verbose output, --ignore-preflight-errors="DirAvailable--etc-kubernetes-manifests" due to the existing file /etc/kubernetes/mainfest/kube-vip), below command is run on node B.

kubeadm join kube-vip:16443 --token pa0bw2.gn6bqnyjlmh0o7xn \
   --discovery-token-ca-cert-hash sha256:fd7bb5afe0307b8694c218f07c1f3adbf270254d1f37bcec75ed292b7223cc8b \
   --control-plane \
   --certificate-key 44995042d21c87ea5ed4f62443fe665cbfd7c71397485ca9f06d1483548c1883 \
   --ignore-preflight-errors="DirAvailable--etc-kubernetes-manifests" 
   --v=8

The following message are shown then(172.16.80.4 is the ip of node B):

[kubelet-check] Initial timeout of 40s passed.
I0226 11:12:44.981744   11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded
I0226 11:12:52.890038   11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded
I0226 11:13:03.915500   11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded
I0226 11:13:19.337921   11128 etcd.go:468] Failed to get etcd status for https://172.16.80.4:2379: failed to dial endpoint https://172.16.80.4:2379 with maintenance client: context deadline exceeded

I found that there is no etcd container created on node B:

root@devops1ar01n02:~# docker ps | grep -v pause
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
b0188090c251        ae5eb22e4a9d           "kube-apiserver --ad…"   20 seconds ago      Up 19 seconds                           k8s_kube-apiserver_kube-apiserver-devops1ar01n02_kube-system_50f7004f736896db78d143e1d44bfbb5_4
c8c93ad432e9        7f92d556d4ff           "/usr/bin/launch.sh"     3 minutes ago       Up 3 minutes                            k8s_weave-npc_weave-net-lthlv_kube-system_eac41670-a119-4085-99e7-7cf08185deb7_0
e9946edd52ba        5f8cb769bd73           "kube-scheduler --au…"   3 minutes ago       Up 3 minutes                            k8s_kube-scheduler_kube-scheduler-devops1ar01n02_kube-system_90280dfce8bf44f46a3e41b6c4a9f551_0
4ffe61f78cf5        a00c858e350e           "/kube-vip start -c …"   3 minutes ago       Up 3 minutes                            k8s_kube-vip_kube-vip-devops1ar01n02_kube-system_dd4d116d758ec63efaf78fc4112d63e6_0
7019afbd1497        0a41a1414c53           "kube-controller-man…"   3 minutes ago       Up 3 minutes                            k8s_kube-controller-manager_kube-controller-manager-devops1ar01n02_kube-system_9375c16649f1cd963bdbc6e4125314fc_0
32035400ad9d        c29e6c583067           "/usr/local/bin/kube…"   3 minutes ago       Up 3 minutes                            k8s_kube-proxy_kube-proxy-v5nwr_kube-system_d0c1ce98-b066-4349-89e0-6113b8fa1708_0

And when I went back to node A to check the pods, kubectl command hang and then timeout:

root@devops1ar01n01:~# kubectl get pod -n kube-system
Unable to connect to the server: net/http: TLS handshake timeout

And I checked the containers, found that the kube-apiserver is always restarting:

root@devops1ar01n01:~# docker ps -a | grep -v pause | grep kube-apiserver
d5f85d72d2dc        ae5eb22e4a9d           "kube-apiserver --ad…"   8 seconds ago        Up 8 seconds                                        k8s_kube-apiserver_kube-apiserver-devops1ar01n01_kube-system_860fed4d3a137b129887eb23f07be1b6_6
a3bd40ba5552        ae5eb22e4a9d           "kube-apiserver --ad…"   About a minute ago   Exited (1) About a minute ago                       k8s_kube-apiserver_kube-apiserver-devops1ar01n01_kube-system_860fed4d3a137b129887eb23f07be1b6_5

I ran docker logs <container_id> on node A to get the exited kube-apiserver container's log, output is as below:

Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0226 06:22:58.186450       1 server.go:632] external host was not specified, using 172.16.80.3
I0226 06:22:58.187781       1 server.go:182] Version: v1.20.4
I0226 06:22:59.289621       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0226 06:22:59.294839       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0226 06:22:59.294939       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0226 06:22:59.299670       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0226 06:22:59.299772       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0226 06:22:59.318732       1 client.go:360] parsed scheme: "endpoint"
I0226 06:22:59.318985       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379  <nil> 0 <nil>}]
I0226 06:23:00.290273       1 client.go:360] parsed scheme: "endpoint"
I0226 06:23:00.290377       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379  <nil> 0 <nil>}]
Error: context deadline exceeded

And systemctl status kubelet outputs below:

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Fri 2021-02-26 10:31:19 CST; 3h 58min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 19602 (kubelet)
    Tasks: 17
   Memory: 67.0M
      CPU: 20min 47.367s
   CGroup: /system.slice/kubelet.service
           └─19602 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --p

Feb 26 14:29:34 devops1ar01n01 kubelet[19602]: Trace[336098419]: [10.001217223s] [10.001217223s] END
Feb 26 14:29:34 devops1ar01n01 kubelet[19602]: E0226 14:29:34.695789   19602 reflector.go:138] object-"kube-system"/"kube-proxy-token-x5lsv": Failed to watch *v1.Secret: failed to list *v1.Secret: Get "
Feb 26 14:29:36 devops1ar01n01 kubelet[19602]: E0226 14:29:36.056753   19602 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://kube-vip:16443/apis/coordination.k8s.
Feb 26 14:29:40 devops1ar01n01 kubelet[19602]: E0226 14:29:40.068403   19602 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"
Feb 26 14:29:40 devops1ar01n01 kubelet[19602]: E0226 14:29:40.068717   19602 event.go:218] Unable to write event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k
Feb 26 14:29:40 devops1ar01n01 kubelet[19602]: E0226 14:29:40.069126   19602 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "devops1ar01n01": Get "https://kube-vi
Feb 26 14:29:44 devops1ar01n01 kubelet[19602]: I0226 14:29:44.012843   19602 scope.go:111] [topologymanager] RemoveContainer - Container ID: 3ab3a85e785ae39f705ca30aad59a52ec17d12e9f31cbf920695d7af9cf93
Feb 26 14:29:44 devops1ar01n01 kubelet[19602]: E0226 14:29:44.031056   19602 pod_workers.go:191] Error syncing pod 860fed4d3a137b129887eb23f07be1b6 ("kube-apiserver-devops1ar01n01_kube-system(860fed4d3a
Feb 26 14:29:50 devops1ar01n01 kubelet[19602]: E0226 14:29:50.070777   19602 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"
Feb 26 14:29:50 devops1ar01n01 kubelet[19602]: E0226 14:29:50.072142   19602 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "devops1ar01n01": Get "https://kube-vi

I've tried kubeadm reset both nodes and repeat the operations, but this situation will always show up. How shall I debug this furthur?

Hi Owen, welcome to S.F. What troubleshooting steps have you already taken, especially around the culprit message `failed to dial endpoint https://172.16.80.4:2379`? is etcd listening on that IP? Is it running? do you have firewall rules interfering? We are not on your machine, so you will have to provide more insight into what you have already tried for anyone to help you. And with k8s, as goes etcd, so goes everything else — mdaniel, Feb 27 '21 at 19:41
thanks for your help! etcd container is not even created on host B(172.16.80.4), but I finally found the problem, I will answer the question myself. — Owen Chen, Feb 28 '21 at 10:41

score 1 · Accepted Answer · answered Feb 28 '21 at 10:57

The problem is finally solved by myself. This is probably caused by a wrong health check url in the load balancer.

I followd this guide https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#kube-vip to create the load balancer for my HA cluster and I picked the way of kube-vip, which seemed to be the most convenient one.

If you use kube-vip, he health check url might be /healthz as described in the haproxy config file of the second method(https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#keepalived-and-haproxy). However, I found that the health check url is /livez in /etc/kubernetes/mainfest/kube-apiserver.yaml

I guess that it should be the wrong health check url in kube-vip which caused the health check failure, and then lead to continous restarting of kube-apiserver.

To verify this, in order to edit the health check url in the load balancer, I chose the second way to create load balancer(https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#keepalived-and-haproxy) and changed /healthz to /livez in /etc/haproxy/haproxy.cfg

Then I followd the guide to run kube init and kube join, it can work well.

kube-apiserver exits while control plane joining the HA cluster

1 Answers1