2

I have a kubeadm installed kubernetes cluster. Recently it stopped working. kubelet is running but seems stuck in initialization phases. I think the root cause is this recurring log in kube-apiserver:

1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2021-06-02T13:18:50Z is after 2021-05-29T15:48:22Z

So there is a certificate issue, also kubectl is failing with unauthorized. The thing is, kubeadm certs check-expiration seems happy, and I even manually checked a few yaml config files (base64 decoded certificates, and run them through openssl to check the date). Nevertheless, I asked kubeadm to renew all certificates and rebooted everything, to no effect.

Any idea how I can identify which certificate exactly has expired ?

Antoine
  • 281
  • 3
  • 8

3 Answers3

3

[acknowledgment and reference] I was helped by a kubernetes' dev here

The expired certificate was /var/lib/kubelet/pki/kubelet/pki/kubelet-client-2020-*.pem. The certificates in /var/lib/kublet/pki/ are not handled by kubeadm cert but by kubelet itself, so it's supposed to be renewed automatically, but for some reason this didn't happen as planned for us. The kubelet-client-current.pem had been renewed, but something was still using an old (and expired) certificate.

Here is how I fixed the issue:

  • /etc/kubernetes/kubelet.conf was obsolete, in particular using default-user instead of system:node:node_name. I deleted the file, created a kubeadm conf file and ran kubeadm init phase kubeconfig kubelet to recreate a clean kubelet.conf
  • /var/lib/kublet/pki/kubelet-client-current.pem is supposed to be a symlink, which was not the case for me. So I removed it.
  • restart kubelet and apiserver (kill the pod using containerd, docker, etc. since kubectl is unavailable) and wait for a new kubelet-client-current.pem to be created ; it should be a symlink.
  • run kubeadm init phase kubelet-finalize all
  • restart kubelet again
  • run kubeadm certs renew all
  • reboot (or restart kubelet and all control plane pods)
  • update your kubectl conf from /etc/kubernetes/admin.conf
Antoine
  • 281
  • 3
  • 8
  • 1
    For anyone interested, the documentation ended up at https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert – Joel Pearson Nov 11 '21 at 01:50
0

Similar issues seems to be linked to NTP desynchronization.
Try forcing time synchronization (run as root):

# service ntp stop
# ntpd -gq
# service ntp start
p10l
  • 386
  • 1
  • 7
0

Renew Kubernetes certificates (RUN on all master node)

#kubeadm alpha certs check-expiration
#kubeadm init phase kubelet-finalize all
#kubeadm alpha certs renew all
#cd /etc/kubernetes
#kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf
#mkdir -p $HOME/.kube
#cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#chown $(id -u):$(id -g) $HOME/.kube/config
#restart server
#kubectl get nodes
slightly_toasted
  • 732
  • 3
  • 13
Abhishek
  • 1
  • 1