How can I find which kubernetes certificate has expired?

Question

I have a kubeadm installed kubernetes cluster. Recently it stopped working. kubelet is running but seems stuck in initialization phases. I think the root cause is this recurring log in kube-apiserver:

1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2021-06-02T13:18:50Z is after 2021-05-29T15:48:22Z

So there is a certificate issue, also kubectl is failing with unauthorized. The thing is, kubeadm certs check-expiration seems happy, and I even manually checked a few yaml config files (base64 decoded certificates, and run them through openssl to check the date). Nevertheless, I asked kubeadm to renew all certificates and rebooted everything, to no effect.

Any idea how I can identify which certificate exactly has expired ?

score 3 · Accepted Answer · answered Jun 03 '21 at 19:41

[acknowledgment and reference] I was helped by a kubernetes' dev here

The expired certificate was /var/lib/kubelet/pki/kubelet/pki/kubelet-client-2020-*.pem. The certificates in /var/lib/kublet/pki/ are not handled by kubeadm cert but by kubelet itself, so it's supposed to be renewed automatically, but for some reason this didn't happen as planned for us. The kubelet-client-current.pem had been renewed, but something was still using an old (and expired) certificate.

Here is how I fixed the issue:

/etc/kubernetes/kubelet.conf was obsolete, in particular using default-user instead of system:node:node_name. I deleted the file, created a kubeadm conf file and ran kubeadm init phase kubeconfig kubelet to recreate a clean kubelet.conf
/var/lib/kublet/pki/kubelet-client-current.pem is supposed to be a symlink, which was not the case for me. So I removed it.
restart kubelet and apiserver (kill the pod using containerd, docker, etc. since kubectl is unavailable) and wait for a new kubelet-client-current.pem to be created ; it should be a symlink.
run kubeadm init phase kubelet-finalize all
restart kubelet again
run kubeadm certs renew all
reboot (or restart kubelet and all control plane pods)
update your kubectl conf from /etc/kubernetes/admin.conf

For anyone interested, the documentation ended up at https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert — Joel Pearson, Nov 11 '21 at 01:50

score 0 · Answer 2 · answered Jun 03 '21 at 10:43

0

Similar issues seems to be linked to NTP desynchronization.
Try forcing time synchronization (run as root):

# service ntp stop
# ntpd -gq
# service ntp start

answered Jun 03 '21 at 10:43

p10l

386
1
7

Thanks for the suggestion. Time was correct, and I resynchronized with no effect. – Antoine Jun 03 '21 at 12:38

score 0 · Answer 3 · edited Aug 08 '21 at 17:37

Renew Kubernetes certificates (RUN on all master node)

#kubeadm alpha certs check-expiration
#kubeadm init phase kubelet-finalize all
#kubeadm alpha certs renew all
#cd /etc/kubernetes
#kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf
#mkdir -p $HOME/.kube
#cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#chown $(id -u):$(id -g) $HOME/.kube/config
#restart server
#kubectl get nodes

How can I find which kubernetes certificate has expired?

3 Answers3