1

My k8s version v1.17.13
My certificate expired today , so I ran
kubeadm alpha certs renew all
systemctl restart kubelet
on all my master servers.
All the kubectl commands that I ran worked fine .. like
kubectl get nodes , kubectl scale , kubectl describe ...
However , running kubectl logs gives the following error
error: You must be logged in to the server (the server has asked for the client to provide credentials

Any idea why ... I believe my ~/.kube/config is ok because I am able to run other kubectl commands. I deleted the kube-apiserver to force to restart .. but still same issue ..

May you please help me with this issue. Thanks,

bdaoud
  • 31
  • 1
  • 4

1 Answers1

2

While looking around ... I saw this on my worker nodes
Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid

After further troubleshooting ... only 2 of my 3 masters were causing this error
kubectl logs error: You must be logged in to the server (the server has asked for the client to provide credentials

After checking a lot of resources ,, I really couldn't find what is causing the problem , so I decided to reboot each of the 2 failing masters one at a time and that did the trick. I guess some of the pods in kube-system required restarting.

Additionally , I restarted kubelet on all worker nodes , but not sure if this had an effect or not.

Note that in https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#manual-certificate-renewal there is no mention about rebooting the masters

A final note .. I am not sure why the cert renew was not as smooth .. Before running into the kubectl logs problem described in this post ... I ran into this error on my first master ("bootstrap-kubelet.conf does not exist" issue which would not allow kubelet to restart) so I had to follow fix in https://stackoverflow.com/questions/56320930/renew-kubernetes-pki-after-expired

bdaoud
  • 31
  • 1
  • 4