2

When running kubectl, I get the error

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2021-10-05T11:59:14-04:00 is after 2021-09-29T19:21:40Z

So clearly it says the cert is expired. Only problem is I'm not sure which cert it is.

I've checked

  • HAProxy (Rancher sits behind L7 HAProxy with LE cert)
  • Certs in the secrets shown from sudo k3s kubectl get secrets -n cattle-system
  • Certs in /etc/kubernetes/ssl on the K8s node

All are fine (not expired), as this particular rancher/k8s instance was brought up in June, so all the certs are only a few months old, and expire either 1 year or 10 years later.

So what cert is expired that needs to be updated?

Some information about my setup:

  • Rancher 2.5.9 HA (K3s v1.21.1+k3s1) (single-node, Ubuntu 20.04)
  • Kubernetes 1.20.9-rancher1-1 (single-node, Control plane/Worker/etcd, Ubuntu 20.04)
cclloyd
  • 583
  • 1
  • 13
  • 24
  • Have you checked the cert that is usually embedded in your kubeconfig? And have you actually _looked_ at the cert being returned, as in `echo '' | openssl s_client -servername whatever.example.com -showcerts -connect whatever.example.com:6443 | openssl x509 -noout -text` kind of deal? – mdaniel Oct 06 '21 at 03:09
  • @mdaniel yes, they are all valid. And I just did. The cert returned by that command is `/etc/kubernetes/ssl/kube-apiserver.pem`. – cclloyd Oct 06 '21 at 04:57
  • @cclloyd Have you found the solution? – Adam Nov 22 '21 at 21:07
  • 1
    @Adam the LE cert was invalid because one of the root certs was invalid. Had to update root cert packages on host systems and regenerated cert. – cclloyd Nov 23 '21 at 00:57
  • Hi @cclloyd Could you post an answer, since it would be better for other users to see that you found the solution and also for indexing the answer by the site? – Andrew Skorkin Apr 04 '22 at 12:50

1 Answers1

1

This is a community wiki answer posted for better visibility. Feel free to expand it.

Based on information from comments

Root cause:

One of the root certificates is invalid. This caused the Let's Encrypt certificate to be invalid.

Solution:

  1. Update root cert packages on host systems
  2. Regenerate certificate
Andrew Skorkin
  • 101
  • 5