2

This is a kubeadm based 1.16 cluster. The way I understand mTLS work between an apiserver & kubelet is:

apiserver -> kubelet

--kubelet-client-certificate & --kubelet-client-key => The certs & key given here(apiserver) is for apiserver(client) to kubelet (server)
--client-ca => The CA given here(kubelet) is what verfies if request from apiserver is indeed the right one.

And then we have the other way:

kubelet -> apiserver

--tls-cert-file & --tls-key-file => Passed in kubelet conf. Generally present at /var/lib/kubelet/pki/kubelet.{crt,key}
--kubelet-certificate-authority => The above cert gets verified by this CA passed in apiserver

However, there is a kubelet.conf file present at /etc/kubernetes which again has client-certificate-data and client-key-data . When is this client-cert used?

Also, if we enable auto-rotation in kubelet, there is a client-cert under /var/lib/kubelet/pki/kubelet-client-current.pem which may or may not be referenced in kubelet.conf based on the kubeadm version used (because of the bug in kubeadm prior to 1.17). I'm confused on how & when are the kubelet client cert used. Any help on this?

swetad90
  • 131
  • 5

1 Answers1

1

The client-certificate-data and client-key-data are there due to a bug. This is well reflected by the official docs:

Warning: On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

Some useful sources with more info:

EDIT:

Based on the kubelet integration:

When you run kubeadm join, kubeadm uses the Bootstrap Token credential to perform a TLS bootstrap, which fetches the credential needed to download the kubelet-config-1.X ConfigMap and writes it to /var/lib/kubelet/config.yaml. The dynamic environment file is generated in exactly the same way as kubeadm init.

Next, kubeadm runs the following two commands to load the new configuration into the kubelet:

systemctl daemon-reload && systemctl restart kubelet

After the kubelet loads the new configuration, kubeadm writes the /etc/kubernetes/bootstrap-kubelet.conf KubeConfig file, which contains a CA certificate and Bootstrap Token. These are used by the kubelet to perform the TLS Bootstrap and obtain a unique credential, which is stored in /etc/kubernetes/kubelet.conf. When this file is written, the kubelet has finished performing the TLS Bootstrap.

This indicates that cert/credentials stored in kubelet.conf are used during kubelet startup process. Removing this file or changing its content will be verified during kubelet restart. This can be tested yourself as below:

  • If you remove the file:
failed to run kubelet: unable to load bootstrap kubeconfig
  • If you misconfigure the file or provide wrong certs:
config loaded from file: /etc/kubernetes/kubelet/conf
bootstrap.go:240 unable to read existing bootstrap client config: f.e. invalid configuration
or unable to load TLS certificates from existing bootstrap client config: data does not contain any valid RSA or ECDSA certificate

Also, as a side note:

I strongly recommend using a newer version of Kubernetes as yours is 5 major versions older than the current stable one: 1.21 / April 8, 2021. Keeping things up-to-date will save you a lot of trouble in the future.

Wytrzymały Wiktor
  • 731
  • 3
  • 10
  • Yes, I read about the bug. Have updated my question with the info as well. But when both the embedded certs & certs under /var/lib/kubelet/pki are present, am I right to understand that kubelet actually uses the client certs under pki folder for all the mtls communication & ignores the embedded certs in kubelet.conf ? – swetad90 May 17 '21 at 14:33
  • Hi @swetad90. Thanks for your response. I have dug a bit deeper and edited my initial answer to better address your question. Feel free to review it. – Wytrzymały Wiktor May 20 '21 at 11:34
  • Apologies just seeing this. agree to the use of the kubelet.conf file. Were you able to find anything on the use of the client certs under /var/lib/kubelet/pki directory? Because if I delete those, kubelet fails to restart. – swetad90 Jun 09 '21 at 19:22
  • kubelet is run with command line option `--kubeconfig=/etc/kubernetes/kubelet.conf` In the config file, it sets: `client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem` `client-key: /var/lib/kubelet/pki/kubelet-client-current.pem` So it points to the client cert/key under /var/lib/kubelet/pki. The cert/key is used for kubelet to talk to apiserver. – B.Z. May 27 '22 at 19:45