2

I run a kubernetes cluster, installed with kubeadm. I recently upgraded from 1.19 to 1.20 and migrated the container runtime from docker to containerd, since docker is now deprecated.

I configured containerd and kubelet to use it, and uninstalled docker from all nodes. Everything seems to be running fine.

Today, I tried to upgrade from 1.20 to 1.21 but I got two warnings when running kubeadm upgrade plan which make me think the containerd transition was not complete:

  • It tries to use docker:

    cannot automatically set CgroupDriver when starting the Kubelet: cannot execute 'docker info -f {{.CgroupDriver}}': executable file not found in $PATH
    • I probably have a configuration issue because kubeadm seems to not be aware that we don't use docker anymore, but I haven't found the right option in the documentation or local conf, except --cri-socket which doesn't work with kubeadm upgrade.
    • Second, the wording is strange: "when starting the Kubelet". But my kubelet is starting just fine, doesn't complain about missing docker or CgroupDriver.

  • It doesn't detect the cgroup driver setting:

    The 'cgroupDriver' value in the KubeletConfiguration is empty. Starting from 1.22, 'kubeadm upgrade' will default an empty value to the 'systemd' cgroup driver. The cgroup driver between the container runtime and the kubelet must match!

    This is really surprising because I have cgroupDriver: systemd in kubectl -n kube-system get cm kubelet-config-1.20 -o yaml, in /var/lib/kubelet/config.yaml, also the flag in /etc/default/kubelet and /var/lib/kubelet/kubeadm-flags.env, and it is even printed by kubeadm --v=10 !

How can I find out if there is an underlying configuration issue, or is I can safely ignore these warnings ?

I'm not sure what files, configmap or logs may be useful to help me solve this, but I will gladly provide them if needed.

Antoine
  • 281
  • 3
  • 8
  • Downvoting without telling why is kind of rude :\ – Antoine Apr 22 '21 at 15:16
  • After investigating further, I found this is not only a warning but an hard error for `kubeadm upgrade apply`. Seems related to issue https://github.com/kubernetes/kubeadm/issues/2364 – Antoine Apr 22 '21 at 15:20

1 Answers1

2

Although this was downvoted, in the event that someone has the same issue and feels as lost as I was, I hope google will lead you here :)

  • kubeadm was trying to use docker based on the node's configuration and not the kubelet's configuration, and I didn't change the nodes' config when migrating to containerd (as I was not aware it was set at that level). Fix with:

    kubectl annotate node $node --overwrite kubeadm.alpha.kubernetes.io/cri-socket=unix:///run/containerd/containerd.sock

    for each node. Found this here

  • The cgroupDriver warning is a bug in kubeadm that probably won't be fixed because the whole check is going away soon. So it can be ignored (if your kubelet's conf really sets the cgrouDriver).

Antoine
  • 281
  • 3
  • 8