I have had a k8s cluster running for a while without issues. A few days ago, it wouldn't start and when the kubelet starts it creates a number of control-plane containers (in an apparent infinite loop) all marked as Created with the following error in Docker and the kubelet logs for every created container:
Handler for POST /v1.40/containers/<ID>/start returned error: unable to find user 0: invalid argument
Background information:
- the kubelet version was v1.19.1 when I noticed it, the cluster was still running v1.19.0
- docker is version 19.03.12 on Debian (tested with both Linux 5.7.0-3 and 5.8.0-1 on testing)
- I initially suspected the upgrade to kubelet v1.19.1 (from v1.19.0) but reversing the upgrade did not address the issue
- I tried a few
docker run -it <image> sh -i
with various images but they run fine including when volume mounting with-v
and forwarding ports with-p
on the command line; these containers run fine - re-installing docker with
apt purge ... && apt install ...
did not resolve the problem even when deleting/var/lib/{docker,kubelet}
before re-installing - neither did
kubeadm reset ... && kubeadm init
- docker uses the overlay2 storage-driver and the systemd cgroup-driver
- I do not have logs from when it started as I wasn't using this (development) cluster for a few days and the subsequent errors filled up the logs
Any hints and suggestions are appreciated. The error message is unfortunately not helpful enough to figure out what or even just where in the dependency chain the problem is.