0

Made a mistake in a puppet config and inadvertently let puppet upgrade a kubernetes cluster to 1.13.0. Since k8 wants a somewhat involved upgrade process the whole thing was fairly botched.

After running the process manually and getting all impacted nodes and master to the same version it seems that kubelet (and thus the api server) will run for 20-30 minutes and then stop responding.

As it starts to get unhappy the log begins seeing more entries like these:

level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:110: decoding init error from pipe caused \\"read parent: connection reset by peer\\"\"\n"

and

file.go:182] Not recursing into manifest path "/etc/kubernetes/manifests/<path>"

with various namespace entries for path. This cycle repeats every 20 seconds until kubelet eventually stops responding. If I manually restart kubelet it works for a while and begins the cycle anew.

Did I miss something in the upgrade process?

Edit:

[root@kube-master ~]# rpm -qa | grep docker
docker-client-1.13.1-84.git07f3374.el7.centos.x86_64
docker-1.13.1-84.git07f3374.el7.centos.x86_64
docker-common-1.13.1-84.git07f3374.el7.centos.x86_64
ethrbunny
  • 2,327
  • 4
  • 36
  • 72

1 Answers1

1

The first error message looks very likely this one.

Maybe you can check if those affetcted nodes are running a recent version docker?

Slacker
  • 26
  • 1