0

One of the nodes in my Kubernetes v1.16.3 cluster has ended up in a strange state. The node is running Debian 10.1 and Docker 19.03.1. When I try to kubectl exec sh into a container on that node I see these error messages in the node's journalctl -u docker (the 2nd is also reported by kubectl itself):

stream copy error: reading from a closed fifo
Error running exec ... in container: failed to create runc console socket: mkdir /tmp/pty...: no space left on device ..."

At this point I don't see which device may have become full. A simple df does not raise any alarms. How can I further diagnose and remedy the situation?

BTW, the problem so far has persisted across kubectl drain, systemctl restart docker, restarting kube-proxy and the CNI's pod, as well as several other measures.

rookie099
  • 345
  • 2
  • 14
  • 1
    Can you provide some more details and output from the `df` and `kubectl describe node ` Can you check also if you are not running out of space allocated for Docker by removing some images and containers and then trying? You could try to use `docker system prune`.This will remove some unused data. – Matt Dec 13 '19 at 10:50
  • @HelloWorld A combination of `docker system prune` and `docker rmi $(docker images -q)` (plus waiting a bit) apparently remedied the situation. If you want to turn your comment into an answer I shall accept it. Thanks. – rookie099 Dec 16 '19 at 15:13

1 Answers1

0

As OP already mentioned in comments, a combination of docker system prune and docker rmi $(docker images -q) remedied the situation.

Matt
  • 528
  • 3
  • 7