10

I'm fairly new to k8s and minikube, and could use help understanding the frequent-but-not-ubiquitous error I get when running kubectl logs -f <POD NAME>

I get this, sometimes, and unpredictably, on pods in my own namespace, and in kube-system, e.g. when trying to tail the ingress controller logs. There is no apparent error in the pods themselves, and the output of the kubectl logs command contains normal-looking log lines right up to, e.g. failed to watch file "/var/log/pods/kube-system_nginx-ingress-controller-586cdc477c-95pgh_4b8310a4-5f9b-11e9-9666-0800270e7244/nginx-ingress-controller/1.log": no space left on device$

What's the ultimate source of this, and how do I resolve it?

Ben
  • 241
  • 1
  • 3
  • 10

1 Answers1

14

It sounds like the node has run out of inotify watches.

You can check or set the number of available watches using the sysctl fs.inotify.max_user_watches on each Kubernetes node.

You will probably find it has been set to some ridiculously low number; by default it is autotuned based on the amount of memory the system has.

[root@small ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

[root@large ~]# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 1048576

Whatever the amount is currently, double it until the problem stops occurring.

(Use minikube ssh to access a Minikube node.)

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • 3
    hey thanks! for those like me, new to k8s `sysctl` the command to set it is `sudo sysctl fs.inotify.max_user_watches=1048576` – Ben Apr 18 '19 at 18:31