3

kubelet has an option to store log files in a log-dir (https://github.com/kubernetes/kubernetes/issues/21248).

I start kubelet with the options --logtostderr=false --log-dir=/var/log/kubelet (I have already created the directory /var/log/kubelet) and I indeed see log files with the format kubelet.(hostname).root.log.(date) getting generated in /var/log/kubelet.

In my case I see three files that look like this:

kubelet.hostname.root.log.ERROR.20180301-152321.161945
kubelet.hostname.root.log.INFO.20180301-152320.161945
kubelet.hostname.root.log.WARNING.20180301-152320.161945

Moreover, a symbolic link that points to the latest log file is created for each of the ERROR, INFO and WARNING log files:

kubelet.ERROR -> kubelet.hostname.root.log.ERROR.20180301-152321.161945
kubelet.INFO -> kubelet.hostname.root.log.INFO.20180301-152320.161945
kubelet.WARNING -> kubelet.hostname.root.log.WARNING.20180301-152320.161945

My question is how do I rotate and delete the log files that are older than e.g. 5 days old? It seems these files are generated by glog, but it also seems that although glog rotates the log files, it doesn't support deleting old log files (https://github.com/google/glog/issues/36).

I am aware of logrotate, but then the files should have a fixed filename in order to be rotated by logrotate.

Is there a way to enable kubelet logging in files with proper log rotation and deletion of old files?

Vangelis Tasoulas
  • 196
  • 1
  • 7
  • Why not cron + find? By default, with systemd, kubelet will write to journald. If you are taking control of log management, why not use cron and find to do whatever you need to do with old logs? – Jonah Benton Mar 06 '18 at 00:12
  • That's the *quick* solution I was thinking about (cron + find). However, I was looking if I could somehow use either logrotate or some *hidden* kubelet configuration parameter that I am not aware of. Then I would have a more *streamlined* solution that is closer to the way the *standard* log rotation works. – Vangelis Tasoulas Mar 06 '18 at 16:07
  • Well, distros have moved to systemd. Thats the standard. kubelet does journald by default. Plenty of people dont like systemd, of course. But if you have requirements not met by systemd, would be good to add them to the question. – Jonah Benton Mar 06 '18 at 16:13
  • Yes, this is true about systemd. The problem is that this machine is facing some issues unrelated to this question, but one of the issues is that journald was crashing. That's why kubelet was configured to log into files instead. – Vangelis Tasoulas Mar 06 '18 at 20:36
  • Did @Wytrzymały Wiktor answer help you to solve your problem?If yes,Please consider accepting and upvoting it. What should I do when someone answers my question? – Ramesh kollisetty Apr 05 '22 at 09:08

1 Answers1

0

Kubernetes doesn’t provide built-in log rotation, but this functionality is available in many tools.

Basing on the official Logging Architecture:

An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour. You can also set up a container runtime to rotate an application's logs automatically.

As an example, you can find detailed information about how kube-up.sh sets up logging for COS image on GCP in the corresponding configure-helper script.

Wytrzymały Wiktor
  • 731
  • 3
  • 10