0

Is there a built-in or elegant way to automatically restart a kubernetes pod if a certain log line is seen in the logs?

Nate Houk
  • 111
  • 2
  • What have you already tried and what outcome is it producing for you? Are the logs on disk, or only going to stdout? – mdaniel Jun 08 '22 at 15:40
  • 1
    the elegant way to restart a Pod would be for its livenessProbe to return unhealthy – SYN Jun 08 '22 at 17:07

1 Answers1

1

There is no kind of built-in liveness probe which directly examines the output of kubectl logs respectively what the process within your container is writing to stdout.

But, if the process of your container writes its log messages to a file or if you are able to redirect the output of the process to a file within the container (e.g., using tee, but be aware to properly rotate the file), you can use a liveness probe.

Let's assume, that logs are written to /var/log/my-service.log and whenever we observe time to say good bye, we need to restart the container.

We can achieve this behavior, by using the following pod spec:

spec:
  containers:
  - image: busybox
    name: sf-1102768
    command: ["/bin/sh", "-c", "mkdir -p /var/log && echo hello && sleep 60 && echo time to say good bye > /var/log/my-service.log && sleep 3600 && echo hm... still here"]
    livenessProbe:
      exec:
        command: ["/bin/sh", "-c", "! grep -q 'time to say good bye' /var/log/my-service.log"]
      initialDelaySeconds: 5
      periodSeconds: 5
Gerald Mayr
  • 111
  • 2