Is there a built-in or elegant way to automatically restart a kubernetes pod if a certain log line is seen in the logs?
Asked
Active
Viewed 200 times
0
-
What have you already tried and what outcome is it producing for you? Are the logs on disk, or only going to stdout? – mdaniel Jun 08 '22 at 15:40
-
1the elegant way to restart a Pod would be for its livenessProbe to return unhealthy – SYN Jun 08 '22 at 17:07
1 Answers
1
There is no kind of built-in liveness probe which directly examines the output of kubectl logs
respectively what the process within your container is writing to stdout
.
But, if the process of your container writes its log messages to a file or if you are able to redirect the output of the process to a file within the container (e.g., using tee
, but be aware to properly rotate the file), you can use a liveness probe.
Let's assume, that logs are written to /var/log/my-service.log
and whenever we observe time to say good bye
, we need to restart the container.
We can achieve this behavior, by using the following pod spec:
spec:
containers:
- image: busybox
name: sf-1102768
command: ["/bin/sh", "-c", "mkdir -p /var/log && echo hello && sleep 60 && echo time to say good bye > /var/log/my-service.log && sleep 3600 && echo hm... still here"]
livenessProbe:
exec:
command: ["/bin/sh", "-c", "! grep -q 'time to say good bye' /var/log/my-service.log"]
initialDelaySeconds: 5
periodSeconds: 5
Gerald Mayr
- 111
- 2