9

In have scaled my pods to 20 in my cluster and when I see the next day the few of the scaled pods were recreated.

When I say POD recreated, it is getting deleted and created freshly and the timestamp of the recreated pod and the scaled pods vary.

I was unable to find the reasons for the recreate of the PODs.

I could not find which POD went for a recreate as the POD is deleted and gone away. There are no logs in the journalctl regarding which POD got recreated. Is there any way I can debug further to find the reason for the POD recreate. or What might be the reason for the PODs getting deleted.

Note: I have readiness and liveness probes defined, but these probes would act on container and would not lead to POD recreate is my understanding.

Nikhil
  • 151
  • 1
  • 1
  • 6
  • Those liveness probes are _designed_ to kill off bogus Pods, that's their job in life, but you are correct that a liveness fail will cause a restart event, and will not change the `.metadata.name` of the Pod, only its restart count. It is also supremely likely that if the Pod names are changing, it's due to a ReplicaSet change, not a Pod event. Either way, you have a lot of words that are not `kubectl describe`, `kubectl get events` and similar investigations and their outcomes. No one can *guess* what is wrong with your setup without knowing what concrete experiments you have already tried – mdaniel May 16 '21 at 18:11
  • Add the commands you have tried so far to debug and their outputs. Also include the outputs from commands mentioned by mdaniel – Matt May 17 '21 at 08:09
  • Hi @Matt, the POD is gone... I cannot do describe or events using kubectl and events does not give the details of all the pods. – Nikhil May 17 '21 at 08:16
  • @mdaniel, the Pods were scaled to 20 and there was a load executed for these pods. After sometime few of the Pods recreated. This is the only operation done on the cluster. – Nikhil May 17 '21 at 08:20
  • "*When I say POD restart, it is getting deleted and created freshly*" - So its not being restarted, it's being recreated. Can you change every occurance of a word **restarted** with a word **recreated** in your question, where this is what you actually mean. Because this is misleading. `restarted != recreated` - these are two different things. – Matt May 17 '21 at 09:35
  • Thanks @Matt. Have edited my question. – Nikhil May 17 '21 at 11:20

1 Answers1

7

Basically you need to check pod's events (here is a tip how to do it).

Keep in mind, that events retention period is too short (approximately 1h), and you may need to store them somewhere else. Here is a good article how to do it with EFK stack

The most common reason of PODs recreation - node unavailability. If pod's node is not available, Kubernetes recreates pods at other nodes.

Denis Romaniuk
  • 96
  • 2