I have a single node kubernetes cluster in google container engine to play around with.
Twice now, a small personal website I host in it has gone offline for a couple minutes. When I view the logs of the container, I see the normal startup sequence recently completed, so I assume a container died (or was killed?) and restarted.
How can I figure out the how & why of this happening?
Is there a way to get an alert whenever a container starts/stops unexpectedly?
You can view the last restart logs of a container using:
kubectl logs podname -c containername --previous
As described by Sreekanth, kubectl get pods should show you number of restarts, but you can also run
kubectl describe pod podname
And it will show you events sent by the kubelet to the apiserver about the lifecycled events of the pod.
You can also write a final message to /dev/termination-log, and this will show up as described in the docs.
Beside the previous answers another command that helped me to find an error is:
kubectl get event [--namespace=my-namespace]
It lists events from Pods, Jobs, Nodes too
I follow these steps to define the reason for failure:
# print out a pod logs (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs)
kubectl logs {name_of_pod} -n {namespace} --since=2h --timestamps
# print the logs for the _previous_ instance of the container in a pod if it exists
kubectl logs -p {name_of_pod} -n {namespace} --since=2h --timestamps
# check events (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe)
kubectl describe pod {pod_name} -n {namespace}
# look at the 'Events' at the end of the output
# ..
# Events:
# Type Reason Age From Message
# ---- ------ ---- ---- -------
# Warning BackOff 40m kubelet, gke-xx Back-off restarting failed container
# ..
# observe all events (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get)
kubectl get events -n {namespace} --sort-by=.metadata.creationTimestamp
# check logs, etc. in pod container (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec)
kubectl exec -it {pod_name} -n {namespace} -- sh
kubectl get pods will actually list any restarts of the container also the describe command can be of help cause it lists any events associated with the pod.
Liveness probes and readiness probes can be configured for better handling check here
Additionally hooks can be configured to be consumed in the container at specific points in the life cycle of the container check here
https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
When restarted
kubectl describe pod your-pod-name
Look for a section like this:
State: Running
Started: Wed, 23 Jun 2021 23:52:05 +1000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 23 Jun 2021 23:46:48 +1000
Finished: Wed, 23 Jun 2021 23:46:52 +1000
Ready: True
The interpretation of the above is as follows:
Wed, 23 Jun 2021 23:46:52 +1000 after having started at Wed, 23 Jun 2021 23:46:48 +1000, and is now running and ready, having been last started at Wed, 23 Jun 2021 23:52:05 +1000A pull request has now been merged into the kubernetes 1.22 milestone, to add LAST RESTART column to kubectl get pods, and will be available once that's released - see here.
https://github.com/kubernetes/kubernetes/pull/100142
To see your current version - kubernetes version
(1.21 is the latest release as at 28th June 2021)
If restarted
kubectl get po [your-pod-name]
The pod was restarted at some stage if there is any number in the RESTARTS column
Why restarted
kubectl describe pod [your-pod-name] will show a Last State which gives you a high level indication. To see what happened on the pod before it restarted, use kubectl logs your-pod-name --previous. You can pipe this to a file for inspection e.g.
kubectl logs your-pod-name --previous > pod_previous_log.txt
(See also above under 'When restarted')
