Can a liveness probe prevent a cluster from scaling down?

Question

I have a liveness probe which is configured to check an endpoint's availability:

livenessProbe:
   httpGet:
      path: /path_example/
      port: 8000
   initialDelaySeconds: 10
   periodSeconds: 60

The cluster has autoscaling enabled as per instructions here - https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler with 1 minimum and 3 maximum

Although ten plus minutes go by the cluster always displays "current total size - 3". Other than the liveness probe nothing is using the application.

Could this be causing the nodes to remain and never get scaled down?

I cannot see any other reason why the nodes don't ever go down.

UPDATE: I've set resource for cpu and set autoscaler on the deployments so now for 'kubectl get hpa' I get:

NAME                   REFERENCE                         TARGETS   MINPODS   MAXPODS   REPLICAS   AGE

one-deployment         Deployment/one-deployment         34%/80%   1         3         1          2m8s

two-deployment         Deployment/two-deployment         47%/80%   1         3         1          8m16s

three-deployment       Deployment/three-deployment       35%/80%   1         3         1          3m29s

four-deployment        Deployment/four-deployment        33%/80%   1         3         1          2m48s

five-deployment        Deployment/five-deployment        47%/80%   1         3         1          2m24s

But still I remain at the max of 3 nodes.

Another update: I'd appreciate any feedback on what I believe is the summary of my learning. I'm quite new to Kubernetes and GKE I'm so please forgive me.

Firstly I now better understand there is autoscaling of nodes on clusters and then there is autoscaling of pods on nodes.

The part I needed to get right first was autoscaling of nodes on clusters. When autoscaling is enabled on a cluster, with for example, --enable-autoscaling --no-of-nodes 2 --min-nodes 1 --max-nodes 3. Then I'm causing the deployment to run on 2 nodes, if there is so little resource required and pods can move nodes then it may go down to 1. If a specified --num-of-nodes 3 then I'd deploy to three and this could result in unmoveable pods being spread across all three nodes preventing the ability to downscale to 2 or 1.

As starting with 1 caused my application to fail to fully deploy I've set this to 2.

Now to scaling my deployments to potentially increase the number of pods: In the GCP GKE console I've selected 'workload' and then one of my deployments in the list of pods. Then from here I select 'Action' from the menu at the top and then 'autoscaling', I've left the default of 1 minimum and 3 maximum and ok'd this. I've repeated this for the other 4 deployments I have. This is the horizontal pod scaling that I was getting mixed up with when I first started looking at cluster scaling. This is what I get details about when I run 'kubectl get hpa'. This doesn't relate to the cluster's node scaling at all as far as I can tell.

Now my application runs and when there is sufficient load on my pods the hpa autoscaling will kick in and create new pods. These pods will be run in my existing two nodes unless there is insufficient space at which point the cluster (configured to have max 3) will add a third node and assign the new pod to this node.

So hopefully my final question - Have I put two and two together and got 5?

Answer 1

Welcome on stack! Here are couple of things that might help you:

Cluster Autoscaler will decrease the size of the cluster when some nodes are consistently under utilized for a significant amount of time. A node is unneeded when it has low utilization and all of its important pods can be moved elsewhere.

Without having more information about your environment (how did you created cluster, whats running there, what is your utilization etc) is it very difficult to guess but here are some pod types that can prevent you cluster from scaling down (if you have low utilization my guess would be on the pods that are cannot be evicted):

• Pods that are utilizing your cluster
• Pods with restrictive PodDisruptionBudget.
• Kube-system pods that:
  • are not run on the node by default, *
  • don't have a pod disruption budget set or their PDB is too restrictive (since CA 0.6).
• Pods that are not backed by a controller object (so not created by deployment, replica set, job, stateful set etc). *
• Pods with local storage. *
• Pods that cannot be moved elsewhere due to various constraints (lack of resources, non-matching node selectors or affinity, matching anti-affinity, etc)
• Pods that have the following annotation set:

"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

*Unless the pod has the following annotation (supported in CA 1.0.3 or later):

"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"

Or you have have overridden this behavior with one of the relevant flags. See below for more information on these flags.

How do you scale your cluster then just to have one node?

Prior to version 0.6, Cluster Autoscaler was not touching nodes that were running important kube-system pods like DNS, Metrics Server, Dashboard, etc. If these pods landed on different nodes, CA could not scale the cluster down and the user could end up with a completely empty 3 node cluster. In 0.6, we added an option to tell CA that some system pods can be moved around. If the user configures a PodDisruptionBudget for the kube-system pod, then the default strategy of not touching the node running this pod is overridden with PDB settings. So, to enable kube-system pods migration, one should set minAvailable to 0 (or <= N if there are N+1 pod replicas.)

For troubleshooting this you might want to check also I have a couple of nodes with low utilization, but they are not scaled down. Why?