We have a GKE cluster with autoscaling enabled. We are having google schedulers to shut down instances daily at a specific time. This also shuts down the GKE nodes. Since these clusters has autoscaling enabled and has a minimum node count, the nodes are getting recreated to match the minimal level, even after they are shut down by the script. Hence we are temporarily making the node count to 0 in the cluster and manually changing it back to the required number, when the nodes need to be started. When the node count is made 0, the original nodes are deleted from the cluster and new nodes are created when nodes are started. Is there any way in which nodes can just be shut down through the scripts(like normal GCP instances) instead of deleting and re-creating? There is a functionality in AWS autoscaling group where the suspend settings can be changed. So is there anything similar in GCP as well?
-
Could you share more information about your script and autoscale (YAMLs, commands)? Are you using command like `gcloud container clusters resize cluster-name --node-pool pool-name --num-nodes num-nodes`,or it's more advanced? As per https://cloud.google.com/kubernetes-engine/docs/how-to/resizing-a-cluster#decrease_with_drain : Warning: Do not use both Cluster Autoscaler and manual resize commands simultaneously on a node pool, as this can cause interactions that result in unstable and/or incorrect node pool size. – PjoterS Apr 14 '20 at 19:17
-
Yeah, we are using gcloud commands to resize the pool to 0 as of now. And it looks like nodes cant be shut down as such in GCP, unline AWS. We are going with this approcah only as of now. Thanks. – Meghana B Srinath Apr 15 '20 at 05:18
-
Could you provide more information. What would be the use case? You want to have Cluster Autoscaler since 06:00 till 22:00 and then use script to change node number to 0 and next create new nodes and turn on cluster autoscaler again? What do you mean by without deleting them? You would like to keep data which was stored in the node? – PjoterS Apr 16 '20 at 15:35
1 Answers
When you are using GKE Cluster Autoscaler you will change nodes number specified in your CA configuration based on some metrics.
Cluster autoscaler works on a per-node pool basis. When you configure a node pool with cluster autoscaler, you specify a minimum and maximum size for the node pool. Cluster autoscaler increases or decreases the size of the node pool automatically, based on the resource requests (rather than actual resource utilization) of Pods running on that node pool's nodes. It periodically checks the status of Pods and nodes, and takes action
As was mentioned in GKE docs - Decreasing the size of your cluster.
Do not use both Cluster Autoscaler and manual resize commands simultaneously on a node pool, as this can cause interactions that result in unstable and/or incorrect node pool size.
In this situation when you manually change node pool to lower than it was specified in configuration, Cluster autoscaler mechanism worked as designed and change number of nodes to minimal.
As workaround for this situation you could consider to turn off Cluster Autoscaler before manually resizing the node pool. When you would need autoscaling of the nodes you would need to run CA again.
Just for additional information, when you are using Cluster Autoscaling you can also use Horizontal Pod Autoscaler.
- 615
- 3
- 11