Is it possible to set an AWS autoscaling policy for Kubernetes nodes using kops?

Question

I've setup a private K8s cluster using kops on AWS, and I'd like to be able to autoscale the nodes based on CPU use. I've read that this is possible with GCE, but is it possible with AWS?

Answer 1

Yes it is possible, you can do this by using Cluster Autoscaler or CA

As for how to do it with kops. First, you need to edit instance groups and add extra labels.

$ kops edit ig nodes

spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/node-template/label: ""
    kubernetes.io/cluster/<CLUSTER_NAME>: owned

Cluster Autoscaler has its own auto-discovery which is recommended if you have multiple instance groups. With auto-discovery there is no need to set min and max size in two places, and there is no need to change CA config if you add group later.

You should add additional IAM policy rules for nodes:

$ kops edit cluster

spec:
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:DescribeAutoScalingInstances",
            "autoscaling:SetDesiredCapacity",
            "autoscaling:DescribeLaunchConfigurations",
            "autoscaling:DescribeTags",
            "autoscaling:TerminateInstanceInAutoScalingGroup"
          ],
          "Resource": ["*"]
        }
      ]

And apply the configuration:

$ kops update cluster --yes

Now you can install CA, but keep in mind to check with CA version is recommended for Kubernetes version. For this you should check the releases.

Deployment

Cluster Autoscaler is designed to run on Kubernetes master node. This is the default deployment strategy on GCP. It is possible to run a customized deployment of Cluster Autoscaler on worker nodes, but extra care needs to be taken to ensure that Cluster Autoscaler remains up and running. Users can put it into kube-system namespace (Cluster Autoscaler doesn't scale down node with non-mirrored kube-system pods running on them) and set a priorityClassName: system-cluster-critical property on your pod spec (to prevent your pod from being evicted).

Once you have deployed CA, you need to choose the right AWS region.

Now you can choose an expander.

Expanders provide different strategies for selecting the node group to which new nodes will be added. Expanders can be selected by passing the name to the --expander flag, i.e. ./cluster-autoscaler --expander=random

Currently Cluster Autoscaler has 4 expanders:

random - this is the default expander, and should be used when you don't have a particular need for the node groups to scale differently.

most-pods - selects the node group that would be able to schedule the most pods when scaling up. This is useful when you are using nodeSelector to make sure certain pods land on certain nodes. Note that this won't cause the autoscaler to select bigger nodes vs. smaller, as it can add multiple smaller nodes at once.

least-waste - selects the node group that will have the least idle CPU (if tied, unused memory) after scale-up. This is useful when you have different classes of nodes, for example, high CPU or high memory nodes, and only want to expand those when there are pending pods that need a lot of those resources.

price - select the node group that will cost the least and, at the same time, whose machines would match the cluster size. This expander is described in more details HERE. Currently it works only for GCE and GKE (patches welcome.)

Cluster Autoscaler does support following providers: GCE ,GKE ,AWS ,Azure, Alibaba Cloud

Hope this will be helpful.