2

I'm trying to write an autoscaling configuration for ingress-nginx, deployed via helm chart.

my goals are:

  • 3 minimum replicas (because I have 3 nodes minimum)
  • ensure only one nginx per node, but:
  • be elastic, if autoscale says we need 4 nginx allow for a node in the cluster to have 2
  • if a fourth node gets added, ensure a new nginx gets spawned

https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L326 https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L343 https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L256

I tried playing with the below settings and a combination of them but there's always something off, for example right now I have a fourth nginx that want to spawn for some reason and it can't because of the antiaffinity rule.

Can anyone share some ideas how to achieve that?

  • always one nginx per node, if a new node gets created, a new nginx gets created
  • preserve autoscaling, if hpa wants to spawn a fourth nginx on a 3 node cluster it should be free to do so
      replicaCount: 3
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - ingress-nginx
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - ingress-nginx
              - key: app.kubernetes.io/component
                operator: In
                values:
                - controller
            topologyKey: "kubernetes.io/hostname"

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/instance: ingress-nginx

      autoscaling:
        enabled: true
        minReplicas: 3
        maxReplicas: 6
        targetCPUUtilizationPercentage: 75
        targetMemoryUtilizationPercentage: 100
John Smith
  • 248
  • 1
  • 3
  • 9
  • If you need only one nginx instance per node, wouldn't it be easier to deploy it as [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) ? – mario Sep 14 '21 at 17:34
  • is it possibile to deploy as daemonset via the helm chart? – John Smith Sep 15 '21 at 18:06
  • 1
    just found out https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L174 – John Smith Sep 15 '21 at 18:12
  • 2
    thanks for the hint, seems working, if you wanna put an official reply I'll accept it – John Smith Sep 15 '21 at 18:15

1 Answers1

1

If you need only one nginx instance per node, wouldn't it be easier to deploy it as DaemonSet ?

As you can read in the official docs:

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

And this seems like the right solution in your use case.

As you rightly suggested in your comment, nginx-ingress official helm chart can be deployed both as a Deployment or as a DaemonSet. It can be done by adjusting its values.yaml file.

mario
  • 525
  • 3
  • 8