How to have different NodeSelector config on pods within a StatefulSet

Question

I have an Openshift cluster which spans multiple datacenters. Each data center has a zone configured such that a NodeSelector can be used to choose which data center a pod gets allocated in. I also have a StatefulSet which configures 10 pods.

I would like to configure the StatefulSet such that there is typically an even distribution of pods across the data centers. This can't be a strictly defined configuration though, as in the event of a disaster in one data center, I would like the pods to be allocated to the remaining data center.

Is this possible without implementing a custom controller?

score 0 · Answer 1 · answered Oct 10 '19 at 14:34

What you're looking for is affinities / anti-affinities [1], [2]

You should be able to add, in your StatefulSet definition, some mention of an anti-affinity rule:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: openldap
spec:
  selector:
    matchLabels:
      name: openldap
  replicas: 3
  template:
    metadata:
      labels:
        name: openldap
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: name
                  operator: In
                  values: [ openldap ]
              topologyKey: kubernetes.io/hostname
            weight: 42
...

The above would ensure that the scheduler won't place two Pods with a name=openldap label, onto machines sharing the same topologyKey.

While common samples would refer to kubernetes.io/hostname as a topology key, you could very well rely on some custom label. Say your nodes from DC-A have a label dc=a, and DC-B dc=b, then you could use dc as your topologyKey.

Note the above sample expresses a preference, rather than a mandatory rule. Instead of preferredDuringSchedulingIgnoredDuringExecution (or preferredDuringSchedulingRequiredDuringExecution), you could use requiredDuringSchedulingIgnoredDuringExecution or requiredDuringSchedulingRequiredDuringExecution -- whith a slightly different syntax, you won't be able to set weights on mandatory placement rules, ...

[1] https://docs.openshift.com/container-platform/3.11/admin_guide/scheduling/pod_affinity.html#admin-guide-sched-affinity-config-pod-pref

[2] https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

How to have different NodeSelector config on pods within a StatefulSet

1 Answers1