1

Resource Management for Pods and Containers describes how to set resource requets and limits for "regular" pods in Kubernetes. Is there a supported/reccomended way to set these limits for control plane components such as kube-apiserver?

Things I considered:

  • Modifying static manifests, e.g. in /etc/kubernetes/manifests/kube-apiserver.yaml. This could work but it will be overwritten by kubeadm during next upgrade.
  • Setting kube-reserved or system-reserved flags. This could work too however again - they are defined in just one ConfigMap (e.g. kubelet-config-1.21) and will be overwritten by kubeadm during node upgrade. The same limits will apply to control plane nodes and worker nodes and I don't want that.

I can overcome this with something like ansible but then ansible will be "figting" with kubeadm and I'd like to avoid that.

What problem am I trying to solve?

I have a small homelab kubernetes installation. I'd like to allow running regular pods on control plane node(s), but I'd like to be able to reserve some resources (primarily memory) for control plane components. I.e. I'd like to be able to set requests on things like kube-apiserver so that scheduler knows not to put any other pods (they will also have appropriate requests) in its place.

rvs
  • 4,027
  • 1
  • 25
  • 30

1 Answers1

1

Is there a supported/reccomended way to set these limits for control plane components such as kube-apiserver?

Yes, you can use kubeadm init with patches command line flag. Look at this github page. The documentation of this thing could be also interested. See also official documentation: Customizing the control plane with patches:

Kubeadm allows you to pass a directory with patch files to InitConfiguration and JoinConfiguration on individual nodes. These patches can be used as the last customization step before the control plane component manifests are written to disk.

You can pass this file to kubeadm init with --config <YOUR CONFIG YAML>:

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
patches:
  directory: /home/user/somedir

Here's example how to set resources on kube-apiserver:

Create kube-apiserver.yaml file in some directory (e.g. /home/user/patches) with following contents:

spec:
  containers:
    - name: kube-apiserver
      resources:
        requests:
          memory: 512Mi
        limits:
          memory: 1024Mi

Then use --patches flag every time during node upgrade: use kubeadm upgrade node --patches /home/user/patches/ or kubeadm upgrade apply v1.22.4 --patches /home/user/patches/

The other option will be supply extra flags to control-plane components. For this, check this guide: Customizing the control plane with flags in ClusterConfiguration:

The kubeadm ClusterConfiguration object exposes a way for users to override the default flags passed to control plane components such as the APIServer, ControllerManager, Scheduler and Etcd. The components are defined using the following structures:

  • apiServer
  • controllerManager
  • scheduler
  • etcd
rvs
  • 4,027
  • 1
  • 25
  • 30
Mikołaj Głodziak
  • 323
  • 1
  • 10
  • 1
    Thank you, the patches route works. I've updated your answer with a more specific example of how I did that. Now I can use ansible to maintain different files in patches directory (if needed) and remember to use --patches flag when doing init or upgrades. – rvs Jan 13 '22 at 10:25