2

I deployed a Kubernetes cluster using kubeadm.

$ kubectl get nodes
NAME                                          STATUS     ROLES                  AGE   VERSION
ip-172-31-33-9.us-east-2.compute.internal     NotReady   <none>                 48m   v1.22.2
ip-172-31-46-53.us-east-2.compute.internal    NotReady   control-plane,master   49m   v1.22.2
ip-172-31-47-245.us-east-2.compute.internal   NotReady   <none>                 6s    v1.22.2

The nodes are NotReady because I haven't installed a networking plugin. I want to use keep it simple and use kubenet. The instructions say:

Kubelet must be run with the --network-plugin=kubenet argument to enable the plugin

but I can't figure out how to do that. I have tried adding it to various configuration files, such as /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf and /etc/sysconfig/kubelet, and restaring kubelet, but none of it works.

How do I use kubenet with kubeadm?

Wytrzymały Wiktor
  • 731
  • 3
  • 10
timkay
  • 129
  • 2
  • Hi timkay welcome to S.F. You have said you "tried it" but "none works" but yet you shared none of your attempts with us in order to gauge what you're already tried and why it might not be working. Please read the [how to ask](https://serverfault.com/help/how-to-ask) page. In the spirit of trying to be a little helpful, did you remember to run `systemctl daemon-reload` in between edits? systemd does not use the latest values on disk -- it must be forced to re-read them. Good luck! – mdaniel Oct 06 '21 at 03:06
  • As I said, the docs say to add `--network-plugin=kubenet` to the kubelet command line, but I never have a kubelet command line, as kubelet gets started by kubeadm. Thus, I don't have any attempts to share with you. – timkay Oct 07 '21 at 19:15

2 Answers2

1

You can use the /etc/default/kubelet file to override kubelet arguments. As can be found in the 10-kubeadm.conf file:

...
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
...

I'll show you how it works.

First, I set up the Kubernetes control plane:

# kubeadm init
[init] Using Kubernetes version: v1.22.2
...

# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config

After successful initialization, we can check the value of the --network-plugin argument that kubelet is currently using and the status of the coredns Pods:

# ps aux | grep "kubelet" | grep "network-plugin"
root     27488  5.1  3.6 1816612 145808 ?      Ssl  10:42   0:01 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.5

# kubectl get pod -n kube-system | grep "coredns"
coredns-78fcd69978-bbc52          0/1     Pending   0          2m26s
coredns-78fcd69978-fdcv9          0/1     Pending   0          2m26s

From the above output, we can see that --network-plugin=cni and coredns Pods are in the Pending state.

Let's create the /etc/default/kubelet file and write the kubelet arguments there:
NOTE: You may need to customize the --pod-cidr and --pod-infra-container-image to suit your needs.

# touch /etc/default/kubelet

# echo 'KUBELET_KUBEADM_ARGS="--network-plugin=kubenet --pod-cidr=10.20.0.0/24 --pod-infra-container-image=k8s.gcr.io/pause:3.5"' > /etc/default/kubelet

# cat /etc/default/kubelet
KUBELET_KUBEADM_ARGS="--network-plugin=kubenet --pod-cidr=10.20.0.0/24 --pod-infra-container-image=k8s.gcr.io/pause:3.5"

Then we need to reload systemd manager configuration and restart kubelet:

# systemctl daemon-reload
# systemctl restart kubelet

Finally, we can check if it works as expected:

# ps aux | grep "kubelet" | grep "network-plugin"
root     27841  6.9  3.5 1890600 143760 ?      Ssl  10:47   0:01 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=kubenet --pod-cidr=10.20.0.0/24 --pod-infra-container-image=k8s.gcr.io/pause:3.5
   
# kubectl get pod -n kube-system | grep "coredns"
coredns-78fcd69978-bbc52          1/1     Running   0          6m51s
coredns-78fcd69978-fdcv9          1/1     Running   0          6m51s   

# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
kmaster   Ready    control-plane,master   15m   v1.22.2

You can follow these steps for all of your nodes.

Additionally, please remember that (more information can be found in the kubenet documentation):

Kubenet is a very basic, simple network plugin, on Linux only. It does not, of itself, implement more advanced features like cross-node networking or network policy. It is typically used together with a cloud provider that sets up routing rules for communication between nodes, or in single-node environments.

matt_j
  • 350
  • 2
  • 6
  • Hello @timkay and welcome to ServerFault! Please remember to [react to answers for your questions](https://stackoverflow.com/help/someone-answers). That way we know if the answers were helpful and other community members could also benefit from them. Try to [accept answer](https://stackoverflow.com/help/accepted-answer) that is the final solution for your issue, upvote answers that are helpful and comment on those which could be improved or require additional attention. Enjoy your stay! – Wytrzymały Wiktor Oct 07 '21 at 07:58
  • Thank you, @matt_j. I followed your instructions, which I summarized this way: 1. Install as before, using kubeadm. 2. Create a file /etc/default/kubelet, with extra kubelet parameters. 3. Restart kubelet with systemctl daemon-reload && systemctl restart kubelet. I did that, and the command line parameters for kubelet didn't change. The env variable KUBELET_KUBEADM_ARGS would tell me that it's used by kubeadm to start kubelet, but we don't run kubeadm at all after making changes as per your instructions. – timkay Oct 08 '21 at 00:13
  • How did you check that the kubelet parameters did not change ? Could you please use this command: `ps aux | grep "kubelet" | grep "network-plugin"` ? – matt_j Oct 08 '21 at 10:06
  • Hello @timkay. Any updates? – Wytrzymały Wiktor Oct 12 '21 at 11:12
  • I gave up on Kubernetes and wrote my own replacement. – timkay Jan 09 '22 at 05:54
  • No need to run `systemctl daemon-reload`, since no `systemd` configuration changed that would need to be reloaded. I also don't like using `/etc/default` for custom configurations. Otherwise very helpful, thanks! – Minding Apr 16 '22 at 18:44
  • Actually `kubeadm` ["does not support kubenet"](https://unofficial-kubernetes.readthedocs.io/en/latest/getting-started-guides/kubeadm/) according to this wiki. – Minding Apr 16 '22 at 22:53
0

kubeadm gives the possibility to use a --config yaml file that is documented here. In this file you can use the cli parameters from kubelet as extra args like --network-plugin=kubelet as network-plugin: kubelet:

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
...
nodeRegistration:
...
  kubeletExtraArgs:
    ...
    network-plugin: kubenet
    pod-cidr: 10.20.0.0/24
Oskar
  • 1
  • 1
  • Kubernetes is so messed up. I gave up on it a long time ago and wrote my own replacement, which is only a few hundred lines of code. I am quite disappointed that the industry seems to be a pyramid scheme. People write complex software that only highly trained people can use. Where is the craftmanship? As Elon says, "no part is the best part." Stop fixing complexity problems by making everything more complex. – timkay May 28 '22 at 16:03
  • @timkay True words, I can fully understand your statement :) – Oskar May 30 '22 at 11:31