Wrapping Kubernetes with Wireguard

Question

I've a scenario with many different nodes. Some have public IPv4, some have IPv6, some are dual stack. So I've created a wireguard network (10.11.12.0/24), so that any peer can reach any other inside a private network regarding of IP-stack and location. I'd like to build a Kubernetes over this wireguard networks.

I've build a small test cluster ...

node   public ip        wireguard ip
vm1    192.168.10.10    10.11.12.10
vm2    192.168.10.11    10.11.12.11
vm3    192.168.10.12    10.11.12.12
...

... in my local playground with kubeadm 1.23.5 based on docker.io (debian default):

vm01> kubeadm init --apiserver-advertise-address=10.11.12.10 --pod-network-cidr=10.20.0.0/16
vm01> kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml
vm01> kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
...
all nodes> kubeadm join 10.11.12.10:6443 --token ... --discovery-token-ca-cert-hash sha256:...
...
vm01> helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace

When I look from vm1 to vm2 via tcpdump -n host 192.168.10.11, I can see only traffic thru wireguard UDP packets. Fine...

Then I've defined a simple Deployment, a Service, a ClusterIP, an Ingress and it's deployed

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: auth0blog/kubernetes-tutorial
        ports:
          - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-tutorial-cluster-ip
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app: kubernetes-tutorial-deployment
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-tutorial-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: test.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-tutorial-cluster-ip
            port:
              number: 80

When I check with the browser, I'm getting response. But...

The response is very slow (I can confirm via a simple curl, it takes 10-20sec for the service to respond to a single request - that is strange slow for such a simple deployment.

When I look via tcpdump I see traffic outside the wireguard network, which is much more strange.

18:39:18.341836 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 128
18:39:18.344382 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 176
18:39:18.344563 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 1452
18:39:18.344571 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 1452
18:39:18.344572 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 1452
18:39:18.344573 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 96
18:39:18.344711 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:18.344711 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:18.344711 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:20.566833 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 128
18:39:20.566833 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 592
18:39:20.567003 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 96
18:39:20.570978 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 128
18:39:20.571309 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:20.572538 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 176
18:39:20.572566 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 592
18:39:20.572764 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:20.572764 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:23.540401 ARP, Request who-has 192.168.10.11 tell 192.168.10.10, length 28
18:39:23.540646 ARP, Reply 192.168.10.11 is-at 7a:1d:d9:fc:fa:eb, length 28
18:39:23.608703 IP 192.168.10.10.42274 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.5.55222 > 10.20.4.2.3000: Flags [S], seq 3011291899, win 64860, options [mss 1410,sackOK,TS val 2531657982 ecr 0,nop,wscale 7], length 0
18:39:23.609071 IP 192.168.10.11.59205 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.3000 > 10.20.0.5.55222: Flags [S.], seq 1444377380, ack 3011291900, win 64308, options [mss 1410,sackOK,TS val 2546470618 ecr 2531657982,nop,wscale 7], length 0
18:39:23.609112 IP 192.168.10.10.42274 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.5.55222 > 10.20.4.2.3000: Flags [.], ack 1, win 507, options [nop,nop,TS val 2531657983 ecr 2546470618], length 0
18:39:23.609140 IP 192.168.10.10.42274 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.5.55222 > 10.20.4.2.3000: Flags [P.], seq 1:749, ack 1, win 507, options [nop,nop,TS val 2531657983 ecr 2546470618], length 748
18:39:23.609370 IP 192.168.10.11.59205 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.3000 > 10.20.0.5.55222: Flags [.], ack 749, win 501, options [nop,nop,TS val 2546470618 ecr 2531657983], length 0
18:39:23.610441 IP 192.168.10.11.36593 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.33592 > 10.20.0.2.53: 53349+ A? test.example.com.default.svc.cluster.local. (60)
18:39:23.610713 IP 192.168.10.10.58646 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.2.53 > 10.20.4.2.33592: 53349 NXDomain*- 0/1/0 (153)
18:39:23.611018 IP 192.168.10.11.32846 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.40077 > 10.20.0.2.53: 57710+ A? test.example.com.svc.cluster.local. (52)
18:39:23.611134 IP 192.168.10.10.41066 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.2.53 > 10.20.4.2.40077: 57710 NXDomain*- 0/1/0 (145)
18:39:23.611427 IP 192.168.10.11.51546 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.59046 > 10.20.0.3.53: 18849+ A? test.example.com.cluster.local. (48)
18:39:23.611567 IP 192.168.10.10.39789 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.3.53 > 10.20.4.2.59046: 18849 NXDomain*- 0/1/0 (141)
18:39:23.611831 IP 192.168.10.11.50067 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.34442 > 10.20.0.3.53: 49768+ A? test.example.com.sol.system. (45)
18:39:25.329861 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 208
18:39:25.330138 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:25.613106 IP 192.168.10.10.52981 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.3.53 > 10.20.4.2.34442: 49768 ServFail- 0/0/0 (45)
18:39:25.613542 IP 192.168.10.11.33388 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.59146 > 10.20.0.3.53: 49768+ A? test.example.com.sol.system. (45)
18:39:27.021478 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 224
18:39:27.021876 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:27.614533 IP 192.168.10.10.48157 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.3.53 > 10.20.4.2.59146: 49768 ServFail- 0/0/0 (45)
18:39:27.614906 IP 192.168.10.11.52721 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.33596 > 10.20.0.3.53: 32196+ A? test.example.com. (34)
18:39:28.500696 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 128
18:39:28.503146 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 256
18:39:28.503158 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 1452
18:39:28.503159 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 1452
18:39:28.503161 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 1452
18:39:28.503162 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 96
18:39:28.503453 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:28.503453 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:28.503453 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:28.503453 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:28.503453 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:28.627012 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 128
18:39:28.627292 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 128
18:39:28.627636 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:29.615282 IP 192.168.10.10.52590 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.3.53 > 10.20.4.2.33596: 32196 ServFail- 0/0/0 (34)
18:39:29.615672 IP 192.168.10.11.37175 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.50957 > 10.20.0.3.53: 32196+ A? test.example.com. (34)
18:39:29.877400 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 192
18:39:29.877722 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:30.898243 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 128
18:39:30.898243 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 592
18:39:30.898330 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 96
18:39:30.902126 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 128
18:39:30.902362 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:30.903556 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 176
18:39:30.903696 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 592
18:39:30.904023 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:30.904023 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96
18:39:31.617136 IP 192.168.10.10.38253 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.3.53 > 10.20.4.2.50957: 32196 ServFail- 0/0/0 (34)
18:39:31.619778 IP 192.168.10.11.59205 > 192.168.10.10.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.4.2.3000 > 10.20.0.5.55222: Flags [P.], seq 1:114, ack 749, win 501, options [nop,nop,TS val 2546478629 ecr 2531657983], length 113
18:39:31.619911 IP 192.168.10.10.42274 > 192.168.10.11.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.20.0.5.55222 > 10.20.4.2.3000: Flags [.], ack 114, win 507, options [nop,nop,TS val 2531665993 ecr 2546478629], length 0
18:39:33.434382 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 128
18:39:33.434488 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 96
18:39:33.434537 IP 192.168.10.10.59120 > 192.168.10.11.59120: UDP, length 128
18:39:33.434860 IP 192.168.10.11.59120 > 192.168.10.10.59120: UDP, length 96

What is the possible reason, why the response is so slow in a LAN network. Is it because of wrong routing to "public" IPs instead of using the wireguard IP? Is it possible to configure the Kubernetes to use the wireguard address for port 8472?

score 0 · Answer 1 · answered Mar 31 '22 at 11:53

Ok, I found the solution.

I've tested the cluster installation without Wireguard. And in that case the application auth0blog/kubernetes-tutorial also hangs multiple seconds. So I switched to a simple nginx http service and that responses in an expected time.
The port 8472 is used by flannel. There are issues on Github (from 2018...) showing that it uses by default the external network interface. It must be configured to use the wireguard interface. See https://github.com/rancher/rancher/issues/15133 and https://github.com/rancher/rancher/issues/14721#issuecomment-417913067

So - actually this is more or less a duplicate of https://stackoverflow.com/questions/66449289/is-there-any-way-to-bind-k3s-flannel-to-another-interface

And because I'm new to Kubernetes I wondered how to change that value in my given commands (see question). The documentation only talks about the parameters but isn't saying how and where to set it. I've looked around and found https://stackoverflow.com/questions/47845739/configuring-flannel-to-use-a-non-default-interface-in-kubernetes

There you can see, that the solution is to download the flannel.yml and add the parameter --iface=wg_k8s to that file in the correct location. In the current version (2022) it is in line 200+.

Wrapping Kubernetes with Wireguard

1 Answers1