How to configure Kubernetes with multiply NIC / VLAN on vSphere?

Question

Playing with provisioning of Kubernetes cluster on vSphere using Kubespray and Terraform to automate deployment process (no problems with that).

My goal is to have ability to connect Pods to different VLANs (or Port Groups), so then each app will run in it's own network like my VMs do.

I think there is 2 approaches: 1. Create 'All VLANs Port Group (0-4095)' and manage VLANs inside K8S node OS 2. Attach multiply NICs to K8S node VMs each of them will be connected to different VLAN

After googling for weeks I'm still can't find solution for similar requirements.

So my problem is that I'm stuck trying to figure out how to build K8S cluster this way and in the same time I'm not completely sure that I'm on the right way at all.

Please help me before I'm go mad!

you are probably on the wrong track. in general, it is not recommended to have multiple interfaces on a single pod. you can create multiple networks inside the cluster (that work as overlay networks across several compute nodes, too!). you can also decide how to expose certain services through ingress controllers. messing with the internal networking inside the cluster outside the standard functionality of k8s is going to end in tears. — rmalchow, Jul 22 '19 at 13:30
another things that isn't clear ist: what exactly are you tying to achieve? as in, what would be the purpose? — rmalchow, Jul 22 '19 at 13:31
@rmalchow >> you are probably on the wrong track - thanks for clarification! — tiv, Jul 22 '19 at 15:06
@rmalchow >> what exactly are you tying to achieve - I was trying integrate Kubernetes cluster into existing VMs \ Network infrastructure. now I have many VLANs for envs, infrastructure, etc. as example: I have network named "app1" here lives all VMs running app1, if I want to migrate part of app1 to K8S I want to preserve this approach for some reasons like firewall rules. so in one word - I want to minimize infrastructure changes during K8S integration. — tiv, Jul 22 '19 at 15:12
the separation you did with the VMs, you probably did that not just for the fun of it, but to achieve a certain goal. this goal can probably be achieved in k8s as well, albeit in a completely different way. — rmalchow, Jul 23 '19 at 08:09

score 1 · Answer 1 · edited Dec 12 '21 at 13:09

This can be achieved with multus-cni plugin. It creates NetworkAttachmentDefinition custom resource, where you can specify network interface, i.e.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.200",
        "rangeEnd": "192.168.1.216",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
      }
    }'

And then, you attach this configuration to the pod:

apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf

Furthermore, you can add more interfaces to a pod by creating more custom resources and then referring to them in pod’s annotation

More details.

How to configure Kubernetes with multiply NIC / VLAN on vSphere?

1 Answers1