0

So I've been working on a project that involves a cluster spanning a cloud provider VM and a local VM (details here and here) connected through a Wireguard VPN with both the control plane and intra-cluster data plane running through the VPN. Intra-cluster network is Flannel and I am using Multus to get a second interface for pods on the local VM. The two nodes are not in the same routing domain and have no connectivity except through their Internet addresses and through the Wireguard VPN, which is a sort of atypical case for a K8s cluster.

I've been trying out different service types and I can't find any difference between NodePort and ClusterIP service types, except the NodePort service gets a port assigned above 30000. According to the documentation ClusterIP services are for intra-cluster communication only and NodePort services from outside the cluster, so I would expect that an attempt to access a ClusterIP service from outside the cluster would fail. But I've found that I can access both types of services from outside the cluster. Furthermore, both types of services are discoverable through CoreDNS from outside the cluster, if I use dnsmasq and configure DNS resolution to forward names in the cluster.local domain to the kube-dns service IP address.

Here's a table of results based on using curl to access the services from a shell window running outside the cluster (N = curl hangs, NA = not applicable, Y = curl returns the web page):

Service Type ClusterIP NodePort
service-address:node port NA N
service-address:target port Y Y
endpoint-address:endpoint port Y Y
service name:target port Y Y

I don't know why the first row doesn't work, I assume it is due to my nodes not being in the same routing domain.

Furthermore, if I use traceroute to try to find out how routing is occuring from the local VM, I get a trace that looks like this:

  1   192.168.0.1  2.159ms  0.003ms  4.331ms 
  2   192.168.1.254  2.748ms  0.004ms  2.100ms 
  3   75.59.236.1  19.179ms  0.004ms  21.638ms 
  4   *  *  * 
  5   *  *  * 
  6   *  *  * 
  7   *  *  * 
  8   71.148.148.109  19.746ms !N  0.006ms !N  21.813ms !N

The first hop is my Wifi access point router, the second is my Internet router, the other two are on AT&T's network. None of the interfaces on my Ubuntu 20.04 VM are configured to route the IP addresses being assigned to services, I assume the addresses are somehow being handled by the flannel.1 or cni0 interfaces.

For reference, I am using kubeadm v1.24.3, kubectl version 1.24.3, kubelet version 1.24.3, Flannel version 0.18.1, CNI version 0.4.0 Multus version 3.9

Unfortuntely, I don't know enough about Linux networking to figure out what is going on, but maybe someone could explain?

Thanx.

            jak
JAK
  • 1
  • From your test results, it sounds like your VPN is accepting/routing traffic to your SDN subnets. This should not be the case: SDN communications should encapsulated, underlying network "should not" route SDN subnets. Yet if one of your router sends those subnets to a working k8s node, then you may enter the SDN through that node. Regarding the "service-address:node port" test, it is normal you would not get an answer: you should test "node-address:node-port" instead. node-port is not meant to be reachable through an SDN address. – SYN Jul 23 '22 at 08:05

0 Answers0