Network Routing Policy for Kubernetes Service IPs published via MetalLB Loadbalancer

Question

we have a routing problem for Kubernetes Services which are published over MetalLB Loadbalancer on a Loadbalancer-LAN attached to the Kubernetes Worker Nodes.

Working with only default/main routing table and static routing entries is working fine.

But when we work with a routing policy so that Kubernetes Services use a different Default Gateway than the Host’s Default Gateway and static Routes, we have routing issues:

The routing policy configured on the kubernetes worker nodes seems to be not valid and not working for the IP addresses which are published via MetalLB Loadbalancer.

We need support here to understand the specific routing requirements for MetalLB (in Layer-2 Mode).

Environment:

Kubernetes Cluster, each Node with three NICs:

User-LAN (“ens192”)

Service-LAN („ens160“)

Loadbalancer-LAN (“ens224”) (for k8s Service IPs on MetalLB Loadbalancer)

CNI Plugin: Flannel

MetalLB configured in ARP/Layer-2 Mode.

“Main/Default” Routing Table (ip route list)

Default Gateway: On User-LAN

Some static Routes: On Service-LAN

For the Routing Policy (different Default Gateway for k8s Services) requirement, we created an additional Routing Table:

cat /etc/iproute2/rt_tables <snip> 100 ens224defaultroute

Additional routes and rules configured for routing table “ens224defaultroute”:

ip route add <LB-LAN-NET/24> dev ens224 table ens224defaultroute ip route add default via <LB-LAN-NET-GWIP> dev ens224 table ens224defaultroute ip rule add from <LB-LAN-NET/24> table ens224defaultroute ip rule add to <LB-LAN-NET/24> table ens224defaultroute

Check Routing information:

ip rule list table ens224defaultroute 32764: from all to <LB-LAN-NET/24> lookup ens224defaultroute 32765: from <LB-LAN-NET/24> lookup ens224defaultroute

ip route list table ens224defaultroute default via <LB-LAN-NET-GWIP> dev ens224 <LB-LAN-NET/24> dev ens224 scope link

ip route get <ANY-TARGET-IP-IN-REMOTE-NET> from <LB-LAN-HOST-IP> <ANY-TARGET-IP-IN-REMOTE-NET> from <LB-LAN-HOST-IP> via <LB-LAN-NET-GWIP> dev ens224 table ens224defaultroute uid 0 cache

Eveything works so far, e.g. we can reach the Hosts IP address from Remote Client e.g. .

But we can’t access another IP address within the same LAN when it was published by the MetalLB Loadbalancer Service in Kubernetes.

It seems that the Routing Policy is not used for IP addresses, that are published by MetalLB Loadbalancer for the k8s Services IPs.

The following command is using the IP address from a Kubernetes Service which is published via MetalLB Loadbalancer:

ip route get <ANY-TARGET-IP-IN-REMOTE-NET> from <LB-LAN-k8s_SVC_LB-IP> RTNETLINK answers: Invalid argument

Result:

• From Remote Client, the Loadbalancer published IPs are not accessible.

• From Remote Client, only the Nodes’s IP address is accessible.

Thank you very much in advance!