How does Azure Loadbalancer work with Azure AKS

Question

I recently tried to figure out how an Azure Loadbalancer in front of an Azure AKS cluster actually routes traffic to the cluster nodes.

Our (quite basic) setup:

• AKS cluster version 1.23.x with the default Azure Loadbalancer in front of it
• Traefik Ingress controller deployed with service type Loadbalancer

Maybe someone can answer me the following questions:

1. The Ingress backend pool contains ALL cluster nodes, not just the ones where the ingress pods are running (also they only show the pod's private IP). If I understand the docs (https://docs.microsoft.com/en-us/azure/aks/concepts-network#services) correctly, the loadbalancer distributes traffic between ALL cluster nodes, and from there, it is forwarded to the nodes which actually run the Ingress controller pods. Potentially, this adds unnessesary hops and wastes bandwidth due to the forwarding. Is my understanding here correct?

2. I have troubles to understand how traffic is forwarded from the loadbalancer to the cluster nodes. The backend pool contains entries with the nodes private IPs (no k8s service IPs or pod IPs), and the loadbalancing rule shows port 443 as source and destination port. However, there is no process listening on port 443 on any of the cluster nodes. Can someone explain to me (or point me to the docs I failed to find) that explain how this works?

Answer 1

Potentially, this adds unnessesary hops and wastes bandwidth due to the forwarding. Is my understanding here correct?

That's one way to look at it, the other way is that if the load balancer were only pointed at the Nodes containing "currently Ready" pods, it would need to be updated when one of them goes un-Ready, or that Node dies for any reason. It has never been my experience that cloud changes happen quickly, where as for most private networking setups, intra-cluster traffic is basically free

Can someone explain to me (or point me to the docs I failed to find) that explain how this works?

I can't speak to any of the Azure terms you used, but the underlying mechanism kubernetes uses is a NodePort through which it picks a (mostly random, although you can pick one if you have a strong opinion about it) free port on all the Nodes and then points the LoadBalancer's traffic at that port, which internal to the cluster arrives on the matching traffic port of the Service and

You are actually welcome to do that process "manually" if you had some specific reason for it (such as using an unsupported cloud provider that does not understand provisioning the fronting LoadBalancer) since aside from keeping all Nodes who are reporting Ready assigned to the LoadBalancer, there's no further in-cluster changes from the point the NodePort is assigned

small asterisk: I just learned that as of 1.24, the NodePort assignment can be disabled, and I'd presume that's a concession to the VPC CSI provider used by AWS, which enjoys provisioning actual ENIs in your VPC for each Pod address, so I guess they can skip the network hop in those cases, but (in general) that NodePort fronted by LoadBalancers is a good mental model to have and is also likely going to apply to easily 99% of the clusters in the wild