I setup a Kubernetes cluster using Kubernetes the Hard Way tutorial, and the connection is hanging whenever a Pod connects to another Pod on the same node through a ClusterIP (hairpin traffic).
If I access the pods directly, without going through the ClusterIP, everything works fine.
So, visually, this doesn't work:
PodA -> ServiceA ClusterIP -> PodA
PodA -> ServiceB ClusterIP -> PodB on same node
However, this works 100% great, as does any Pod contacting another Pod directly by it's IP:
PodA -> ServiceB ClusterIP -> PodB on other node
I found Kubernetes Documentation about debugging services and went through it, and everything seems fine, up to the section A Pod can't reach itself via Service VIP.
I see rules added for my services in iptables-save
output (and I confirmed using iptables
mode):
-A KUBE-SERVICES ! -s 10.200.0.0/16 -d 10.32.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.32.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.200.0.0/16 -d 10.32.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.32.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
I can see from the kubelet's logs promiscuous-bridge
hairpin mode flag:
kubelet[12496]: I1204 04:13:29.761707 12496 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"
I don't see logs such as Hairpin mode set to "promiscuous-bridge"
specifically confirming the mode, so I set it explicitly in the kubelet-config.yml also
Also, I edited the CNI plugin to add promiscMode: true
(docs), and I see PROMISC
on the cnio0
interface:
cnio0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
At this point I think either 1) this tutorial doesn't have hairpin traffic working or 2) I screwed up something obscure that's breaking this, but I can't figure out what it is!
Since this Kubernetes the Hard Way tutorial is known to be the canonical setup reference, I'm doubting it would be #1... anyone have other suggestions to determine #2?