I'm running an EKS cluster on AWS and I've been having periodic outages recently. I tracked down the issue to duplicate network interfaces on one of my nodes, which I believe are created by the AWS CNI plugin.
When the node is created, it has 2 NICs. But at some point later in the node's life, a 3rd NIC is created (not by me) which causes strange behavior. It looks like the rp_filter is rejecting packets from the ELB as the SYN packet shows up in the messages log as a martian.
I should add that this only seems to happen for services that are annotated as service.beta.kubernetes.io/external-traffic: OnlyLocal
When I detach the 3rd NIC, everything returns to normal.
My questions are
- Why might the 3rd NIC cause this problem? Is this a bug in CNI?
- How can I workaround this issue?
- Is there some way I can prevent the 3rd NIC from being attached?
- Is there some way to configure CNI so that it attaches the 3rd NIC correctly? Do I have EKS configured incorrectly somehow?
Some evidence:
netstat -s is showing IPReversePathFilter: 666179 and increments by one for every SYN from the ELB exactly.
tail /var/log/messages shows the martian packets like so
Nov 22 18:37:17 ip-192-168-90-83 kernel: IPv4: martian source 192.168.102.150 from 192.168.236.246, on dev eth0
Nov 22 18:37:17 ip-192-168-90-83 kernel: ll header: 00000000: 0e 76 b3 cc 10 f4 0e 6f 51 60 fa be 08 00 .v.....oQ`....
When I set rp_filter=0, then the SYN from the ELB isn't dropped anymore and I see the SYN-ACKs head toward the ELB, but they never make it there.
