OS: Ubuntu 18.04.6 Firewall type: IPtables. UFW is Disabled.
I have Kubernetes cluster with 3 nodes that provide the controlplane and etcd services, named cp01, cp02 and cp03.
I see that etcd traffic from each of the Controlplane/etcd nodes works fine. The etcd application seems to be working, although I noticed that I cannot do some things like etcdctl elect a new leader.
I noticed that sometimes the return traffic is getting blocked. Here's an example log message showing the block:
May 23 09:34:55 cp02 kernel: [1245818.175864] DROP-INPUT: IN=eth2 OUT= MAC=00:50:AA:BB:CC:DD:00:50:AA:BB:CC:11:08:00 SRC=192.168.101.188 DST=192.168.101.189 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=2380 DPT=36532 WINDOW=0 RES=0x00 RST URGP=0
I don't understand why these packets are getting dropped. IPtables has a rule to accept all RELATED,ESTABLISHED traffic. It seems like on occasion, that's not happening and the packets are actually getting dropped.
Here is the relevant section of IPtables. Note the following:
- Lines 1-3 are inserted by Kubernetes.
- Lines 4-7 are standard rules (I believe they are.)
- Lines 22 & 23 will log then drop all packets that didn't match an existing rule
- Line 7 accepts all RELATED,ESTABLISHED traffic-- that is, if the traffic went out through this host it can return to this host since it marked as related or established traffic. However, the log message above suggests this is not happening.
cp03:~ # iptables -t filter -L INPUT --line-numbers -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 2763 407K cali-INPUT all -- any any anywhere anywhere /* cali:Cz_abcdefghijklm */
2 2763 407K KUBE-FIREWALL all -- any any anywhere anywhere
3 32 6559 KUBE-EXTERNAL-SERVICES all -- any any anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
4 1 84 ACCEPT icmp -- any any anywhere anywhere /* 000 accept all icmp */
5 773 112K ACCEPT all -- lo any anywhere anywhere /* 001 accept all to lo interface */
6 0 0 REJECT all -- !lo any anywhere localhost/8 /* 002 reject local traffic not on loopback interface */ reject-with icmp-port-unreachable
7 1958 288K ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED /* 003 accept related established rules */
8 0 0 ACCEPT tcp -- any any admin.example.org anywhere multiport dports ssh /* 101 Allow SSH from management servers from 192.168.100.16 */
9 0 0 ACCEPT udp -- any any cp01.example.org anywhere multiport dports 8472 /* 101 Canal/Flannel VXLAN overlay networking from 192.168.101.188 */
10 0 0 ACCEPT udp -- any any cp02.example.org anywhere multiport dports 8472 /* 101 Canal/Flannel VXLAN overlay networking from 192.168.101.189 */
11 0 0 ACCEPT udp -- any any cp03.example.org anywhere multiport dports 8472 /* 101 Canal/Flannel VXLAN overlay networking from 192.168.101.190 */
12 0 0 ACCEPT tcp -- any any cp01.example.org anywhere multiport dports 6443 /* 101 Kubernetes apiserver from 192.168.101.188 */
13 0 0 ACCEPT tcp -- any any cp02.example.org anywhere multiport dports 6443 /* 101 Kubernetes apiserver from 192.168.101.189 */
14 0 0 ACCEPT tcp -- any any cp03.example.org anywhere multiport dports 6443 /* 101 Kubernetes apiserver from 192.168.101.190 */
15 4 220 ACCEPT tcp -- any any cp01.example.org anywhere multiport dports 2379:2380 /* 101 etcd client requests from 192.168.101.188 */
16 4 220 ACCEPT tcp -- any any cp02.example.org anywhere multiport dports 2379:2380 /* 101 etcd client requests from 192.168.101.189 */
17 0 0 ACCEPT tcp -- any any cp03.example.org anywhere multiport dports 2379:2380 /* 101 etcd client requests from 192.168.101.190 */
18 0 0 ACCEPT tcp -- any any cp01.example.org anywhere multiport dports 10250 /* 101 kubelet API from 192.168.101.188 */
19 0 0 ACCEPT tcp -- any any cp02.example.org anywhere multiport dports 10250 /* 101 kubelet API from 192.168.101.189 */
20 0 0 ACCEPT tcp -- any any cp03.example.org anywhere multiport dports 10250 /* 101 kubelet API from 192.168.101.190 */
21 1 40 LOG all -- any any anywhere anywhere limit: avg 3/min burst 5 /* 998 Log all drops */ LOG level warning prefix "DROP-INPUT: "
22 1 40 DROP all -- any any anywhere anywhere /* 999 drop all other requests */
