We are seeing the COREDNS pod is unable to lookup for primary DNS server which is resulting in readtimeout. Connectivity to primary DNS server work fine. Tried deploying Flannel/Calico/CANAL CNI but we still see the problem persist. nameresolution to kubdns and kubeapi server is working when i try to do a nslookup from a pod. We see ISTIO pod are unable to reach the below API server DNS address due to which pods are going to CrashLoopBackOff.
021-05-17T05:31:35.318792Z error k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Secret: Get https://10.96.0.1:443/api/v1/namespaces/istio-system/secrets?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
Core DNS readiness/liveness probe failed
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
kubectl logs coredns-775956cf78-zn9fr -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 25c756253dfbdb23a23bab618a74ca05
CoreDNS-1.6.7
linux/amd64, go1.13.15, da7f65b
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0517 06:22:43.839379 1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2021-05-17 06:22:13.838597206 +0000 UTC m=+0.029441714) (total time: 30.000641853s):
Trace[2019727887]: [30.000641853s] [30.000641853s] END
E0517 06:22:43.839413 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0517 06:22:43.839462 1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2021-05-17 06:22:13.83850082 +0000 UTC m=+0.029345241) (total time: 30.000767756s):
Trace[1427131847]: [30.000767756s] [30.000767756s] END
cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true