Summary
I can only resolve hostnames from and for the default
namespace. Requests from and for other namespaces, like kube-system
, return NXDOMAIN from coredns. Is there some authorisation that needs to be done?
Details
I have set up a learning cluster by following Kubernetes the Hard Way except using VirtualBox instead of GCP. All else appears to work except DNS queries in non-default namespaces. More concretely:
- Works (run from default namespace): kubernetes.default
- Does Not work (run from kube-system namespace, get NXDOMAIN): kubernetes.kube-system
Configuration
I have upgraded to Coredns 1.12 because 1.2.2 failed to resolve anything around 50% of the time. With 1.12 requests resolve 100% of the time (if in the default
namespace).
I had to edit the config (in the configmap) by removing the "loop" and "proxy" (replace proxy with forward) plugins to get it working. My config is as follows:
.:53 {
forward . 192.168.99.254:53 # dnsmasq on host machine
log
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
cache 30
reload
loadbalance
Troubleshooting
Following this guide I see:
The pods are running OK
$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-6f8fb48d88-dz9cd 1/1 Running 0 14m
coredns-6f8fb48d88-s5rvp 1/1 Running 0 14m
Checking the console I see healthy logging
2019-05-24T04:41:13.990Z [INFO] plugin/reload: Running configuration MD5 = 7c717fabd578b2514cf60fee426fabc6
2019-05-24T04:41:13.992Z [INFO] CoreDNS-1.5.0
2019-05-24T04:41:13.992Z [INFO] linux/amd64, go1.12.2, e3f9a80
The DNS service is available:
$ kubectl get svc --namespace=kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.32.0.10 <none> 53/UDP,53/TCP 18d
The endpoints are exposed:
$ kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns 10.200.21.33:53,10.200.22.17:53,10.200.21.33:53 + 1 more... 18d
As mentioned above, DNS works but only from the default
namespace.