1

Summary

I can only resolve hostnames from and for the default namespace. Requests from and for other namespaces, like kube-system, return NXDOMAIN from coredns. Is there some authorisation that needs to be done?

Details

I have set up a learning cluster by following Kubernetes the Hard Way except using VirtualBox instead of GCP. All else appears to work except DNS queries in non-default namespaces. More concretely:

  • Works (run from default namespace): kubernetes.default
  • Does Not work (run from kube-system namespace, get NXDOMAIN): kubernetes.kube-system

Configuration

I have upgraded to Coredns 1.12 because 1.2.2 failed to resolve anything around 50% of the time. With 1.12 requests resolve 100% of the time (if in the default namespace).

I had to edit the config (in the configmap) by removing the "loop" and "proxy" (replace proxy with forward) plugins to get it working. My config is as follows:

    .:53 {
        forward . 192.168.99.254:53 # dnsmasq on host machine
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        cache 30
        reload
        loadbalance

Troubleshooting

Following this guide I see:

The pods are running OK

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-6f8fb48d88-dz9cd   1/1     Running   0          14m
coredns-6f8fb48d88-s5rvp   1/1     Running   0          14m

Checking the console I see healthy logging

2019-05-24T04:41:13.990Z [INFO] plugin/reload: Running configuration MD5 = 7c717fabd578b2514cf60fee426fabc6
2019-05-24T04:41:13.992Z [INFO] CoreDNS-1.5.0
2019-05-24T04:41:13.992Z [INFO] linux/amd64, go1.12.2, e3f9a80

The DNS service is available:

 $ kubectl get svc --namespace=kube-system
NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns         ClusterIP   10.32.0.10   <none>        53/UDP,53/TCP   18d

The endpoints are exposed:

$ kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                                                     AGE
kube-dns   10.200.21.33:53,10.200.22.17:53,10.200.21.33:53 + 1 more...   18d

As mentioned above, DNS works but only from the default namespace.

Friedrich 'Fred' Clausen
  • 2,298
  • 22
  • 25

1 Answers1

0

It appears that it was an issue with the busybox image - if I use generic Ubuntu then records resolve OK and the services that need records are able to get them. E.g. metrics-server can resolve the nodes.

Friedrich 'Fred' Clausen
  • 2,298
  • 22
  • 25