Kubernetes Pod DNS Resolution

Question

Currently able to resolve all services to IP addresses and telnet and ping them. Unable to resolve pods to IP addresses. Though can lookup pod IP addresses with kubectl and telnet and ping them. How do I need DNS configured to resolve pods in Kubernetes 1.9.6, dns-controller:1.9.1.

Answer 1

@mk_sta 's answer a bit easier to work with you can run this one liner to test your DNS:

kubectl run busybox --image=busybox --rm --attach --command -- sh -c "cat /etc/resolv.conf; nslookup $POD.$NAMESPACE.pod.cluster.local"

Example output:

If you don't see a command prompt, try pressing enter.

Server:         100.64.0.10
Address:        100.64.0.10:53


*** Can't find $POD.$NAMESPACE.pod.cluster.local: No answer

deployment.apps "busybox" deleted

Bonus is that it deletes the deployment/pods after running.

---

For me the problem was that my pods are in a statefulset and therefore Pod DNS resolution is a bit different. You have to use (for example):

web-{0..N-1}.nginx.default.svc.cluster.local

pod-N.$GOVERNING_STATEFULSET.$NAMESPACE.svc.cluster.local

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

That's one thing that can go wrong. You may want to provide more info about your particular case.

Answer 2

The best way that I have seen to resolve this is by creating a Jump Pod.

Step 1. Create the pod with a spec file.

cat << ENDL >> jumpod.yml
apiVersion: v1
kind: Pod
metadata:
  name: jumpod
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
ENDL

Step 2. Ether apply or create the pod into the cluster and on to a node. Note that it will be located at the default namespace.

kubectl apply -f jumpod.yml

Now you can use this pod to either get the DNS of a Pod or Service.

For Example:

Pod nslookup Step 3. Get the ip address of the pod you are looking for.

kubectl get pods -o wide

Step 4. Use the Jump Pod to do an nslookup

kubectl exec -it jumpod ping 10.244.0.149

Service nslookup Step 5. Lookup the services ip address that you may want to use.

kubectl get services --all-namespaces

Step 6. Now use the jump pod to do an nslookup on the service ip address

kubectl exec -it jumpod nslookup 10.245.150.103

Below is an example on how the main cluster using nslookup will not work and how the busybox (aka jump pod) will work.

$ kubectl exec -it jumpod nslookup 10.245.150.103
Server:    10.245.0.10
Address 1: 10.245.0.10 kube-dns.kube-system.svc.cluster.local
Name:      10.245.150.103
Address 1: 10.245.150.103 hello.develop.svc.cluster.local
$ nslookup 10.245.150.103
Server:     8.8.8.8
Address:    8.8.8.8#53
** server can't find 103.150.245.10.in-addr.arpa: NXDOMAIN

Hope this helps!

P.S If by chance you need to export the DNS file you created within the pod you can use this.

kubectl cp default/POD_NAME:bin/FILE_NAME /Users/username/FILE_NAME

or just use > to export it to a file.

Answer 3

You can create test Pod with busybox in order to check DNS resolving function and debug the issue:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

Check /etc/resolv.conf file on the busybox Pod:

kubectl exec busybox cat /etc/resolv.conf

The content should be like this:

search default.svc.cluster.local svc.cluster.local cluster.local .....
nameserver 10.0.0.10
options ndots:5

Make a lookup request to the target Pod in your cluster:

kubectl exec -ti busybox -- nslookup XXX-XX-XX-XX.default.pod.cluster.local

In case you have not been resolved Pods DNS name, you can check DNS components logs withkubectl logs command:

kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz