'dig' between docker containers returns internal IP instead of external IP

Question

First, sorry for the lenghty post, but I don't know how to boil it down to less than this. I have already cut a lot of unnecessary configuration stuff to end up with the basic info needed.

I am setting up a dns recursor pair, ns1 and ns2, each on a separate server. Each server contains the powerdns-pair of dnsdist frontent and recursor backend, running in separate containers, one for each, managed by docker-compose, based on tcely/dnsdist from Docker Hub.

ns1: 192.168.0.1
ns2: 192.168.0.2

My objective is to have dnsdist able to query both recursors, in it's loadbalancing function, so i have a dnsdist.conf that, cut down to the essentials look like this:

newServer{address='192.168.0.1:5300', order=1}
newServer{address='192.168.0.2:5300', order=2}
setServerPolicy(firstAvailable)
setLocal('0.0.0.0')

Largely the same for both servers but with the order reversed, so primary recursor should be the "local" one.

I won't confuse things with the recursor.conf, suffice to say that it allow-from 0.0.0.0/0, so the recursor does not block anyone from querying it at this point.

Finally we get to the point:

When I start the pod, dnsdist comes up with the following log errors:

...
Marking downstream 192.168.0.1:5300 as 'down'
Marking downstream 192.168.0.2:5300 as 'up'
...

When I try to find out why it thinks the local recursor is down, by entering into the dnsdist container and run "dig" from there, i get this:

/ # dig @192.168.0.2 -p 5300 a.root-servers.net. +short
198.41.0.4
/ # dig @192.168.0.1 -p 5300 a.root-servers.net. +short
;; reply from unexpected source: 172.20.0.1#5300, expected 192.168.0.1#5300

So - it seems when i query the other container's external IP, it replies with the internal IP. If I could query the other containers internal ip, i..e in this case 172.20.0.1, I would do so, but that IP is local to the container and changes every time it is restarted.

So, the question is, how do I get around this? Is there a way in Docker to hardcode the internal IP addresses used, or to get the container/dnsdist to ignore the fact that it get a reply from another IP address than expected?

EDIT - for clarification: Docker automatically assigns network names, so for example, i can 'ping recursor' and it will resolve to the resolver-containers private IP address. I just can't use this for anything because dnsdist.conf only accepts IP addresses, and as I started out explaining, the IP addresses are set randomly by docker when the container starts.

They're virtual machines so in theory they could have multiple interfaces each, but the idea is to just have one per server, to limit the administrative overhead. — Dokbua, Jul 16 '19 at 13:50
Seems like you have network configuration issue. That is why I've asked you about number of interfaces. — Anton Danilov, Jul 16 '19 at 14:50
Ok, maybe I misunderstood your question. The servers only have one nic at this time, the standard ensxxx type. In addition it has docker0, some virtXXXX and some BR-XXXX interfaces, but I suppose they were created by docker. — Dokbua, Jul 17 '19 at 08:28

score 0 · Accepted Answer · answered Jul 18 '19 at 11:10

The (or at least one) solution is to create a local network for the pod (container pair):

version: '3'

services:
    dnsdist:
      image: 'tcely/dnsdist'
      container_name: powerdns-dnsdist
      restart: 'unless-stopped'
      tty: true
      stdin_open: true
      command: ["--disable-syslog", "--uid", "dnsdist", "--gid", "dnsdist", "--verbose"]
      volumes:
        - /data/docker/volumes/dnsdist-recursor/dnsdist:/etc/dnsdist
      expose:
        - '53'
        - '53/udp'
      ports:
        - '53:53'
        - '53:53/udp'
      networks:
          dnsnet:
              ipv4_address: 172.28.1.10
    recursor:
      image: 'tcely/powerdns-recursor'
      container_name: powerdns-recursor
      restart: 'unless-stopped'
      command: ["--disable-syslog=yes", "--log-timestamp=no", "--local-address=0.0.0.0", "--setuid=pdns-recursor", "--setgid=pdns-recursor"]
      volumes:
        - /data/docker/volumes/dnsdist-recursor/pdns-recursor:/etc/pdns-recursor
      ports:
        - '5300:53'
        - '5300:53/udp'
      networks:
          dnsnet:
              ipv4_address: 172.28.1.11

networks:
    dnsnet:
      ipam:
        driver: default
        config:
          - subnet: '172.28.1.0/24'

Now you can setup dnsdist.conf as follows

newServer{address='172.28.1.11:53', order=1}
newServer{address='192.168.0.2:5300', order=2}
setServerPolicy(firstAvailable)
setLocal('0.0.0.0')

restart

repeat and rinse.

'dig' between docker containers returns internal IP instead of external IP

1 Answers1