I'm having a very odd reproducible issue with Docker Swarm. I'm attempting to deploy a Dgraph server cluster via Docker Swarm across four LXD containers. For context, Dgraph Zero is the control server and each Dgraph Alpha server does the lifting and must connect to a Zero server on launch. Ratel is just a web UI server for database queries and mutations. The topography looks like this:
- Host: KDE Neon workstation
- LXD Container: zero
- Docker Node: dg-zero
- Docker Container: dgraph_zero
- Docker Container: dgraph_ratel
- Docker Node: dg-zero
- LXD Container: alpha1
- Docker Node: dg-alpha1
- Docker Container: dgraph_alpha1
- Docker Node: dg-alpha1
- LXD Container: alpha2
- Docker Node: dg-alpha2
- Docker Container: dgraph_alpha2
- Docker Node: dg-alpha2
- LXD Container: alpha3
- Docker Node: dg-alpha3
- Docker Container: dgraph_alpha3
- Docker Node: dg-alpha3
- LXD Container: zero
These are all deployed in as a swarm via docker stack deploy
using the following docker-compose.yml config:
version: "3"
networks:
dgraph:
services:
zero:
image: dgraph/dgraph:latest
hostname: "zero"
volumes:
- data-volume:/dgraph
ports:
- "5080:5080"
- "6080:6080"
networks:
- dgraph
deploy:
placement:
constraints:
- node.hostname == dg-zero
command: dgraph zero --my=zero:5080 --replicas 3 --bindall=true
alpha1:
image: dgraph/dgraph:latest
hostname: "alpha1"
volumes:
- data-volume:/dgraph
ports:
- "8080:8080"
- "9080:9080"
networks:
- dgraph
deploy:
placement:
constraints:
- node.hostname == dg-alpha1
command: dgraph alpha --my=alpha1:7080 --lru_mb=1024 --zero=zero:5080 --bindall=true
alpha2:
image: dgraph/dgraph:latest
hostname: "alpha2"
volumes:
- data-volume:/dgraph
ports:
- "8081:8081"
- "9081:9081"
networks:
- dgraph
deploy:
placement:
constraints:
- node.hostname == dg-alpha2
command: dgraph alpha --my=alpha2:7081 --lru_mb=1024 --zero=zero:5080 -o 1 --bindall=true
alpha3:
image: dgraph/dgraph:latest
hostname: "alpha3"
volumes:
- data-volume:/dgraph
ports:
- "8082:8082"
- "9082:9082"
networks:
- dgraph
deploy:
placement:
constraints:
- node.hostname == dg-alpha3
command: dgraph alpha --my=alpha3:7082 --lru_mb=1024 --zero=zero:5080 -o 2 --bindall=true
ratel:
image: dgraph/dgraph:latest
hostname: "ratel"
ports:
- "8000:8000"
networks:
- dgraph
command: dgraph-ratel
volumes:
data-volume:
All of the services are deploying correctly and running. The problem is they cannot communicate with one another because the hostnames are not resolving to the correct IP.
I have a swarm currently running and when I run docker container inspect
on my running Zero container I get the following output for the network config:
"NetworkSettings": {
"Bridge": "",
"SandboxID": "4548013a15833d086b281a8c2dd61ced6ea5c92f815a305f7337effe9b04a13a",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"8080/tcp": null,
"9080/tcp": null
},
"SandboxKey": "/var/run/docker/netns/4548013a1583",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"dgraph_dgraph": {
"IPAMConfig": {
"IPv4Address": "10.0.9.3"
},
"Links": null,
"Aliases": [
"8b48711ab0cd"
],
"NetworkID": "lve3kr9vm42rwu1nci897zey7",
"EndpointID": "056ae62475da805ec212d9ec2b2e4a5c9e09e2405c15ad6e8b298e90669b512d",
"Gateway": "",
"IPAddress": "10.0.9.3",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:0a:00:09:03",
"DriverOpts": null
},
"ingress": {
"IPAMConfig": {
"IPv4Address": "10.0.0.157"
},
"Links": null,
"Aliases": [
"8b48711ab0cd"
],
"NetworkID": "vjhpbsc1766lbvtu169fmh81l",
"EndpointID": "29bbc4de97e98b2e05a46dd42020dd1fbb75ff07d8c08a00b8ba6f2f4e00ec2a",
"Gateway": "",
"IPAddress": "10.0.0.157",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:0a:00:00:9d",
"DriverOpts": null
}
}
}
As can be seen, the IP address of my Zero container is 10.0.9.3. However if I login into the shell of any of my Alpha containers and run ping zero
it attempts to ping 10.0.9.2. This is consistently reproducible by destroying and re-deploying the stack. The final octet in the IP address of my Zero container will always be one greater than the IP address the hostname zero
resolves to in the other three containers. I can ping all of my containers from within one another using the correct IP addresses, but I must use host names because I don't know what the other container's addresses will be before swarm creation and I need to configure the server cluster to speak to each other.
There are no firewall rules in place. My LXD containers can all communicate. All LXD container IP addresses are listed as peers in the Docker network config. I don't know if it's related or not, but I'm unable to reach my services from the host containers even though I've published ports. If it's possible that's related I can provide more information I've found but for now I'll leave that out.
How do I figure out why my swarm isn't resolving the correct hostnames to the correct containers?
Docker version is 19.03.5 from the official repo. LXD containers are official 18.04 cloud containers from Ubuntu's LXD image server. Host is KDE Neon based on Ubuntu 18.04.