I am running a swarm cluster with only 3 masters. I have a few ports exposed across various services in the swarm. And an Nginx service running in all 3 nodes acts as a reverse proxy.
When I start the stacks, all these ports are accessible and responding as expected. But after some time, some of these ports become non-responsive to some servers and the Nginx starts throwing 504 timeouts. (i.e) port 5000 is responding to requests from Master 1 but not from Master 2 and 3 and port 6000 is responding to requests from Master 1 and 3 but not from Master 2.
These are the output's of nmap -Pn 172.17.0.1
# Master 1
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
5000/tcp open upnp
6000/tcp open mmcc
# Master 2
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
5000/tcp filtered upnp
6000/tcp filtered mmcc
# Master 3
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
5000/tcp filtered upnp
6000/tcp open mmcc
The Output of netstat -lnpt | grep 5000 (Same in all 3 nodes)
tcp6 0 0 :::5000 :::* LISTEN -
I am unable to figure out if this issue is something related to the docker swarm or Nginx or the host machine itself. Any help in debugging this would be highly appreciated. I would be more than happy to share any more info if required.
Thanks in advance.
