My setup
Two nodes (2GB RAM, 2 vCPU) running docker engine (v17.06.1-ce) -- one swarm and one worker. Internal network bandwidth: 10Gbps. All files and databases are located outside this docker cluster (AWS S3 and different instances for database).
What I am trying to achieve?
I am trying to create a docker based "platform" where I push stateless services and docker handles load balancing, updates etc. Besides this, I am also trying to set up reverse proxy and allow specific services to have access to this proxy.
What I have done so far?
Firstly, I created an overlay network and called it "public." (10.0.9.0/24) Then, I created an nginx service in "global" mode. The service by itself is attached to "public" network. I checked both my worker and swarm nodes and the service runs in both of them without a problem.
Secondly, I created docker compose file for fast deployment of multiple services. For the sake of my testings, I kept one service per compose file:
version: '3.3'
services:
web:
image: app1_image:latest
networks:
- public
networks:
public:
external:
name: public
For the second service, I just changed the image name and kept everything else the same. Ran both "stacks":
docker stack deploy --with-registry-auth --compose-file compose1.yml app1
docker stack deploy --with-registry-auth --compose-file compose2.yml app2
After inspecting both services, I see that both services are in "overlay" network with IPs such as 10.0.9.5 (app1_web) and 10.0.9.6 (app2_web). app1_web is created in swarm node and app2_web is created in worker node.
So, I create two nginx config files for both of my services in the following way:
server {
listen 80;
server_name app1.example.com;
location / {
proxy_pass http://app1_web; # This line is important
# Other proxy parameters
}
}
As you see I am passing the service name in nginx configuration. For easier config management I use docker configs:
docker config create nginx_app1.conf app1.conf
docker config create nginx_app1.conf app1.conf
docker service update --config-add source=nginx_app1.conf,target=/etc/nginx/conf.d/app1.conf nginx_proxy
docker service update --config-add source=nginx_app2.conf,target=/etc/nginx/conf.d/app2.conf nginx_proxy
Adding these configs automatically restarts nginx services and runs them. This is all. I wanted to give you a gist of my process before moving forward.
The problem
app1_web is created in swarm; so, when I go to app1.example.com, nginx proxies my request to the service and I get a proper output. This is what is expected and I am happy with the outcome.
However, because app2_web is created in worker node, nginx gives me an error that app2_web does not exist. So, I started troubleshooting.
From swarm, I found the docker instance ID and tried to run a command from nginx proxy:
docker exec nginx-proxy-id ping app2_web
This gave me an "Bad address" error. So, I went into compose2.yml and added ports:
ports:
- 5380:80
When I went to swarm.example.com:5380, it basically gave me 404. However, opening the same port from worker.example.com:5380 opened app2.
I tested the same for app1. I replicated app1 using docker service scale app1=2 and the service got created in worker node. Then I paused the service in swarm using docker pause app1-id. When I went to app1.example.com, it would work half the time. I think it was still weird because I was expecting Docker to know that service is paused and only proxy the service to worker node but whatever. At least it was working. Replicating app2 did not help though. I still kept getting error that host name does not exist. After this, I went further and told the worker node to leave the swarm: docker swarm leave and coincidentally, everything worked normally...
After spending at least 10 hours on this, I am lost on what I am doing wrong here. For some reason, when service is created in worker first, Docker doesn't like it.
Sorry for such a long wall of text. I wanted to share all the steps I took. I would really appreciate your help.
