Why can't I use Digital Ocean private IPs for reverse proxying with this Salt Cloud setup?

Question

I'm following along with this DigitalOcean tutorial that lays out how Salt Cloud works, with two app servers and a reverse proxy. The expected result near the end is that you should have a pure JS app running on two machines behind a reverse proxy, telling you what IP address the application is being served from, which should demonstrate that both app servers are running and being proxied. However, I'm only getting the Nginx welcome page.

Here's the nginx configuration file, first as templated, and then as generated:

Templated

### /srv/salt/nginx/files/awesome-app.conf.jin ###
##################################################

### Configuration file for Nginx to act as a
### reverse proxy for an app farm.

# Define the app servers that we're in front of.
upstream awesome-app {
    {% for server, addrs in salt['mine.get']('roles:appserver', 'network.ip_addrs', expr_form='grain').items() %}
    server {{ addrs[0] }}:1337;
    {% endfor %}
}

# Forward all port 80 http traffic to our app farm, defined above as 'awesome-app'.
server {
    listen       80;
    server_name  {{ salt['network.ip_addrs']()[0] }};  # <-- change the '1' to '0' if you're not using
                                                       #     DigitalOcean's private networking.

    access_log  /var/log/nginx/awesome-app.access.log;
    error_log  /var/log/nginx/awesome-app.error.log;

    ## forward request to awesome-app ##
    location / {
     proxy_pass  http://awesome-app;
     proxy_set_header        Host            $host;
     proxy_set_header        X-Real-IP       $remote_addr;
     proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
   }
}

Generated

    ### /srv/salt/nginx/files/awesome-app.conf.jin ###
    ##################################################

    ### Configuration file for Nginx to act as a
    ### reverse proxy for an app farm.

    # Define the app servers that we're in front of.
    upstream awesome-app {

        server 10.12.0.6:1337;

        server 10.12.0.8:1337;

    }

    # Forward all port 80 http traffic to our app farm, defined above as 'awesome-app'.
    server {
        listen       80;
        server_name  10.12.0.7;  # <-- change the '1' to '0' if you're not using
                                                           #     DigitalOcean's private networking.

        access_log  /var/log/nginx/awesome-app.access.log;
        error_log  /var/log/nginx/awesome-app.error.log;

        ## forward request to awesome-app ##
        location / {
         proxy_pass  http://awesome-app;
         proxy_set_header        Host            $host;
         proxy_set_header        X-Real-IP       $remote_addr;
         proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
       }
    }

That doesn't look right to me: the public IP of the reverse proxy is certainly not a 10.* address, but something starting with 192.241... as salt -G 'roles:rproxy' network.ip_addrs shows. All of the minions respond to a test ping and seem to be provisioned correctly otherwise.

I changed server_name {{ salt['network.ip_addrs']()[0] }}; to server_name {{ salt['network.ip_addrs']()[2] }}; in the nginx server block, since salt -G 'roles:rproxy' network.ip_addrs has three IP addresses of which only the third starts with 192.241.... Now I get a 502 Bad Gateway from nginx.

Curl output

Here is a call to curl from my home machine to the public IP of the nginx droplet:

curl -iv --trace-time <SERVER_IP>                                                                                                                                                                                                     ~
10:43:47.827486 * Rebuilt URL to: <SERVER_IP>/
10:43:47.832671 *   Trying <SERVER_IP>...
10:43:47.841036 * Connected to <SERVER_IP> (<SERVER_IP>) port 80 (#0)
10:43:47.841122 > GET / HTTP/1.1
10:43:47.841122 > Host: <SERVER_IP>
10:43:47.841122 > User-Agent: curl/7.43.0
10:43:47.841122 > Accept: */*
10:43:47.841122 >
10:43:53.847953 < HTTP/1.1 502 Bad Gateway
HTTP/1.1 502 Bad Gateway
10:43:53.848079 < Server: nginx/1.10.0 (Ubuntu)
Server: nginx/1.10.0 (Ubuntu)
10:43:53.848119 < Date: Tue, 10 Jan 2017 18:43:53 GMT
Date: Tue, 10 Jan 2017 18:43:53 GMT
10:43:53.848169 < Content-Type: text/html
Content-Type: text/html
10:43:53.848224 < Content-Length: 182
Content-Length: 182
10:43:53.848285 < Connection: keep-alive
Connection: keep-alive

10:43:53.848346 <
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.0 (Ubuntu)</center>
</body>
</html>
10:43:53.848490 * Connection #0 to host <SERVER_IP> left intact

Access logs from that request:

<HOME_IP> - - [10/Jan/2017:18:41:07 +0000] "GET / HTTP/1.1" 502 182 "-" "curl/7.43.0"
<HOME_IP> - - [10/Jan/2017:18:41:45 +0000] "GET / HTTP/1.1" 502 182 "-" "curl/7.43.0"
<HOME_IP> - - [10/Jan/2017:18:43:53 +0000] "GET / HTTP/1.1" 502 182 "-" "curl/7.43.0"

Error logs from those requests:

2017/01/10 18:41:04 [error] 7865#7865: *1 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER1_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"
2017/01/10 18:41:07 [error] 7865#7865: *1 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER2_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"
2017/01/10 18:41:42 [error] 7865#7865: *4 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER2_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"
2017/01/10 18:41:45 [error] 7865#7865: *4 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER1_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"
2017/01/10 18:43:50 [error] 7865#7865: *7 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER2_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"
2017/01/10 18:43:53 [error] 7865#7865: *7 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER1_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"
2017/01/10 18:52:02 [error] 7865#7865: *11 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER2_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"
2017/01/10 18:52:05 [error] 7865#7865: *11 connect() failed (113: No route to host) while connecting to upstream, client: <HOME_IP>, server: <APP_SERVER1_IP>, request: "GET / HTTP/1.1", upstream: "http://<APP_SERVER1_PRIVATE_IP>:1337/", host: "<APP_SERVER1_IP>"

server_name usually specifies the domain name. Please edit your question to include a curl (showing headers and a timestamp), applicable access and error logs. — Tim, Jan 10 '17 at 03:03
I'm voting to close this question as off-topic because errors and omissions and general issues with third party tutorials should be addressed to their author. — user9517, Jan 10 '17 at 08:22
@Hanginoninquietdesperation If you do that, you force people into a difficult situation where they're trying to learn about something they haven't quite gotten right, but aren't permitted to quote their references. This setup is well-encapsulated and reproducible, but too long to fit into the question, which is why I referenced the document I'm working from. I can rephrase the question to move the focus away from the tutorial, if that suits you. — bright-star, Jan 10 '17 at 17:32

score 1 · Answer 1 · answered Jan 10 '17 at 19:10

This is not the best answer since it relies on public IPs, but it is a workaround.

The Salt mine comes back with two IP addresses for each app server, of which the first is the private IP. On the other hand, the call to salt['network.ip_addrs']() in the template comes back with three IP addresses for the reverse proxy, of which the third is the public IP.

So, if you change the reverse proxy to use only public IPs, traffic should pass through:

upstream awesome-app {
    {% for server, addrs in salt['mine.get']('roles:appserver', 'network.ip_addrs', expr_form='grain').items() %}
    server {{ addrs[1] }}:1337;
    {% endfor %}
}

# Forward all port 80 http traffic to our app farm, defined above as 'awesome-app'.
server {
    listen       80;
    server_name  {{ salt['network.ip_addrs']()[2] }};  # <-- change the '1' to '0' if you're not using
                                                       #     DigitalOcean's private networking.

In this case, curl reports the right result:

curl -iv --trace-time <SERVER_PUBLIC_IP>                                                                                                                                                                                                     ~
11:08:32.790871 * Rebuilt URL to: <SERVER_PUBLIC_IP>/
11:08:32.794702 *   Trying <SERVER_PUBLIC_IP>...
11:08:32.802812 * Connected to <SERVER_PUBLIC_IP> (<SERVER_PUBLIC_IP>) port 80 (#0)
11:08:32.802986 > GET / HTTP/1.1
11:08:32.802986 > Host: <SERVER_PUBLIC_IP>
11:08:32.802986 > User-Agent: curl/7.43.0
11:08:32.802986 > Accept: */*
11:08:32.802986 >
11:08:32.816784 < HTTP/1.1 200 OK
HTTP/1.1 200 OK
11:08:32.817015 < Server: nginx/1.10.0 (Ubuntu)
Server: nginx/1.10.0 (Ubuntu)
11:08:32.817131 < Date: Tue, 10 Jan 2017 19:08:32 GMT
Date: Tue, 10 Jan 2017 19:08:32 GMT
11:08:32.817201 < Content-Type: text/plain
Content-Type: text/plain
11:08:32.817265 < Transfer-Encoding: chunked
Transfer-Encoding: chunked
11:08:32.817327 < Connection: keep-alive
Connection: keep-alive

11:08:32.817389 <
11:08:32.817536 * Connection #0 to host <SERVER_PUBLIC_IP> left intact
["APP_SERVER1_IP","APP_SERVER1_PRIVATE_IP","APP_SERVER1_PRIVATE_IP2"]~

Why can't I use Digital Ocean private IPs for reverse proxying with this Salt Cloud setup?

Templated

Generated

Curl output

1 Answers1