Nginx - Empty response in error page with proxy_pass

Question

Task: nginx config for reverse proxy to docker containers with dynamic lookup and caching.

Process workflow:

1. check cache. if upstream found — proxy
2. if not found — request it, cache and proxy
3. if found but cache is invalid — clear cache and run again

Current config (template):

server {
    listen *;
    server_name {{host}};

    set $attempt 0;

    location / {
        try_files '/dev/null' @run;
    }

    location @run {
        internal;
        set $container_name "{{container_name}}";
        set $upstream "";

        rewrite_by_lua '
             local attempt = tonumber(ngx.var.attempt)

             if attempt > 1 then
                ngx.log(ngx.ALERT, "Upstream down")
                ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
             end

             local routes = ngx.shared.upstream_cache

             if attempt > 0 then
                ngx.log(ngx.ALERT, "Deleteing cache")
                routes:delete(ngx.var.http_host)
             end

             ngx.var.attempt = attempt + 1

             -- try cached route first
             local route = routes:get(ngx.var.http_host)
             if route == nil then
                 ngx.log(ngx.ALERT, "Asking docker about IP of " .. ngx.var.http_host)
                 local handle = io.popen("docker inspect --format \'{{ .NetworkSettings.IPAddress }}\' " .. ngx.var.container_name)
                 local result = handle:read("*a")
                 handle:close()
                 route = result
             end

             if route ~= nil then
                 ngx.var.upstream = route:gsub("^%s*(.-)%s*$", "%1")
                 routes:set(ngx.var.http_host, route)
             else
                 ngx.exit(ngx.HTTP_NOT_FOUND)
             end
        ';

        error_page 504 @run;

        proxy_buffering             off;
        proxy_set_header            Host $host;
        proxy_set_header            X-Real-IP  $remote_addr;
        proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect              off;
        proxy_send_timeout          30;
        proxy_read_timeout          30;
        proxy_connect_timeout       2;

        proxy_pass $scheme://$upstream;

    }

}

It almost works. Everything is ok except scenario when cache is invalid and error_page triggers.

In this case, the process runs as it should, that is seen by log:

[error] 7238#0: *6 upstream timed out (110: Connection timed out) while connecting to upstream
[alert] 7238#0: *6 [lua] [string "rewrite_by_lua"]:12: Deleteing cache
[alert] 7238#0: *6 [lua] [string "rewrite_by_lua"]:21: Asking docker about IP

And it makes a correct request to upstream.

BUT response is empty!

And on next request — everything is ok, upstream is taken from cache.

Why and how to fix this?

Answer 1

Damn simple. error_page 504 = @run; instead of error_page 504 @run;