http headers vanish when proxying http errors through nginx

Question

I'm using nginx as TLS terminator in front of an Apache 2.4 server.

I'm using add_header X-Content-Type-Options nosniff; in nginx to add this header to every response.

If the HTTP status code returned by Apache is below 400 the header is correctly set, but if the status is greater or equal 400 the header is omitted.

Same for the gzip module. If the status code is below 500, the gzip module automatically compresses the response body. But if the HTTP status code geturned by Apache is greater or equal 500 the gzip module just does nothing.

Some relevant parts of my nginx config:

proxy_intercept_errors    off;
proxy_ignore_client_abort off;
proxy_http_version        1.1;
proxy_hide_header         X-Powered-By;
proxy_set_header          Connection  "";     #for keepalive to backend
add_header X-Content-Type-Options nosniff;

### gzip ###
gzip on;
gzip_min_length 20;     #default: 20
gzip_comp_level 9;
gzip_proxied any;
gzip_vary on;
gzip_types *;
gunzip on;

Is there anything I can do to deactivate this behaviour and add my header to HTTP error responses and even gzip HTTP 500 responses?

Edit: Here is the rest of my config:

/etc/nginx/nginx.conf

user proxy;
worker_processes 4; # 2 * num_cpus
worker_priority -20;
pid /run/nginx.pid;

include /etc/nginx/modules-enabled/*;

events {
    worker_connections 32768;
    multi_accept on;
    use epoll;
}
worker_rlimit_nofile 70000; # slightly more than double of worker_connections

http {
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

/etc/nginx/mime.types is the standard debian file.

in /etc/nginx/modules-enabled are the following modules:

50-mod-http-auth-pam.conf
50-mod-http-dav-ext.conf
50-mod-http-echo.conf
50-mod-http-geoip.conf
50-mod-http-image-filter.conf
50-mod-http-subs-filter.conf
50-mod-http-upstream-fair.conf
50-mod-http-xslt-filter.conf
50-mod-mail.conf
50-mod-stream.conf

/etc/nginx/conf.d/sslproxy.conf

### global ###
server_tokens           off;
server_name_in_redirect off;
ignore_invalid_headers  on;
if_modified_since       before;
root                    /etc/nginx/content/;
ssi                     off;
ssi_silent_errors       on; # testing=off
limit_conn_zone     $binary_remote_addr zone=perip:10m;
limit_conn_zone     $server_name zone=perserver:10m;

#all configured cache paths
proxy_cache_path    /var/lib/nginx/proxy/all keys_zone=all:64m inactive=168h;   #7 tage
proxy_cache_path        /var/lib/nginx/proxy/temporaryerror keys_zone=temporaryerror:16m inactive=24h;

#standard cache
proxy_cache_use_stale   updating error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_cache_key     "$request_method|$scheme://$server_name$uri$is_args$args";
proxy_temp_path     /var/lib/nginx/tmp;
proxy_cache_min_uses    1;
proxy_cache_lock    on;
proxy_cache     all;

### tcp ###
tcp_nodelay             on;
tcp_nopush              off;
sendfile                off;
keepalive_requests      100;

### timeouts ###
client_header_timeout   60;
client_body_timeout     60;
send_timeout            900;
#keepalive_disable  none;   #was: non existent
keepalive_timeout       300 300;

### gzip ###
gzip on;
gzip_min_length 20; #default: 20
gzip_comp_level 9;
gzip_proxied any;
gzip_vary on;
gzip_types *;   #text/* application/x-javascript;
gunzip on;

### buffers ###
client_header_buffer_size   4k;
client_body_buffer_size     8m;
large_client_header_buffers 4 8k;
client_max_body_size        128m;
output_buffers              1 32k;
postpone_output             0;
ssl_buffer_size             4k;

### errors ###
recursive_error_pages   on;
error_page              401 402 403 405 406 407 408 409 410 411 413 414 415 416 417 421 422 423 424 426 428 429 431 451 /temporary_error.php;
error_page              500 501 502 503 504 /temporary_error.php;
error_page              404 =410 /temporary_error.php;
error_page              443 =200 /temporary_error.php;

### acl ###
deny            all;

### ssl ###
ssl_prefer_server_ciphers on;
ssl_protocols       TLSv1.2;
ssl_ciphers     "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256";
ssl_ecdh_curve      X25519:secp521r1:secp384r1;
ssl_dhparam     /etc/nginx/ssl/dhparam.pem;
ssl_session_cache       off;
ssl_session_timeout     60m;
ssl_session_tickets on;
ssl_session_ticket_key  /run/nginx_session_ticket.key;
ssl_stapling        on;
resolver        127.0.0.1:53 valid=4s;
resolver_timeout    8s;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/trusted_certs.pem;
ssl_client_certificate  /etc/nginx/ssl/trusted_certs.pem;

/etc/nginx/conf.d/upstream.conf

upstream backend {
    server 127.0.0.1:8080;
    keepalive 32;
}

upstream remote {
    server 10.10.0.2:8080;
    keepalive 32;
}

/etc/nginx/sites-enabled/default

server {
    ssl         on;
    ssl_certificate     ssl/www.example.org.crt;
    ssl_certificate_key ssl/www.example.org.key;
    listen          [::]:443 ssl http2 default_server backlog=256 fastopen=256 so_keepalive=30s:1m:8;
    listen          443 ssl http2 default_server backlog=256 fastopen=256 so_keepalive=30s:1m:8;
    server_name     "";
    allow           all;
    return          400;
}

/etc/nginx/sites-enabled/subdomain

server {
    ssl         on;
    ssl_certificate     ssl/subdomain.example.org.crt;
    ssl_certificate_key ssl/subdomain.example.org.key;
    listen          [::]:443 ssl http2;
    listen          443 ssl http2;
    allow           all;
    server_name     subdomain.example.org;
    location = /temporary_error.php {
        include /etc/nginx/proxy_params;
        include /etc/nginx/temporaryerror_params;
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
        proxy_pass http://remote;
    }
    location / {
        include /etc/nginx/proxy_params;
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
        proxy_pass http://remote;
    }
}

/etc/nginx/proxy_params

proxy_intercept_errors  off;
proxy_ignore_client_abort off;
#proxy_redirect          off;

proxy_connect_timeout   4;
proxy_send_timeout      16;
proxy_read_timeout      900;

proxy_http_version  1.1;

add_header X-Content-Type-Options nosniff;
#proxy_set_header        Accept-Encoding        "";
proxy_set_header        Host            $server_name;
proxy_set_header    X-Real-IP       $remote_addr;
proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
proxy_set_header        X-Forwarded-By      $server_addr:$server_port;
proxy_set_header        X-Forwarded-Proto   $scheme;
proxy_hide_header       X-Powered-By;
proxy_hide_header   X-Accel-Expires;
proxy_hide_header   X-Accel-Buffering;
proxy_set_header    Connection      ""; #for keepalive

proxy_buffering     off;
proxy_buffers       32 4k;
proxy_buffer_size   4k;
proxy_busy_buffers_size 16k;
proxy_max_temp_file_size 0;

limit_conn perip 128;
limit_conn perserver 512;

/etc/nginx/temporaryerror_params

proxy_cache_use_stale   updating error timeout invalid_header http_500 http_502 http_503 http_504 http_403 http_404;
proxy_cache_key     "$request_method|$scheme://$server_name$uri$is_args$args";
proxy_temp_path     /var/lib/nginx/tmp;
proxy_cache_min_uses    1;
proxy_cache_lock    on;
proxy_cache     temporaryerror;

I hope this helps.

Please edit your question to include your entire Nginx and server block config as it's likely relevant. — Tim, May 29 '17 at 01:56
@Tim I added my whole server config as requested. Hope this helps. — Thilo, May 29 '17 at 22:02
Thanks, I wanted to see if you were doing something odd. Looks pretty standard. Do other headers get added in the error section? What problem is this causing, or is it a theoretical question? Regarding gzip I'm going to guess (complete guess) that something in the Nginx code tells Nginx not to bother compressing pages with an error status - errors should be rare enough that it makes little difference. Is it causing a problem? — Tim, May 30 '17 at 02:13
Well, regarding gzip: that's not a real problem, but it would be nice if error pages delivered by the backend would get compressed, too. That would just be consistent imho...regarding the headers: no, only the headers shown in the config (Strict-Transport-Security and X-Content-Type-Options) are added. But both are missing on pages >=400. And that's a real problem (security wise). The error page delivered by the backend is /temporary_error.php by the way. — Thilo, May 30 '17 at 02:36
HSTS doesn't have to be on every page, so that's not a problem. Likewise nosniff isn't likely to be a problem on an error page. So I can't see that there's actually a problem. If there is it's beyond my knowledge to help sorry, I think you'll have to start digging into the Nginx source code — Tim, Jun 04 '17 at 19:56

score 3 · Accepted Answer · answered Aug 05 '17 at 21:48

Today I stumbled upon the solution in an ssl labs thread over here: https://community.qualys.com/thread/17333-hsts-header-not-being-set-by-nginx-on-error

Basically you have to add the always keyword in the add_header config option. Quoting the nginx docs (http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header):

If the always parameter is specified (1.7.5), the header field will be added regardless of the response code.

This helps and now every response (including the errors) have all headers set as expected.

http headers vanish when proxying http errors through nginx

1 Answers1