add_header not working in Nginx location

Question

So, I've been trying to figure this out for the past 8 hours, but it seems I'm stuck...

I have the following Nginx config file:

server_tokens off;

upstream php-handler {
  server unix:/var/run/php5-fpm.sock;
}

server {
  listen 80;
  server_name     domain.net;
  access_log      /var/log/nginx/domain.net-access.log;
  error_log       /var/log/nginx/domain.net-error.log;




  location ~* \.(jpg|jpeg|gif|png|js|css|ico|eot|woff|ttf|svg|cur|htc|xml|html|tgz)$ {
       expires 24h;
  }

  root /var/www/html/domain.net;

  index index.php;


  location ~ ^/cars/sale(.*) {
        add_header X-Robots-Tag "noindex, nofollow" always;
        try_files $uri $uri/ /index.php;
  }

  location ~ ^/(?:\.htaccess|config){
       deny all;
  }

  location / {
        try_files $uri $uri/ /index.php;
  }


 location ~ \.php(?:$|/) {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_pass php-handler;
    fastcgi_read_timeout 120s;
    fastcgi_buffer_size 128k;
    fastcgi_buffers 4 256k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_ignore_client_abort on;
    fastcgi_param  SERVER_NAME $http_host;
  }

}

The issue is that the X-Robots-Tag from the "/cars/sale" location is not added, no matter what I tried. I'm guessing that this is because the request is passed to the final ".php" location where any previously added header is forgotten. Is there any way in which I could add this header only for that specific location without using more_set_headers?

We decided to add it from the web server for some very good reasons... — Tony, Aug 27 '18 at 17:31

score 1 · Accepted Answer · answered Aug 29 '18 at 04:53

You can actually just do this:

map $request_uri $robot_header {
    default "";
    ~^/cars/sale(.*) "noindex, nofollow";
    ~^/bikes/sale(.*) "noindex, nofollow";
    ~^/motorbikes/sale(.*) "noindex, nofollow";
}

But then if they all follow this pattern then you could just do this:

map $request_uri $robot_header {
    default "";
    ~^/(.+?)/sale(.*) "noindex, nofollow";
}

Your config is pretty messy. When using regular expressions Nginx will pick the first block which matches to serve your request so the order you list them is important.

You can nest another php block within your cars location block and add the header in there. If you specify the php handler as an upstream server then you dont have to include all the fastcgi parameters each time, which keeps things neater.

Thank you for the pointer (indeed, adding all locations in one map block is a bit cleaner). And yes, I know that order matters. :) The locations I've shown here are just examples. Also, I did try nesting a php block inside the custom locations, but it did not work (the php application is quite intricate, requiring multiple rewrites which I did not bother to show in my example, so the actual configuration is much more complicated than what I've written here). As I said, I tried all possible scenarios and the only one that works is the one with the map directive. Thank you for your input! — Tony, Aug 31 '18 at 05:16

score 0 · Answer 2 · answered Aug 28 '18 at 07:04

So... I've thought of a solution after a good night's sleep. It's a very dirty fix, but it's literally the only thing that works in my particular case:

In the http block I'm adding:

map $request_uri $robot_header1 {
    default "";
    ~^/cars/sale(.*) "noindex, nofollow";
}

map $request_uri $robot_header2 {
    default "";
    ~^/bikes/sale(.*) "noindex, nofollow";
}

map $request_uri $robot_header3 {
    default "";
    ~^/motorbikes/sale(.*) "noindex, nofollow";
}

(These are just three as an example, but I've actually generated ~200 of these inside a file which I'm including in the http block)

And in the server block I've added:

add_header X-Robots-Tag $robot_header1;
add_header X-Robots-Tag $robot_header2;
add_header X-Robots-Tag $robot_header3;
...

I've also had to increase the Nginx parameter "variables_hash_bucket_size" to 512, because the default of 64 is not enough for so many variables, like I needed. So, I hope this will help someone else too...

add_header not working in Nginx location

2 Answers2