Context
I have a Rails app with a nginx server and Passenger.
The app is dynamically generating pages based from the request url: if the url exists in the database the app renders the corresponding page. Or if the url does not exist in the database the app renders a 404 page.
Problem
Many crawlers are trying to find vulnerabilities and request lots of urls (.git, admin/config.php, wp-login.php etc...)
Each of those requests are reaching the Rails app, which is generating hits in the database.
Solution
I am looking for a way to do this:
- first time a "non existent" url if requested it goes through the Rails app, which responds with a 404
- nginx caches and remember this url
- next time the same url is requested, nginx directly respond with 404 status without going through the Rails app
Also when the Rails app is restarted (through Passenger) this cache should be purged.
Tries
- I tried to add
fastcgi_cache_valid 404 10m;
in the server block, it's not working. - Also tried
proxy_cache_valid 404 10m;
As you may guess I'm new to nginx. Thanks for your help.
Nginx config
server {
listen ...;
server_name ...;
root /path/to/rails/app;
error_page 404 /404;
error_page 500 502 503 504 /500;
# First I tried this, no success so I removed it
fastcgi_cache_valid 404 10m;
# Then I tried this, no success so I removed it also
proxy_cache_valid 404 10m;
location / {
gzip_static on;
etag off;
charset utf-8;
add_header Cache-Control "max-age=0, private, must-revalidate";
add_header Referrer-Policy strict-origin-when-cross-origin;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
add_header X-Content-Type-Options nosniff;
add_header X-Frame-Options deny;
add_header X-XSS-Protection "1; mode=block";
location = / {
try_files /cached/index.html @rails;
}
location / {
try_files /cached$uri.html /cached$uri $uri @rails;
}
}
location @rails {
passenger_enabled on;
passenger_ruby /path/to/ruby;
passenger_app_env production;
}
}