0

Need some collective brainstorming :). There is a heavy PHP application (e.g. Magento) and each time a visitor hits a non-existent page (or there is a problem with a particular part of the application and it returns errors) the load on the application server is snowballing:

  • in case of non-existent pages they are not cached and the application spends quite a lot of resources to check everything and generate a cached copy, which will be a waste of time;

  • in case of a problem with a part of the application, the application will spend valuable time to generate these error responses in expense of the visitors of healthy parts of the site.

The idea here is to cache all negative responses like 404 and 5xx coming from the FastCGI backend on the nginx frontend for, say, 5 minutes - this will significantly lower the performance impact on the application servers.

The caching can be quite easily implemented using fastcgi_cache_* directives. However, once you define fastcgi_cache_* directives in the location block where fastcgi_pass is located it will try to cache everything going through that location block.

Therefore, the question is how to limit fastcgi_cache to, say, 404 responses from the backend scripts only?

Just to illustrate the scenario a bit more. Imagine that a request goes to /app/missing, the application has it's only entry point in /index.php, so nginx will pass a request to the backend FastCGI server calling /index.php/app/missing. Now, since /app/missing does not exist index.php will return HTTP 404. This route will crunch CPU as many times as you request /app/missing. What is desired is that once HTTP 404 was returned back from index.php the /app/missing is cached with the resulting 404 page from index.php on nginx, so if somebody requests /app/missing right away no call is performed to the PHP backend but a cached page is returned.

galaxy
  • 1,974
  • 1
  • 13
  • 15
  • Of course you should really fix your 404 page within Magento. It's a CMS page and to many stores try to provide smart logic which backfires as soon as for example a security scan is hitting the site. Also, just to make sure, requests for `/app` should really be marked internal and thus generate 403. Last but not least, look into naxsi if a security scan prompted this question. –  Sep 18 '14 at 06:46
  • No, this is the opposite of what I was asking. There is no way you can fix Magento or any other heavy CMS to generate light-weight enough 404 page. The problem is the request diving into the PHP application layer. This costs dearly on a high-performance setup. Avoiding unnecessary calls to PHP stack benefits the responsiveness and performance of the site. Also, the question was not related to Magento, but to heavy applications in general. The /app/missing was just an example -- it's nothing to do with the Magento /app folder. – galaxy Sep 19 '14 at 11:49
  • The only thing you can lock down are the things in the filesystem. Any virtual paths are the responsibility of the page generating application, specifically it's router. So you make think the 404 page is the problem, in reality, you are already in php the moment a request for `/deleted/product` comes in. It will simply generate a 404 status code. –  Sep 19 '14 at 13:48
  • OK, this is a pointless discussion. You don't want to see the picture and are discussing something totally irrelevant. The question was about _avoiding_ unnecessary calls to PHP/Ruby/Python, Tero and I understood and solved the issue. It seems that you simply have no clue about the topic. – galaxy Sep 20 '14 at 01:32

1 Answers1

2

This could do the trick:

fastcgi_cache_valid 200 301 302 0;
fastcgi_cache_valid 404 500 503 5m;

If nginx doesn't allow 0 as the value here, then you need to make the PHP application add the HTTP header X-Accel-Expires: 0 to its output on valid pages, and then have this line in nginx config:

fastcgi_cache_valid 404 500 503 5m;
Tero Kilkanen
  • 34,499
  • 3
  • 38
  • 58
  • Well, according to nginx documentation fastcgi_cache_valid has lower priority than headers. Anyway, I'm currently testing a few ideas and will update this question as I go. Also, I'm a sysadmin so I don't have control over applications' source code, therefore the solution should not rely on the code modification. – galaxy Sep 17 '14 at 11:17
  • OK, I ended up with: fastcgi_cache_valid 404 5m; fastcgi_cache_valid any 0; Works perfectly, so I'll accept your answer as the correct one. – galaxy Sep 17 '14 at 11:50