PHP-FPM + nginx producing a permission denied, but only on certain large pages

Question

I have an AWS Lightsail instance (1GB RAM instance) running a relatively new website (i.e. virtually no traffic). It's running nginx and PHP-FPM 7.3 (tried with 7.2 as well) and MariaDB. All of this is under CentOS 7.

Everything worked fine under the AWS free tier. I ran a T2.micro EC2 instance and a T2.micro RDS instance. Lightsail has been a bit... touchier. To make Lightsail work, I switched PHP-FPM to ondemand

ondemand - no children are created at startup. Children will be forked when new requests will connect.

I had to do this, or MariaDB would randomly crash. This doesn't appear to affect the problem below.

The Wordpress admin panel stopped working properly and everyone said to turn CONCATENATE_SCRIPTS off. That works... mostly. The editor for both posts and templates malfunctions. Nobody has been able to give me a clue why. In looking around, I found something myself.

The pages not working are not loading completely. With CONCATENATE_SCRIPTS on, CSS files are loaded in one giant page. Because this fails to completely render, the CSS and JS files are ignored by the browser. CONCATENATE_SCRIPTS works around that by simply splitting them into component files, which are smaller and load easily. But the edit page cannot be split, and debugging the underlying issue has been maddening. I get a 200 response and some data

But the page draw is incomplete. I'd say maybe 80-90% of the HTML is there, but cut off. In the section starting here (JS block)

wp.apiFetch.use( wp.apiFetch.createPreloadingMiddleware( {"\/":{"body":{"name":"S

it just abruptly ends, and at the same point every time. It's as if PHP-FPM or nginx has just stopped, but without any error logs (and most of the other issues out there about this type of setup are for pages not drawing at all). Even stranger is it's not doing it to smaller pages, just really long ones. There's no steal time in top and the instance doesn't seem to be under any serious load, so I'm not sure why it would do that. I reloaded all files fresh, and even set up a separate WP site to test this and they all do it.

Per comments, I turned on nginx debug logging and found

2019/08/07 02:33:08 [crit] 1461#0: *47 open() "/var/lib/nginx/tmp/fastcgi/3/00/0000000003" failed (13: Permission denied) while reading upstream, client: x.x.x.x, server: example.com, request: "GET /wp-admin/post-new.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000",

This doesn't make any sense why it would do this on JUST a handful of large files. No drive is full or close to it. I read this question but both nginx and PHP-FPM are running under apache. Deleting the tmp files didn't fix it either. The directories are owned by apache:root, but changing them to apache:apache has no effect. SELinux doesn't seem to be the culprit either. I'm also not using proxy_cache.

Answer 1

First of all, the failure is related to FastCGI buffering, rather than proxy cache.

This is obvious from /var/lib/nginx/tmp/fastcgi....

Why you experience an error on particularly large pages only: if your configured FastCGI buffers are not enough to fit the entire response from PHP-FPM into memory, and this will happen more often with large responses of course, NGINX will attempt to write parts of the response to temporary files.

And apparently, the permissions on the directory for holding FastCGI temporary files, does not allow NGINX to save files in there, thus it fails exactly at certain point when response is "too big".

The path /var/lib/nginx/tmp/fastcgi also indicates that you're not using an official NGINX distribution, because if you did, then you would end up with /var/cache/nginx/fastcgi_temp owned by nginx:root.

I would suggest using stock, official NGINX distribution.

but both nginx and PHP-FPM are running under apache

Off-topic, but: This is incorrect setup anyway. The correct setup is running NGINX as one user (be it apache, nginx or anything else), whereas your application's PHP-FPM pool runs under user of its own, e.g. foo. Then make your nginx user member of foo group. There's no excuse to running everything under a single user just because you have only one app on a given server.

Either way, treat it like a typical chmod issue:

• Check what user NGINX worker process runs with (user directive in your configuration)
• Try to list files of the directory in question using that user, top to bottom, until you found the location where it stops working, then fix permissions recursively.

Example, say your NGINX worker process is indeed, like you said, run by apache user and it can't access /var/lib/nginx/tmp/fastcgi:

sudo -u apache ls /var/ 

Did this work? Permissions are fine, you can traverse to this directory via the user of NGINX worker process. It is important to understand that you need to be able to traverse (as in rx permission) to all upper directories in order to be able to read contents of any directory underneath. That is, for our "final destination", which is /var/lib/nginx/tmp/fastcgi, we need to be able to read all of /var, /var/lib, etc..

If reading /var doesn't work (although would indicate a sort of corrupt system) , you need to let "others" reading it, e.g. chmod o+rX /var

sudo -u apache ls /var/lib

Does this work? Permissions for /var/lib are fine. If not, you need to let others reading it: chmod o+rX /var/lib

sudo -u apache ls /var/lib/nginx

Does this work? If not, check both ownership and permissions via stat. Then ensure NGINX user is the owner of the directory /var/lib/nginx and that the chmod (this time, for "owner") allows it to traverse into the directory:

chown apache:root /var/lib/nginx
chmod u+rX /var/lib/nginx

Ensure the same (owned by NGINX's user, can be read (traversed) by it) for /var/lib/nginx/tmp

And finally for /var/lib/nginx/tmp/fastcgi you will need NGINX user to be able to perform all of read, execute (traverse to), and write:

chown apache:root /var/lib/nginx/tmp/fastcgi 
chmod 0700 /var/lib/nginx/tmp/fastcgi

So basically it's a rinse, repeat operation while traversing down to relevant files until it works.

Verify that everything is set properly by attempting to list contents of the directory and creating files in it:

sudo -u apache ls /var/lib/nginx/tmp/fastcgi
sudo -u apache touch /var/lib/nginx/tmp/fastcgi/test.txt