Apache 2.4 Event MPM + php-fpm + php-opcache yields "Connection reset by peer" errors

Question

I have a CentOS 7 server running Apache 2.4.6 with Event MPM and php-fpm version 5.6.10 (from REMI repo). Here's the relevant Apache config to send requests to php-fpm within the vhost (it's the only site on this server):

<FilesMatch \.php$>
    SetHandler "proxy:unix:/var/run/php-fpm/www.sock|fcgi://localhost"
</FilesMatch>

Here's the relevant php-fpm pool conf:

listen = /var/run/php-fpm/www.sock
listen.owner = apache
listen.group = apache
pm = static
pm.max_children = 50
pm.max_requests = 0

Here's the php-opcache config (the default installation config):

zend_extension=opcache.so
opcache.enable=1
opcache.memory_consumption=128
opcache.max_accelerated_files=4000
opcache.blacklist_filename=/etc/php.d/opcache*.blacklist

My problem: Whenever I install and enable php-opcache, the following errors start appearing in my error log:

[Thu Jun 18 08:10:58.499871 2015] [mpm_event:debug] [pid 16546:tid 139798115227392] event.c(986): (104)Connection reset by peer: [client 157.55.39.233:15962] AH00470: network write failure in core output filter
[Thu Jun 18 08:10:58.527424 2015] [mpm_event:debug] [pid 16546:tid 139797922195200] event.c(986): (103)Software caused connection abort: [client 157.55.39.233:15990] AH00470: network write failure in core output filter

If I remove php-opcache, the errors cease. Users are reporting a 502 Service Unavailable error on the frontend whenever this occurs.

I've literally spent at least 6 hours trying to Google and find solutions. Someone said that opcache's fastshutdown option was a problem, but that's disabled in the default config. I changed the php-fpm process management method to static with no recycling (max_requests=0), but that did not change anything either. I also tried using TCP proxy method with Apache (instead of sockets), and that also had no affect.

I'm not sure if this is relevant or not, but regardless if php-opcache is installed or not, I get the following errors reported in the error log (but at a much smaller frequency, <0.5% of all traffic, which may be a separate issue):

[Thu Jun 18 08:32:37.223430 2015] [proxy_fcgi:error] [pid 19187:tid 140598765840128] [client 37.46.115.238:55624] AH01068: Got bogus version 10, referer: ...
[Thu Jun 18 08:32:37.223462 2015] [proxy_fcgi:error] [pid 19187:tid 140598765840128] (22)Invalid argument: [client 37.46.115.238:55624] AH01075: Error dispatching request to :, referer: ...

This issue is very similar to this one, although that person is using ProxyPassMatch with TCP method, which I am not (because is bypasses .htaccess).

Anyone have any ideas that I haven't already mentioned?

Answer 1

When I've seen similar issues in our environments, it appears to have been because of the way that OpCache (by default) shares a single cache across all users on a shared hosting environment. A bug has been submitted (and you can, and should, go and vote to let the maintainers know how important this might be for your use case) though no commitment has been made on delivering a fix.

TL;DR : By default when OpCache is enabled, the cache that is used to store compiled byte-code is shared across all users. In an environment where hosting is shared between multiple sites / users, this can result in a site grabbing the cached output of php scripts from another site or, if specific security settings are enabled even generating errors.

If you plan on using PHP-FPM with PHP 5.5+'s built in opcache, please read the blog post below before you actually do that. Turns out that the opcode cache can be read by any user on the server. This means that if there are say, 10 separate users, with their own vhosts and directories, and you configure one PHP-FPM pool per user, each user can still see what scripts are cached and their locations. Since they have read access to the cache they could potentially view all this data.

This is obviously a massive security concern, and even if no one exploits this, there is still a chance of scripts being read by the wrong user when generating a page, so websites could possibly be displaying the wrong data / info if there are multiple index.php scripts in the cache.

Although no fix has been officially released, if you're using cPanel, this wiki has a documented way of configuring the php-fpm pools to be created and secured on a per-user basis and if you follow the instructions below as well as the IMPORTANT NOTES at the bottom of this answer you should be able to get he functionality you desire without any errors

That post also documents how you might configure this by hand on a per-site/per-user basis (though I'd wager that might become tedious if you're hosting a lot of sites). If you're not using cPanel, you may need to modify the scripts to specify your individual paths and usernames instead of the variables being used by cPanel's config engine.

---

IMPORTANT NOTES

During testing and additional research I came across this article which provide a few clarifications that may be relevant to your specific situation:

1. You need to make sure that the opcache.use_cwd parameter is set to true for your application's configuration of OpCache - it's set to false by default and leaving it set to default will probably cause collisions if you're hosting more than one PHP application on your system:

First of all, probably in each typical project you will have to ensure that the opcache.use_cwd option is set to true. Enabling this setting means that the OpCache engine will look at the full file paths to distinguish between files with the same names. Setting it to false will lead to collisions between files with the same base name.

2. If you're running an application powered by Zend Framework or another similar framework that makes use of annotations, you ALSO need to ensure that the opcache.load_comments and opcache.save_comments directives are set to true. You should double-check this suggestion with your application / framework documentation as most have now updated their docs with specific instructions on enabling the use of OpCache properly for their systems:

There is also a setting that is important in tools and frameworks that make use of annotations. If you use Doctrine, Zend Framework 2 or PHP Unit, remember to set the opcache.load_comments and opcache.save_comments settings to true. In result, the documentation comments from your files will also be included in the precompiled code generated by OpCache. This setting will allow you to work with annotations without any disruptions.

If your project is based on a specific framework or a web application, it’s always a good idea to check the documentation for any guidelines regarding the OpCache configuration

IMPORTANT NOTES

---

Hopefully this helped - and if you're using cPanel, drop a comment to let us know how you tackled that portion of the configuration! See also this question & associated comments.