php session_write_close warnings

Question

All our errors are logged to NewRelic, and we always saw a few warnings of session_write_close in the error log. However, the error rate is increased and it is now flooding our 24h log.

Our server is highly populated and lots of users are logged in at the same time. Most of these users don't don't see these session_write_close warnings. Some do, which makes it almost impossible for us to find the cause and fix it.

This is the full error message:

Error message
E_WARNING: session_write_close(): Failed to write session data (files).    
Please verify that the current setting of session.save_path is correct        
(/opt/php55/var/lib/php/session-nginx)

So I did a check, see how many files are in that directory 9431 and what the rights were -rw------- 1 nginx nginx.

I don't see anything wrong with my configuration, file rights etc..

We are out of options. What can I do to resolve this issue? Currently affecting < 1% of our users, we just want to keep our rate as low as possible.

Here is a list of my php.ini configuration.

Directive   Local Value Master Value
session.auto_start  Off Off
session.cache_expire    180 180
session.cache_limiter   nocache nocache
session.cookie_domain   no value    no value
session.cookie_httponly Off Off
session.cookie_lifetime 0   0
session.cookie_path /   /
session.cookie_secure   Off Off
session.entropy_file    /dev/urandom    /dev/urandom
session.entropy_length  32  32
session.gc_divisor  1000    1000
session.gc_maxlifetime  1440    1440
session.gc_probability  1   1
session.hash_bits_per_character 5   5
session.hash_function   0   0
session.name    PHPSESSID   PHPSESSID
session.referer_check   no value    no value
session.save_handler    files   files
session.save_path   /opt/php55/var/lib/php/session-nginx    /opt/php55/var/lib/php/session-nginx
session.serialize_handler   php php
session.upload_progress.cleanup On  On
session.upload_progress.enabled On  On
session.upload_progress.freq    1%  1%
session.upload_progress.min_freq    1   1
session.upload_progress.name    PHP_SESSION_UPLOAD_PROGRESS PHP_SESSION_UPLOAD_PROGRESS
session.upload_progress.prefix  upload_progress_    upload_progress_
session.use_cookies On  On
session.use_only_cookies    On  On
session.use_strict_mode Off Off
session.use_trans_sid   0   0

Some server stats: CentOS 6.6 PHP 5.5.28 Nginx 1.6.2 Any help is welcome!

Answer 1

With a highly loaded server, I'd use memcached (maybe even redis?) for session storage. So if I were in your situation I'd probably just set that up for its own sake, and then see if the problem had just fortuitously go away.

I also wouldn't use php's session garbage collection, which hangs the garbage collection off web request jobs. I'd set up my own job to handle that, either running from cron, or from some job queuing system.

Do you already have any sort of session clean-up system outside of php's session garbage collection?

Is the rate at which this is occurring 0.1% of the time, which would line up with your session.gc_divisor setting?

Are your php processes running as the nginx user? It's php rather than nginx that does the cleanup based on the session.gc_* settings. If php is running as nginx, that's good in terms of access to the php session files, but probably bad in terms of sharing a user id with the nginx server.

You might need the execute permission on that session directory so your garbage collection can see what's there to clean up.

I'd also be concerned if you are not setting session.save_path to something specific to your application. That would mean that if you have multiple applications sharing the same session directory, then when garbage collection runs, the app with the shortest expiry wins, cleaning out the sessions of the other app.

Answer 2

The obvious observation I made from your question, is that you've got too much of a bottleneck trying to save files to /opt/php55/var/lib/php/session-nginx. So your solution is to ease up the bottleneck, first diagnosing what specifically is going wrong.
Assuming it's racing, to write to disk, and the errors are a sign of it giving up, I'd expect dmesg errors showing issues writing to disk. If that is the case, you can write to memory, or other solutions which amount to a faster 'disk'.
mc0e mentiones memcached instead of using save_handle=files, that's a good option. An alternative to memcached, might be using tmpfs, which essentially puts the session in memory just the same (so has fast write times), but doesn't need a fresh app.
I'd also ask the question what kind of file system is at /opt/php55/var/lib/php/session-nginx? You don't need all the complex journaling of ext3/4 for what is basically a mktmp type operation. You might want to create a folder in /tmp and symlink it into place to ensure less overhead in creating files.
What's the hardware setup? If it's a single disk without caching, you should see issues in dmesg if you're peaking at the limit of it's performance. I used AMCC Raid controllers with Raid-1 in all of my servers. If it's Raid-1 (mirror) it'll be fast to read, but write speed will be dependent on how well the raid is implemented (I know AMCC can spread writes across disks in Raid-1, but not all RAID-1 implementations do that, I know software raid doesn't). My old boss swore by Raid-5 for this reason, and as long as it's real hardware raid (RAID-5 can be expensive on the CPU if it's not), that'll drastically speed up your disk throughput. Another option is a solid state disk, but really if you're going down that path, I'd suggest using memcached or tmpfs, as more memory is always a good plan (over any other new hardware).

The simplest solution though, is going to be to create /tmp/session-nginx and symlink or mount /opt/php55/var/lib/php/session-nginx to /tmp/session-nginx/

Answer 3

Part of the question is about the difficult to track those kinds of errors, then I may suggest to explicitly close the session on your code within a try/catch block. Handle the exception, sleep and try again.

The other part of the question describes an write error that seems to be random in nature. This is not what I expect from incorrect permissions. I would suspect that you have too many open files.

There is a few configurations I would tweak to see what happens:

1. increase the limit of open files you may have some lower limit set on some part of your os. For example, my notebook supports hundred of thousands open files, but only 4000 from the same user.

2. lower the maxrequestperchilds to 1000 this will make each http server to restart after serving 1000 clients.

3. reduce MaxClients and increase ListenBacklog. This is very, very counterintuitive, but if you set MaxClients/Servers too high, a lot of processes will fight for resources in your server and cause bottleneks. That depends heavly on the kinds of bottleneks you have. Mine is the database servers.