Squid running out of filedescriptors on CentOS 7

Question

I'm running Squid 3.3 (EPEL) on CentOS 7 and recently I have been getting the following error message in my cache.log

WARNING! Your cache is running out of filedescriptors

I am slightly confused by this because I seem to have ample descriptors available:

squidclient mgr:info | grep 'file descri'
Maximum number of file descriptors:   16384
Available number of file descriptors: 16326
Reserved number of file descriptors:   100

Squid was also compiled with this flag:

--with-filedescriptors=16384

Squid confirms that these are actually available on startup:

2015/08/18 21:11:45 kid1| With 16384 file descriptors available

However this error keeps occurring. Not long after this error is logged the squid process seems to also hit 100% CPU or use nearly all of the system memory up over 90%, causing the internet speed to drop to a crawl or just hang indefinitely. Killing the process and restarting resolves it but eventually it will happen again.

I have a total of 8 GB of memory available, these are the memory/cache related parameters in my squid.conf

cache_dir ufs /var/spool/squid 16000 16 256
cache_mem 1024 MB

I am also using ufdbguard and additional helper plugins for Kerberos and NTLM authentication.

Any advice?

Have you tried solutions like this: http://www.cyberciti.biz/faq/squid-proxy-server-running-out-filedescriptors/ ? — Navern, Aug 18 '15 at 20:31
How much files in your cache directory? find /var/spool/squid -type f | wc -l — Navern, Aug 18 '15 at 20:31
No, because the descriptor value as it is now seems fine? Most examples show the problem when the descriptors are a low value like 1024, this isn't the case here. — James White, Aug 18 '15 at 20:31
Cache value is 43409. I guess that's quite a lot, but I have enough disk space/RAM in theory. — James White, Aug 18 '15 at 20:32
I think you don't understand what file descriptors is. Increase number of open files in /etc/security/limits.conf to 65535 Also provide output of sysctl fs.nr_open — Navern, Aug 18 '15 at 20:39
I'm beginning to understand now, the cache directories value reported is greater than the current descriptors so it needs to be increased right? — James White, Aug 18 '15 at 20:42
Yep. Each file descriptor is integer associated with opened file. Check man 2 open. — Navern, Aug 18 '15 at 20:45
Okey, restart squid and check for result. You should edit sudo vi /etc/default/squid as link says:) — Navern, Aug 18 '15 at 20:47
Yes, but in my case it would be /etc/sysconfig/squid, as CentOS 7 is systemd. — James White, Aug 18 '15 at 20:51
Set as the same value as ulimit, no change. I am getting `NOTICE: Could not increase the number of filedescriptors` in my cache.log now however. I don't know if SELINUX might be blocking it — James White, Aug 18 '15 at 20:54
Yes full restart of squid process with systemctl, I haven't rebooted the server itself however. — James White, Aug 18 '15 at 21:00
Actually i am missing something. Doublecheck with ulimit command that limits for files are correct. — Navern, Aug 18 '15 at 21:06
I am in the same boat. If you run `squid -v` you will see the compile option `--with-filedescriptors=16384`, this is the wall your hitting. I am debating what to do next myself. I am considering just compiling from source and bumping the descriptors. — , Aug 27 '15 at 13:28

score 2 · Accepted Answer · answered Aug 27 '15 at 15:24

The number of file descriptors is set in the systemd unit file. By default this is 16384, as you can see in /usr/lib/systemd/system/squid.service.

To override this, create a locally overriding /etc/systemd/system/squid.service which changes the amount of file descriptors. It should look something like this:

.include /usr/lib/systemd/system/squid.service

[Service]
LimitNOFILE=65536

Do not edit the default file /usr/lib/systemd/system/squid.service, as it will be restored whenever the package is updated. That is why we put it in a local file to override defaults.

After creating this file, tell systemd about it:

systemctl daemon-reload

and then restart squid.

systemctl restart squid

Interestingly `squidclient mgr:info | grep 'file descri'` still reports the same value, despite changing it in systemd. I think the fact its compiled with that value is a problem. — James White, Aug 27 '15 at 21:27

Squid running out of filedescriptors on CentOS 7

1 Answers1