2

I have a CentOS 7 server with Memcached. There are a lot of requests, and usually after a few hours, the memcached daemon crashes. I always restart the service, but it crashes again a few hours later.

The /var/log/memcached.log file is empty.

These are the /etc/sysconfig/memcached settings:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="32"
OPTIONS="-vv >> /var/log/memcached.log 2>&1"

journalctl -u memcached throws this result (strangely, the last log dates are from december, but i've been restarting the memcached service every single day in january).

Dec 10 23:59:32 myserver systemd[1]: Started Memcached.
Dec 10 23:59:32 myserver systemd[1]: Starting Memcached...
Dec 13 16:44:07 myserver systemd[1]: memcached.service: main process exited, code=killed, status=9/KILL
Dec 13 16:44:07 myserver systemd[1]: Unit memcached.service entered failed state.
Dec 13 16:44:07 myserver systemd[1]: memcached.service failed.
Dec 14 22:22:41 myserver systemd[1]: Started Memcached.
Dec 14 22:22:41 myserver systemd[1]: Starting Memcached...
Dec 15 19:35:50 myserver systemd[1]: memcached.service: main process exited, code=killed, status=9/KILL
Dec 15 19:35:50 myserver systemd[1]: Unit memcached.service entered failed state.
Dec 15 19:35:50 myserver systemd[1]: memcached.service failed.

systemctl cat memcached.service

[Unit]
Description=Memcached 
Before=httpd.service
After=network.target

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/memcached
ExecStart=/usr/bin/memcached -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTION

[Install]
WantedBy=multi-user.target

After a few hours of visitor requests, the apache log error starts displaying this errors, one after the other:

[Wed Jan 08 11:17:11.017280 2020] [:error] [pid 15793] [client X.X.X.X:22992] PHP Warning:  Memcache::connect(): Can't connect to localhost:11211, Connection refused (111) in /www/html/inc/functions.php on line X

These are the memcached settings with a php 5.6.30 environment:

versions

How could I properly detect what's causing the fail/crash behavior?

Andres SK
  • 228
  • 3
  • 7
  • 21
  • 3
    I'd start by deleting `>> /var/log/memcached.log 2>&1` from `OPTIONS` but keep `-vv`as you are using systemd. It should then start logging again (to journal, not to logfile). What's the status of the service in stopped state `systemctl status memcached`? – Lenniey Jan 08 '20 at 17:56
  • @Lenniey after failing/crashing, the status is: `Loaded: loaded (/usr/lib/systemd/system/memcached.service; enabled; vendor preset: disabled) / Active: failed (Result: signal) since Wed 2020-01-08 12:30:05 -05; 2h 51min ago / Process: 21242 ExecStart=/usr/bin/memcached -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS (code=killed, signal=KILL) / Main PID: 21242 (code=killed, signal=KILL)` - i already removed the OPTIONS values except -vv, but after restarting memcached, the journal is still not logging anything. – Andres SK Jan 08 '20 at 20:24
  • 1
    Can you please provide 'systemctl cat memcached.service'. Also, how are you restarting memcached? My own experience with memcached taking down a server has been with a Java client which exhausted the number of file-descriptors. – Cameron Kerr Jan 08 '20 at 20:43
  • if you've removed the logging-to-file, and its not appearing in the journal, then check syslog, you may find it in /var/log/messages or similar. – Cameron Kerr Jan 08 '20 at 20:45
  • @CameronKerr Im using `systemctl restart memcached` to restart the service. I added the result of the systemctl cat command in the original post for you to check it. Thanks. – Andres SK Jan 08 '20 at 20:49
  • @CameronKerr the current `/var/log/messages` file is also empty. – Andres SK Jan 08 '20 at 20:49
  • 1
    You can see that its being killed with SIGKILL (kill -9 PID), so if that's not something you've been doing, something else may be doing that (eg. is it using more memory than some configuration management system is allowing?) – Cameron Kerr Jan 08 '20 at 20:52
  • @CameronKerr isn't memcached supposed to delete old key values when adding new keys if the CACHESIZE is already being used to its maximum? – Andres SK Jan 08 '20 at 21:01

0 Answers0