Redis server high cpu debugging strategies

Question

Recently, we have been noticing CPU spikes on our production environment caused by redis which can be seen below:

To combat this issue, I have been restarting the redis server about twice a day :( which is obviously far from ideal. I'd like to identify the root cause.

Here are some things I have looked into so far:
1) Look into any anomalies in the redis log file. The following seems suspicious:

2) Researched nginx access logs to see if we are experiencing unusually high traffic. The answer is no.

3) New Relic revealed that the issue started on Nov 21st, 16` (about a month ago) but no code was released around that time.

Here are some details about our setup:

Redis server: Redis server v=2.8.17 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=64a9cf396cbcc4c7

PHP: 5.3.27 with fpm

Redis configuration:

daemonize yes
pidfile /var/run/redis/redis.pid
port 6379
timeout 0
tcp-keepalive 0
loglevel notice
logfile /var/log/redis/redis.log
syslog-enabled yes
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error no
rdbcompression yes
rdbchecksum yes
dbfilename redis.rdb
dir /var/lib/redis/
slave-serve-stale-data yes
slave-read-only yes
repl-disable-tcp-nodelay no
slave-priority 100
maxmemory 15GB
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-max-len 128
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
include /etc/redis/conf.d/local.conf

Framework: Magento 1.7.2 with Cm_Cache_Backend_Redis

Please let me know if given the above information there is anything I can do to mitigate the high cpu usage.

Answer 1

VERY IMPORTANT UPDATE:

Your server may have been hacked. It’s not redis that is causing the high CPU usage, but a separate command called yam (take a look at the far right of your htop, I missed it the first time). The yam command is used in a well-known exploit of redis and often results in high CPU usage. You’ll want to double-check to make sure your server is secure.

Here are some articles and links you can refer to if you want to learn more about the vulnerability and how to secure yourself:

• https://gist.github.com/lokielse/d4e62ae1bb2d5da50ec04aadccc6edf1 (a sample lady hack script, in the comments you can see where an original version downloaded yam)
• https://translate.google.com/translate?hl=en&sl=zh-CN&u=http://www.zkreading.com/article/879816.html&prev=search (a translation of a chinese article on the exploit and how to exploit it)
• https://translate.google.com/translate?hl=en&sl=zh-CN&u=http://blog.csdn.net/u012573259/article/details/51803447&prev=search (another chinese post more focused on next steps and clean up)
• http://antirez.com/news/96 an opinionated blog post on redis security describing the means of intrusion that may have been the source of ingress into the OP's server
• https://www.riskbasedsecurity.com/2016/07/redis-over-6000-installations-compromised/ (an overview of how many servers have been affected by poor redis security)
• https://news.ycombinator.com/item?id=13053647 (a thread reporting the bitcoin flavor of the hack)

---

Here is my checklist for magento/redis, er, performance issues:

1. Make sure you are on a newish version of redis, like 3.2, I personally prefer redis32u from the IUS repository if on CentOS.
2. Check the size of your redis database, it should be in /var/lib/redis, and make sure it is relatively small.
3. Verify that you have have enough ram for redis. You’ve specified a maxmemory of 15GB, which is really overkill for magento. I typically use something closer to 256mb. If you are using redis that much (!!!!!!), you likely have other problems in your magento stack.
4. Make sure you have vm overcommit setting set in syscntl. https://redis.io/topics/admin (see this link for more details on what you need)
5. Make sure you have sufficient open file limits to handle the number of connections to redis.

Generally speaking, the log file isn’t suspicious because your redis save settings tell redis to save every minute if there have been > 10000 write, every five minutes if there have been > 10 writes, and every 15 minutes if there have been > 1 write. So it is essentially persisting the info back to disk every minute, which shouldn’t be that burdensome.