2 of 16 cores are constantly maxing out CPU capacity - why?

Question

I have nginx with php-fpm running on a 16 core Ubuntu 16.04 instance. The server is handling more than 10 million requests per hour.

As you can see on the htop screenshot cores 6 and 7 are maxed out and that's the case constantly - even after restarting nginx those two cores stay at that level.

I wonder why is that so and how to balance the load more evenly?

> cat /etc/nginx/nginx.conf | grep -v '^\s*#'

user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
        worker_connections 768;
}
http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        gzip on;
        gzip_disable "msie6";
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

That's going to be syscalls, i.e. kernel code. Are you sure you've configured irqbalance correctly? How many processor sockets are in use? — Michael Hampton, Jan 09 '18 at 17:44
@Tim: both - I observe behavior like not served requests recently — Raffael, Jan 10 '18 at 09:38
@MichaelHampton: I didn't configure irqbalance, at all. Would appreciate input on that. I'm not sure what you mean with "processor sockets in use" here. — Raffael, Jan 10 '18 at 09:41

score 0 · Accepted Answer · answered Jan 12 '18 at 11:39

A look at /proc/interrupts reveals that the high load on CPUs 5 and 6 (0-based indexing) is due to the handling of the network traffic. Those processes are exclusively handled by these processors.

CPU5: xen-pirq-msi-x eth0-TxRx-0
CPU6: xen-pirq-msi-x eth0-TxRx-1

pidstat (from sysstat package) shows that additionally processes from nginx and php-fpm are also handled by CPU5 and CPU6.

That explains the high load.

2 of 16 cores are constantly maxing out CPU capacity - why?

1 Answers1