i have a ubuntu 18.04.1 server running with 8-core processor and 8 Gigabyte's of memory. it's a cloud server which is based on KVM virtualization. i am using haproxy 1.8.8 for load balancing on my servers. problem is when i run load-test on my server with ab or wrk tool i can see that only one of cpu core are filled up to 100% (core7), and that because of too many si(soft interupts), so i checked /proc/interrupts file :
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 30 0 0 0 0 0 0 0 IO-APIC 2-edge timer
1: 0 9 0 0 0 0 0 0 IO-APIC 1-edge i8042
6: 0 0 0 3 0 0 0 0 IO-APIC 6-edge floppy
8: 0 0 1 0 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi
10: 0 0 102 0 0 0 0 24228261 IO-APIC 10-fasteoi virtio0, eth1, eth0
11: 0 0 0 0 0 0 0 32 IO-APIC 11-fasteoi uhci_hcd:usb1
12: 15 0 0 0 0 0 0 0 IO-APIC 12-edge i8042
14: 0 0 0 0 0 0 0 0 IO-APIC 14-edge ata_piix
15: 0 0 0 0 0 0 1453248 0 IO-APIC 15-edge ata_piix
24: 0 0 0 0 0 0 0 0 PCI-MSI 131072-edge virtio1-config
25: 0 0 0 0 15 0 0 0 PCI-MSI 131073-edge virtio1-virtqueues
26: 0 0 0 0 5791 2805745 0 0 PCI-MSI 114688-edge ahci[0000:00:07.0]
NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 14654751 6657243 5811366 5270649 14966993 4797078 5687129 8545399 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 1 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 2572806 2980772 2435576 2151656 1887449 2366833 2404309 1967901 Rescheduling interrupts
CAL: 638862 508650 531191 579853 596146 636037 652622 655700 Function call interrupts
TLB: 62859 43397 20200 6237 4423 11681 18652 4408 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 4706 4706 4706 4706 4706 4706 4706 4706 Machine check polls
HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 0 0 0 0 Nested posted-interrupt event
which shows me there is a lot of interrupts sending from NIC, so i hve tried to some how tune NIC to cause fewer interrupts,
options that i have tried and doesn't have any effect:
- i have tried to disable irqbalance,
- use smp_affinity to distribute irq 10 across multiple cores (that doesn't even worked, irq 10 just stucked with one core no matter how i change smp_affinity, although i have read some articles which say in this situation that won't make performance better)
- increasing size of MTU up to 9000
- increasing rx ring buffer size up to 2048
- and many many sysctl tunings!
i have also noticed that i have many rx error in my eth0 :
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 185.8.174.227 netmask 255.255.255.0 broadcast 185.8.174.255
inet6 fe80::84f9:91ff:fe5e:c862 prefixlen 64 scopeid 0x20<link>
ether 86:f9:91:5e:c8:62 txqueuelen 1000 (Ethernet)
RX packets 19862876 bytes 8071862301 (8.0 GB)
RX errors 1746656 dropped 0 overruns 0 frame 1746656
TX packets 22127410 bytes 13038619281 (13.0 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
here is also my haproxy.cfg global section:
global
nbproc 2
#log /dev/log local0
#log /dev/log local1 notice
chroot /var/lib/haproxy
#stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
#stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 10000
# tune.ssl.default-dh-param 2048
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
#ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
#ssl-default-bind-options no-sslv3
so whats wrong with my server? any help would be accepted.