We have been experimenting rolling out 3.10 series kernel that's provided by centos via the centosplus repo for v6.
We have been noticing something interesting where under the same load the load for the server is 6-7x more on the 3.10 series then then stock 2.6 series. Using top we noticed this process using a lot of CPU
[kworker/u66:2]
That lead us to install powertop and then saw this
Usage Events/s Category Description
1110 ms/s 2045.2 Process php-fpm: pool www
36.0 ms/s 2165.4 Timer tick_sched_timer
57.7 ms/s 1285.0 Process nginx: worker process
13.3 ms/s 416.0 Timer hrtimer_wakeup
39.1 ms/s 350.7 Interrupt [3] net_rx(softirq)
This is the same on a 2.6 series getting the same amount of traffic
Usage Events/s Category Description
1795 ms/s 1654.0 Process php-fpm: pool www
45.3 ms/s 1110.4 Process nginx: worker process
562.8 µs/s 122.4 Process /usr/bin/java -Xms200m -Xmx2000m -Xss256k -XX:MaxDirectMemorySize=516m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Dage
497.1 µs/s 59.3 Process /usr/sbin/gmond
16.0 ms/s 30.2 Process /usr/bin/redis-server 127.0.0.1:6379
4.7 ms/s 32.8 Process python /usr/bin/statsd-relay.py
81.7 ms/s 0.00 Timer tcp_delack_timer
24.8 ms/s 0.00 Timer tick_sched_timer
549.4 µs/s 9.2 Process java -Xmx6g -server -Dfile.encoding=utf-8 -XX:OnOutOfMemoryError=kill -9 %p -XX:+HeapDumpOnOutOfMemoryError -XX:HeapD
15.2 ms/s 0.00 Interrupt [3] net_rx(softirq)
As you can see the net_rx is 0 on 2.6 but we get as many as 4k/s on 3.10. The server specs are the same and removed all sysctl settings. I can replicate the issue just by installing 3.10 on a server.
the nics we have in are
06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
