netperf udp with small packets on 10g link is bounded by ksoftirqd

1

1

I want to fill a 10Gbps link with very small udp packets (64B).

I use the following command

 for i in 1 2 3 4 5 6 7 8; do netperf -H 192.168.56.2 -p 12865 -t UDP_STREAM -l 60 -C -c -- -m 64 -s 16M -S 16M -R 1 &  done

But I cannot get more than 250Mbps throughput. For small messages I saw that a thread called ksoftirqd is using 100% of a CPU core at the receiver. So my question is

  • How to lower the cpu usage of ksoftirqd for small messages?

Current setting: I have two ubuntu machines (8 cores and 16 cores) with Intel X520-T2 NICs and kernel 3.11 connected with a Juniper 10G switch. These are the changes I have for sysctlcfg.conf

net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.netdev_max_backlog = 250000
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_congestion_control = htcp

I also increased txqueue size of NIC

/sbin/ifconfig eth0 txqueuelen 10000

I looked for interrupt coalescing using rx-usecs parameter but increasing that worsened the throughput.

Masood_mj

Posted 2014-10-16T23:10:15.560

Reputation: 123

No answers