Intel 10Gbps card bandwidth

Question

I have an Intel 10Gbps card that has the 82599 10GbE Controller. The card has two ports in it. The datasheet of the controller says it supports PCIe 2.0 (2.5 GT/s or 5.0 GT/s)

Now, according to PCIe SIG's faq page (link: https://www.pcisig.com/news_room/faqs/pcie3.0_faq/#EQ3) says that for a 5.0 GT/s symbol rate PCIe gives an interconnect bandwidth of 4Gbps and a per lane per direction of 500MB/s)

I ran a netperf test on the card (I connected two of these cards via OFC back-to-back with no switches in-between) and the bandwidth of around 3.3Gbps (which is around 400MB/s)

Is my card under-utilized or does those numbers add up? Why wouldn't I get a full 10Gbps on the card (and only get 3.3Gbps)

(The card is x4 on an x8 slot)

Update: The network card goes to a slot that is configured as PCIe 3.0 and its an x8 slot (it supports upto 8.0 GT/s). And as to the board itself, well its a Freescale board (Processor: T4240). So I figured that board might be ok, with the card being slower of the two.

Thanks in advance.

Answer 1

There are many reasons why you may not be seeing 10Gbps across the link. I can offer the following:

• PCIe 2.0 offers an effective bandwidth of 4Gbps per lane. A PCIe 2.0 4x card in a PCIe 2.0-or-better 8x slot will have a 4x link, providing 20Gbps of effective bandwidth. This is enough to handle both links being fully saturated assuming the rest of your hardware can handle it.
• Many general-purpose desktop and server operating systems are not configured by default to handle high-bandwidth networking.

To get full performance out of that card, you'll want to:

• Disable anything that will restrict performance of networking or CPU speed/interrupt processing:

Linux Example:

service irqbalance stop
service cpuspeed stop
chkconfig irqbalance off
chkconfig cpuspeed off

• Enable 9K jumbo frames with a high transmit queue length:

Linux Example:

ifconfig eth2 mtu 9000 txqueuelen 1000 up

• Increase the network buffers so that they can keep the card saturated with data:

Linux Example:

# -- 10gbe tuning from Intel ixgb driver README -- #

# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0

# memory allocation min/pressure/max.
# read buffer, write buffer, and buffer space
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 524287
net.core.wmem_max = 524287
net.core.rmem_default = 524287
net.core.wmem_default = 524287
net.core.optmem_max = 524287
net.core.netdev_max_backlog = 300000

There is further tuning you can do to the PCI link, such as bumping the maximum block size to 4K. Properly tuned, you should be able to push about 9.90Gbps across each link.

Keep in mind that server and client, and every hop along the way (switch/router) must be similarly tuned in order to not bottleneck the data flow.

Answer 2

Same same here... turns out it is because the 10Gbps protocol revived the old modem encoding .. with a start/stop bit and 8 bits of data.

Today's rate:

R:512+0 records in
512+0 records out
4294967296 bytes (4.0GB) copied, 6.37415s, 642.6MB/s
W:512+0 records in
512+0 records out
4294967296 bytes (4.0GB) copied, 6.78951s, 603.3MB/s

(this is run on a Win7 client talking to null files on the linux end -- /dev/zero for reads, and /dev/null for writes).

For 'smb/cifs' and a single client, bonding 2 cards together doesn't help throughput (since smb/cifs is a 1 connection/client protocol). :-(

p.s.-This was not, BTW true on 1Gb and I don't think it is true on 40Gb... Lame! Feels like the diskspace MB!= 1024**2 Bytes issue when it first came out ... a way of making it sound better than it actually is...