11

On our R610s with igb version 4.1.2 I do see the ioatdma module loading (and loaded with lsmod):

ioatdma: Intel(R) QuickData Technology Driver 4.00
ioatdma 0000:00:16.0: irq 80 for MSI/MSI-X
ioatdma 0000:00:16.1: irq 81 for MSI/MSI-X
ioatdma 0000:00:16.2: irq 82 for MSI/MSI-X
ioatdma 0000:00:16.3: irq 83 for MSI/MSI-X
ioatdma 0000:00:16.4: irq 84 for MSI/MSI-X
ioatdma 0000:00:16.5: irq 85 for MSI/MSI-X
ioatdma 0000:00:16.6: irq 86 for MSI/MSI-X
ioatdma 0000:00:16.7: irq 87 for MSI/MSI-X

We have a quad port Intel nic that should support it:

07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

However, it looks like it isn't being used for any network traffic:

[root@or-rtlb01 ~]# cat /sys/class/dma/dma*/bytes*
0
0
0
0
0
0
0
0

Does anyone know how to make it so ioat is used for network traffic?

Kernel is Linux or-rtlb01.ds.stackexchange.com 3.7.7-1.el6.elrepo.x86_64 #1 SMP Mon Feb 11 22:28:07 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

Kyle Brandt
  • 82,107
  • 71
  • 302
  • 444
  • redhat 6 right? – mdpc Feb 14 '13 at 19:03
  • @mdpc: CentOS 6.3, tried a much newer kernel to see if it would help, but it didn't. This was a secondary issue discovered when having high si% on ksoftirqd with ipsec traffic (not that I expect offloading on the ipsec traffic itself, but would seems there is no ioatdma at all. – Kyle Brandt Feb 14 '13 at 20:00
  • 2
    Does the motherboard chipset have support for IOAT? Is IOAT turned on in the system BIOS? Do you see "Intel(R) I/OAT DMA Engine found" in dmesg? This would indicate a NIC which supports IOAT has been located. Do you see "DCA is disabled in BIOS" in dmesg? This would indicate no BIOS support or enablement for DCA. – suprjami May 22 '13 at 13:10
  • When I last asked Dell (several years ago) they said the R610 BIOS does not support DCA nor do they have plans to add support for it. – Craig Jun 21 '13 at 22:49
  • 2
    Right, as guys stated what does "dmesg |grep DCA" show? – Danila Ladner Aug 31 '13 at 19:54

2 Answers2

4

How big are the packets you're sending through? By default the IOATDMA driver is only used on large packet sizes. You can view the current setting with:

cat /proc/sys/net/ipv4/tcp_dma_copybreak

If you want the value to be lower, you can set it using sysctl. For example, to set the packet size to 1, use:

sysctl net.ipv4.tcp_dma_copybreak=1
Boscoe
  • 564
  • 3
  • 6
2

Before starting, update complete server like BIOS including hard drive firmware.

Secondly, always use latest Intel Network Driver igb. Just compile it from the source.

BIOS enable both DCA and IOATDMA. In linux try this:

# modprobe ioatdma

# dmesg

You should see the ioatdma driver loaded.

Now, the igb driver is different to ixgbe which is 10GBps one. Since I dont use igb at all, I can tell you to try to stop network /etc/init.d/network stop and rmmod igb and modprobe igb again to see if there are any differences and post results if you can pls.

Andrew Smith
  • 1,123
  • 13
  • 23