1

first time dealing with 10gbe.

I have two boxes, one is older with i7 2.8ghz from 2009 generation, pcie 2.0 and I just put the Intel X550T there to make use of absolutely similar but builtin adapter on my newer box with AMD EPYC gen 1.

I'm dling file from tmpfs on either box and never go above 415mbyte/s which is about half of potential bandwidth there, right?

I tried most stuff from https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf with a little improvement from 393mbyte/s to current 415. that's the result of me setting mtu 9000, txqueuelen 10000.

Also tried this to no improvements over current result: https://darksideclouds.wordpress.com/2016/10/10/tuning-10gb-nics-highway-to-hell/

And a few other pages out there on google, most stating same options.

I don't see a 100% load burst on i7 box when I dl file, it's around 10% across the cores. So CPU shouldn't be a bottleneck, cable(it's 1meter cat8, with a nice thick shielding) shouldn't be a bottleneck and ram speed and pcie 2.0 speed shouldn't be. iperf also reports same speed.

Tried these tests in both directions.

There's gotta be something obvious that I'm missing, right?

Pretty sure it doesn't matter but it's lfs on i7 and debian 10 on epyc, both sharing same sysctl settings and ethtool-controlled options.

lspci -n for it on both computers: 62:00.0 0200: 8086:1563 (rev 01) so it's similar device. I bought this particular one for compatibility but didn't expect these issues.

Maybe someone here would suggest something, I'm out of ideas.

TIA

tinfoil3d
  • 31
  • 3

1 Answers1

2

Solved, not a software problem, my computer has way too many drives, PCI legacy devices and a GPU too, that leaves me without x4 available PCIe lanes for the card, it is throttled down to x1, which on my PCIe 2.0 MB gives me 4gbit/s max speed which is exactly what i'm seeing.

[    4.559030] ixgbe 0000:02:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:05.0 (capable of 31.504 Gb/s with 8 GT/s x4 link)

There is no beautiful solution for this, and no other CPU with socket 1156 looks any more promising with 16 PCIe lanes max so potential work-arounds include:

  1. checking BIOS menu to see if you can manually set longer PCIe slots lane count to further downgrade GPU(which is already downgraded to x8 at my system) to x4 or x1.

  2. using x16->x1 riser that miners use at their rigs for GPU.

Both solutions would free up x4 for 10gbit NIC. Yet both solutions, on PCIe 2.0 would result in dramatic drop in GPU performance and gaming and HD video would likely be impossible. It is sufficient for mining though.

My best wishes to intel. Thanks a lot for your continued support of 16-lane CPUs. It was, is, and always will be like this with you. Compare my i7 from legacy generation from 2000s to the top notch https://ark.intel.com/content/www/us/en/ark/products/202329/intel-core-i7-10875h-processor-16m-cache-up-to-5-10-ghz.html

Mark my words, in 2040 they'll still be producing 16 lane CPUs.

Special thanks goes to the-architect from unix.com forums who gave me an idea to check ALL of the dmesg not just the "link up/down" messages which in turn lead me to this conclusion.

Peace.

tinfoil3d
  • 31
  • 3