3

We have a machine with an Intel x550 and a Cisco Nexus N9K-C9372-PX-E.

For whatever reason, it will only negotiate at 1Gb, even though it is 10Gb capable and is being plugged into a 10Gb switch. This is on an Asus x299 10Gb board.

We have about 15 other machines with a mixture of Intel x550, Intel x540 and other various 10Gb cards that do negotiate fine on this same switch. All negotiate (slow to negotiate but does) to 10Gb without problems except this one machine with its onboard X550. To me this would rule out the switch as an issue.

To rule out cabling issue I pulled a cable from a current working 10Gb machine and used the working cable and plugged it into the machine I’m having trouble with. I got the same result, the non working machine (X550 ports) negotiates at 1Gb and not the expected 10Gb.

However, if I plug the non working X550 directly into another machine with a 10Gb NIC, both machines will negotiate at 10Gb without issue.

I put an Intel X540-T1 card into the machine and that card negotiates fine to 10Gb. If I move the cable to the on board X550 ports, those will only negotiate to 1Gb

I’ve upgraded the non working machine to the latest 18.04 Ubuntu. I've upgraded the networking card drivers to test, I've upgraded the motherboard BIOS, I've upgraded the x550 firmware. Not sure what else the issue could be.

Any suggestions?

Kevin Vasko
  • 185
  • 5
  • I'd be looking to swap the systemboard to see if that helps. – Chopper3 Jan 14 '19 at 16:59
  • @Chopper3 The argument that the supplier gives is that it is "working" correctly is that it works when you plug it in directly into a different 10G port (e.g. port on a different computer). – Kevin Vasko Jan 14 '19 at 17:34
  • Sorry, so let me get this straight - if you plug this machine into a port that's currently working at 10Gb for another machine then it works, and if you plug this 'odd' machine into any other switch port then it works too? – Chopper3 Jan 14 '19 at 17:45
  • @Chopper3 No. Working PC with 10Gbps to switch, connection establishes as 10Gb (as expected). Take cable from working PC, plug into "odd" machine, connection establishes as 1Gb (not as expected). Take cable run it directly from "odd" machine, directly to Working PC (by-passing switch) both the Working PC and the "odd" machine negotiate at 10Gbps. – Kevin Vasko Jan 14 '19 at 17:57
  • That is odd, could you take the cable currently plugged into the 'odd' machine and plug it into the currently good machine to see what happens? – Chopper3 Jan 14 '19 at 18:33
  • @Chopper3 Its one cable. Plug into "good" machine and to switch, get 10Gbps. Pull same cable from "good" machine to "odd" machine and keep it going to switch, get 1Gbps. Pull that same cable from switch (leave it plugged into "odd" machine), plug into "good" machine, get 10Gbps. `"good" machine <--> Switch (10Gbps)` `"odd" machine <--> Switch (1Gbps)` `"odd" machine <--> "good" machine (`10Gbps)` `"<-->"` is the same cable. – Kevin Vasko Jan 14 '19 at 18:38
  • Are we talking about problems with LOM soldered to the motherboard here? I am not sure why you assume it's X550. According to this ASUS page: https://www.asus.com/microsite/mb/cross/X299/ only one design has 10G ethernet and its Aquantia AQC-107... – Tomek Jan 14 '19 at 19:26
  • @Tomek It is this motherboard. https://www.asus.com/us/Motherboards/WS-X299-SAGE-10G/specifications/ `Intel® X550-AT2 , 2 x 10 Gigabit LAN Controller(s)` – Kevin Vasko Jan 14 '19 at 19:28
  • This looks like FW or driver issue. BTW, what OS are you using and what driver version? I am excluding network cable as you tried one which works with different NIC... Ah, one more thing - have you checked if there are any switch FW updates available as well? – Tomek Jan 14 '19 at 19:58
  • @Tomek Ubuntu 18.04 latest kernel. I also tried compiling and manually upgrading the driver version (5.5.3) from intel with no luck. – Kevin Vasko Jan 14 '19 at 20:45
  • @Tomek I have not tried any FW switch updates. That is something that I'll try next. – Kevin Vasko Jan 14 '19 at 20:46
  • You can give it a try with Windows (even with WinPE) and see if and how it links there. But at this point I am leaning towards FW issue on either side of the cable... You can try to fiddle with `advertise` bits in `ethtool -s` and check if it gets you anywhere as well. – Tomek Jan 14 '19 at 20:52
  • @Tomek I've tried to force the advertise (I assume this is the same as auto-negotiate) to 10000. It "links" and states its link speed is 10000 however, it will only transfer at 1Gbps. I'm going to have to dig into figuring out how to update this Cisco switch to the latest version. I assume you mean the latest OS for the switch or do you specifically mean a FW update? – Kevin Vasko Jan 14 '19 at 20:56
  • Yes, I mean Cisco switch OS, it should have all of firmwares embedded (at least this is how I understand it but it has been some time since I've used any Cisco switch). – Tomek Jan 14 '19 at 20:58
  • @Tomek Ok slight update. For whatever reason in the “odd” PC if I plug it up it will automatically negotiate at 1Gb. If I now do ‘ethtool -s enp179s0f0 advertise 0x1000’ wait for it to link again, it will now get the full 10Gbps in iperf. Previously I would get a 10Gbps link but it would only transfer at 1Gbps. So this is new, and I guess changed after me doing updates. Just to be clear I don’t think it’s the switch with auto negotiate bc this happens on 3 known working ports on the switch that _will_ auto negotiate 10Gbps with other good PCs. If I unplug this odd plug back in it only gets 1g. – Kevin Vasko Jan 14 '19 at 22:42
  • I've got the information this is very similar to some other issue reported against other distro but with very close HW setup. This is being worked on. – Tomek Jan 15 '19 at 09:33
  • @Tomek Not quite following you? Some other information, I booted a live Fedora 29, which has 4.18 kernel. Same exact issue. This "odd" machine has a x299 chipset. The "good" machine has a x99 chipset. – Kevin Vasko Jan 15 '19 at 15:39
  • i have exactly the same problem with a Dell 740xd with the Intel X550's. only negs at 1gbps. did you ever solve this problem beyond the ethtool advertise solution? – yee379 Apr 27 '19 at 00:21
  • @yee379 I did not. I think it’s a driver issue but I can’t get Intel to look into it any further. – Kevin Vasko Apr 28 '19 at 02:57
  • thanks @KevinVasko. in the end i added the `advertise` limit to rc.local so at least it persists on bootup. not a great solution, but works nonetheless. – yee379 Apr 28 '19 at 03:41
  • @yee379 What OS/kernel were you running? – Kevin Vasko Apr 29 '19 at 21:36
  • rhel7.6 (3.10.0-957.12.1.el7.x86_64) – yee379 Apr 30 '19 at 22:02
  • @yee379 Yeah and I'm on 4.x+ with Ubuntu 16.04 and Ubuntu 18.04. What driver are you using? – Kevin Vasko May 01 '19 at 13:26

0 Answers0