9

Can not find anything to that ;)

Building a Hyperconvergend Cluster as development system. This is for us state of the art.

Got 2 machines for a start, with a switch ordered to come in about 4 weeks. Both servers have 2x100gb ethernet cards from Mellanox (ConnectX 4)

Plugging a cable directly between them I get a link down. I see that the link trie(d) to come up, but failed, connection breaks down immediately.

Is direct server connection between two ethernet cards not support by QSFP28? Anything else i should be aware? Worst case I "box" the 100g link until the switch arrives.

TomTom
  • 50,857
  • 7
  • 52
  • 134

3 Answers3

11

There's no old-style crossover patch cord or whatever. Properly working passive copper (You don't use optics, do you?) QSFP link should work regardless of you switch Vs switch-less use. Are you using one of those or is it anything third-party?

http://www.mellanox.com/products/interconnect/ethernet-direct-attach-copper-cables.php

Reference Mellanox thread (unanswered so far).

https://community.mellanox.com/thread/4125

BaronSamedi1958
  • 12,510
  • 1
  • 20
  • 46
  • Nah, got some from fs.com - this is the first time we go SFP28. 2 servers, 2 cables so far - both are not connecting. Drivers are current. Firmware just got patched yesterday. The cable in question is https://www.fs.com/products/47096.html – TomTom Jun 30 '18 at 16:37
  • 1
    That should be fine.. Maybe raise it as a support issue with your server vendor. You've done pretty much exactly what I'd do.. – Tom O'Connor Jun 30 '18 at 19:04
  • Done, Also i ncontact with Mellanox. – TomTom Jul 03 '18 at 17:49
  • 1
    Confiormed. I removed the 2 cables, plugged Mellanox adapters (QSFP28->SFP28) in and attached 10G cables and voila, works like a charm (albeit with a reduced performance). Cable problem ;) – TomTom Jul 05 '18 at 14:46
  • @TomTom FS.com has vendor-specific transceivers to avoid issues like this - check https://www.fs.com/c/mellanox-qsfp28-to-qsfp28-3370 – pauska Jul 11 '18 at 08:27
4

QSFP28 ports are all alike (just like SFP, SFP+, ...), there's no MDI vs MDI-X pinout like for twisted pair. Therefore, straight cables aren't used at all. DACs have an integrated crossover, there's nothing to worry about.

Possibly your DAC isn't compatible with the NICs, some are very picky about the branding. It's also possible that the NICs aren't configured alike. You should check the driver messages on why the link doesn't come up.

Zac67
  • 8,639
  • 2
  • 10
  • 28
  • Could it be that the two NICs are waiting on some kind of initiative from each other? IIRC early TP ethernet cards sometimes behaved like that with no switch, even with a correct crossover cable... maybe that old ghost has returned.... – rackandboneman Jun 30 '18 at 22:38
  • Nice. Sucks. Any idea how to trigger this? Also I see the link once coming up, then disappearing. So, it seems to try to get it up. – TomTom Jul 01 '18 at 03:01
  • @TomTom Does the NIC report the link to come up? Or is it merely the link light coming on and going out immediately after? That might just signal the SFP insertion - watch the far side for an actual link response. – Zac67 Jul 01 '18 at 07:07
  • @rackandboneman The situation is somewhat different - 1st generation auto-negotiation was flawed and sometimes didn't work. A DAC *knows* the link parameters for sure. – Zac67 Jul 01 '18 at 07:10
  • No, i see the link initializing every X minutes in Windows. This is followed by a funny bug in the event log. I will try short circuiting - I have dual port cards, I should b able to use one DAC wih one card and it should come up. Allows me to pinpoint a defective card. – TomTom Jul 01 '18 at 07:56
  • Confiormed. I removed the 2 cables, plugged Mellanox adapters (QSFP28->SFP28) in and attached 10G cables and voila, works like a charm (albeit with a reduced performance). Cable problem ;) DAMN. WOuld never have imagined this is still such a wild place. – TomTom Jul 05 '18 at 14:46
0

Yes, they are all crossover. There are no straight cables in the QSFP28 world.

What is unusual for RJ45 users is that all cages (plugs) have the same pinout, independently of whether they are in a switch, router, host, PC or whatever. So the whole thing works perfectly with one cable type for connecting any type of device among each other.

stonux
  • 1