26

We have a number of Supermicro machines with IPMI/BMC features. Some of these machines use an onboard BMC, while others use an add-on card.

We are looking into using sideband due to it's reduced costs and cabling requirements. However, some sideband details don't quite make sense.

Sideband requires one ethernet cable which is plugged into an ethernet port on the motherboard. This network port is then shared between the IPMI system and the operating system. From what I read in this Supermicro manual, "Use the same MAC address you are using for LAN1 for the SIMSO IPMI card". However, the IPMI must have a different IP address then the operating system.

How is it possible to have two devices (the operating system and the IPMI) which can listen and transmit on this same physical network port? When a packet arrives at the interface, how does the system determine if this packet is intended for the Operating System or for the IPMI system?

Are these packets handled by the CPU at all, using CPU interrupts? Can packets to the IPMI interface be viewed by the operating system?

Stefan Lasiewski
  • 22,949
  • 38
  • 129
  • 184

4 Answers4

43

I manage a lot of SuperMicro servers using the onboard IPMI. I have a love/hate relationship with the shared (aka sideband) ethernet. In general, the way these things work is that LAN1 appears to have 2 (different) MAC addresses - one is for the IPMI interface, the other your standard Broadcom NIC. Traffic to the IPMI interface (layer 2, based on the MAC address) is magically intercepted below the operating system level and never seen by whatever OS is running.

You've already hit on the one good point for them: less cabling. Now let me cover some of the downsides:

  • It's particularly difficult to partition the IPMI interface onto a separate subnet in a secure manner. Since the traffic all goes over the same cable, you (almost) always have to have the IPMI interface and the LAN1 interface on the same IP subnet. On the latest motherboards, the IPMI cards now support assigning a VLAN to the IPMI NIC, so you can get some semblance of separation - but the underlying OS could always sniff the traffic for that VLAN. Older BMC controllers don't allow changing the VLAN at all, although tools like ipmitool or ipmicfg will ostensibly let you change it, it just doesn't work.
  • You're centralizing your failure points on the system. Doing configuration on a switch and manage to cut yourself off somehow? Congratulations, you've now cut off the primary network connection to your server AND the backup via IPMI. NIC hardware fail? Congratulations, same problem.
  • Early SuperMicro IPMI BMCs were notorious for doing wonky things with the network interface. Whether to use the onboard vs. dedicated IPMI port was often determined at power-on (not restart), and would not toggle from there. If you had a power outage and your switch didn't provide power quickly enough, you could end up with the IPMI failing to work because it autodetected the wrong setting.
  • I've personally had lots of weird, unexplainable connectivity issues getting the sideband IPMI working reliably. Sometimes I simply couldn't ping the interface IP for a few minutes. Sometimes I'd get a storm of packets on the assigned VLAN, but the traffic all appeared to be dropped.

While this has nothing to do with sideband-vs.-dedicated, I'll also note that the tools for accessing host systems are very poorly written. Older IPMI cards don't support anything other than local authentication, making password rotation a total pain. If you're using the KVM-over-IP functionality, you're stuck using an improperly-signed, expired Java applet or a weird Java desktop application that only works on Windows and requires UAC elevation to run. I've found the keyboard entry to be spotty at best, sometimes getting "stuck keys" such that it's impossible to type a password to login without trying 10 times.

I've eventually managed to get 40+ systems working with this arrangement. I've got mostly newer systems I could VLAN the IPMI interfaces onto a separate subnet, and I mostly use the serial console via ipmitool which works very well. For the next generation of servers, I'm looking at Intel's AMT technology with KVM support; as this makes it into the server space, I can see replacing IPMI with this.

user170914
  • 3
  • 3
natacado
  • 3,317
  • 28
  • 27
  • Thank you for the very detailed explanation. Do you have any more information regarding how the traffic 'is magically intercepted below the operating system level and never seen by whatever OS is running'? We're trying to understand how that works. – Stefan Lasiewski Apr 15 '11 at 16:45
  • A simple hardware bridge will do the trick. – Antoine Benkemoun Apr 19 '11 at 17:38
  • great answer and yes the traffic is bridged – Jim B Apr 20 '11 at 19:55
  • 2
    Stephan - Antoine and Jim explained it in the comments - it's likely a hardware bridge. Think of it like a tiny switch implemented in silicon, connecting the physical NIC interface to 2 virtual NICs - one for IPMI, one for the main computer. – natacado Apr 25 '11 at 17:17
  • Thanks for that explanation. That's exactly how I thought it would work, but when I discuss this with other people (network admins, sysadmins) I get a lot of disagreement. – Stefan Lasiewski Apr 25 '11 at 17:54
  • one thing I just saw -- if there is a large enough disconnect between the IPMI firmware version and the IPMIView software version you'll get generic, cryptic "error connecting" when trying to initiate a KVM session! Updating to latest IPMI firmware fixed it. :p – Jeff Atwood Oct 23 '12 at 05:36
  • 1
    The Java issue is largely taken care of now with any IPMI firmware 2.0 or greater. These addressed issues for Java and moved many iKVM's over to HTML5. IPMI version 2.x also enabled redfish support and credential verification via secure LDAP. The failover works like this: if there is a link up on dedicated port, the system at start will used that. If the link goes down it fails over to the primary onboard nic. If it is configured to "Dedicated", then the failover does not occur. This was addressed in many 1.5x ipmi firmware versions. – Rowan Hawkins Feb 22 '20 at 09:37
5

I haven't used those particular cards before, the ones I have used either have a different MAC for the IPMI traffice or the port is dedicated to IPMI traffic only. It might be possible that IPMI shares the NIC including the MAC though.

IPMI will have a different IP from the OS, so the packets will be directed correctly based on that. IPMI traffic never hits the CPU, it's all handled in the sideband management ICs.

Chris S
  • 77,337
  • 11
  • 120
  • 212
  • I see. Some IPMI cards on some systems will "share the MAC address" (Looks like Supermicro & Dell do this), while others use different MAC addresses (IBM does this). – Stefan Lasiewski Apr 14 '11 at 23:08
  • 3
    On my Dell servers the IPMI interface has a different MAC even though it is the same physical network port. – sciurus Apr 19 '11 at 17:35
3

Wanted to add to the general advice that using sideband means that you can't talk from the server to the BMC. The traffic seems to be filtered out. I've tried this on IBM/Dell/HP kit.

Ed Sykes
  • 248
  • 2
  • 3
  • 1
    to elaborate on this, as it just hit me as well. ESXI and no VMs can access the BMC (but ext hosts can), WHY? Because, traffic outbound on NSCI (shared IPMI) port, would have to hit a switch, and rebound back to same port. AFAIK typical L2 switches don't. – Kevin Aug 17 '17 at 02:58
1

I will backup natacados' final bullet point in his initial response, your IPMI sessions will randomly timeout (i use IPMIView from supermicro to look at the console on my boxes). Things like firmware upgrades and powercycles seem to inexplicably and randomly fail.

Great answer natacado, incredibly thorough.

Ben Lutgens
  • 351
  • 1
  • 4
  • 2
    [**Make sure your IPMI firmware is up to date!**](http://www.supermicro.com/support/bios/firmware0.aspx) If there is a large enough disconnect between the IPMI firmware version and the IPMIView software version you'll get generic, cryptic "error connecting" when trying to initiate a KVM session! Updating to latest IPMI firmware (this can be done from the IPMIView utility main menu under File) fixed it. :p – Jeff Atwood Oct 23 '12 at 05:38