7

When providing dedicated services, what is your best practise to protect IPMI either from public network or through kernel module.

Firstly, I want to be sure that if someone scans my networks and he finds some IPMI cards, he won't be able to get control of it. There were a bug that allows to send emails via supermicro IPMI through Anonymous account. I know it's a good practice to use IPMI only in local networks without public, however clients won't be happy about using VPN to access IPMI much.

Secondly, you can use ipmitool command to manage IPMI configuration without user authentication. I'd like to prevent customers of changing IPMI setting - e.g. IP address, removing my monitoring users, ...

What's your best practise? How would you have solved this issue if you had faced it?

mfinni
  • 35,711
  • 3
  • 50
  • 86
Yarik Dot
  • 1,543
  • 12
  • 26
  • Can you explain me how you can change the IPMI configuration without authentication? I believe this is only possible as root from the host machine? – Jens Timmerman Feb 03 '14 at 13:40
  • Yes, you are right. That's exactly it. ipmitool lan 1 set ... or you can change users, etc. As I am talking about dedicated servers, customers have root access, so they can change whatever they want. However, I don't want to let them, because they can missconfigure it and I'll have to go to DC to fix it. – Yarik Dot Feb 06 '14 at 19:31
  • 2
    Also consider FreeIPMI over IPMItool. FreeIPMI is popular for scripting, and it is very well documented. – Stefan Lasiewski Apr 27 '14 at 17:04

3 Answers3

6

The primary benefit of IPMI is out-of-band access for SHTF occasions, in which the kernel is typically non-functional. So, you should allow access outside the operating system. Set up a VPN or the the very least, a way for your clients to access IPMI via an ssh tunnel or something.

You are right in being wary of exposing IPMI to the public internet. If your clients complain about the additional security, then they're not the kind of customers you're going to want to deal with.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • 1
    I had to google SHTF - I'm going to start using that acronym now! – Mark Henderson Jan 26 '14 at 21:32
  • 2
    A vpn or ssh tunnel will probably be the best way to secure it. You would probably want to make sure customers can only connect to their own servers IPMI interfaces, not those of other customers (either accidentally or maliciously) – Grant Jan 26 '14 at 21:51
  • Of course! I want to give them access. I just want to improve security. I am actually more interested about the second issue. To be honest, it's my proactivity. In one hand, I want to offer them secure service, in the other hand the service must be easy to use. – Yarik Dot Jan 26 '14 at 22:46
  • 1
    Using a VPN or otherwise being on the local management LAN is the *only* way that OOB management tools should be accessed. And if it's multitenant, each customer *should* only be able to hit their own servers. If I were your customer, and found I could access other customers' IPMI interfaces, I'd be leaving you. – mfinni Jan 26 '14 at 23:18
  • @Grant - 100% agreed. Great point. – EEAA Jan 27 '14 at 00:29
3

Firstly, I want to be sure that if someone scans my networks and he finds some IPMI cards, he won't be able to get control of it.

The BMCs on Supermicro systems have had some particularly nasty bugs within the last year. Make sure your systems are running the latest firmware, which addresses most of the bugs like the 'anonymous user' bug and the Infamous cipher zero bug. Note that these firmwares are only available for recent motherboards (The X8, X9 and X10 generations; usually not for the X7 generation which was being shipped 3-4 years ago), so your hardware will need to be modern hardware.

Even with the updates, Supermicro's IPMI is still subject to bugs. Passwords are transmitted over the network in the clear, etc. In my world, IPMI must absolutely be on a private network and available only from a few special management nodes.

I know it's a good practice to use IPMI only in local networks without public, however clients won't be happy about using VPN to access IPMI much.

IPMI is a great management tool for system administrators. By it's nature, this means that its also a nice backdoor for hackers. If I gain access to the IPMI network, I can do interesting things like power off all 200 machines within a few minutes, or tell every single node to PXEboot upon next boot (and potentially overwrite whatever is on the disks at the time). If you can explain this nicely to your customers, they might see the wisdom of a VPN.

Secondly, you can use ipmitool command to manage IPMI configuration without user authentication. I'd like to prevent customers of changing IPMI setting - e.g. IP address, removing my monitoring users, ...

Make sure that ipmitool, FreeIPMI, etc. are not installed by default on the system. In your customer documentation, add a warning that installing these tools on the OS is a potential security problem and if they install these tools they assume some of the risk.

Stefan Lasiewski
  • 22,949
  • 38
  • 129
  • 184
  • 2
    It's a shame that an answer like this didn't received any votes. This has to change! Good post Stefan, thanks for that, +1. – gxx Jul 15 '16 at 11:40
  • 1
    Thanks for that. IPMI is unfortunately arcane and poorly understood, and is now partially obsoleted by virtualization, containers and the cloud (It's also an important tool if you manage your own virtualization and cloud infrastrcuture). Recently I've been reading up on a replacement for IPMI (And PXEboot, and more) called RedFish: https://www.dmtf.org/standards/redfish – Stefan Lasiewski Jul 15 '16 at 17:17
  • I'm operating some private clusters with [ganeti](http://www.ganeti.org/), which I totally recommend, and even in 2016 I'm still using IPMI for out-of-band access and emergencies etc. So, at least in my environments, it's of use and not obsolete. Maybe this is because I don't think that "the cloud" makes that much sense regardless of the needs and expectations. (And I'm a big fan of [this](https://chrome.google.com/webstore/detail/cloud-to-butt-plus/apmlngnhgbnjpajelfkmabhkfapgnoai) :].) Thanks for the link about RedFish, never heard of it before. – gxx Jul 15 '16 at 17:23
  • I think not installing ipmitool by default does not add any security when a use is able to install software. Also most management interfaces like IPMI were intended to be used OOB, i.e.: not available in the normal user's network, specifically not in Internet. A VPN access may help here. – U. Windl Apr 19 '21 at 09:34
1

Ok, my probably final solution is very easy. As I have ADMIN user to the IPMI, I can write a simple interface that will run ipmitool commands to start/stop/restart servers.

For webconsole I did some research. Supermicro generates jviewer.jnlp file that contains only port numbers and some credentials. Firstly, I thought there is some signature and I can't change the content. Nevertheless, there is no signature and I can put whatever I want there.

So, as a part of my interface, I am going to create console button, that will configure a firewall to temporarily allow ports needed by console and also will download the jviewer file, update it and pass it to the client.

It'll cost me some development, but it will allow me to have management cards in local network and also have them accessible from public network when needed.

Yarik Dot
  • 1,543
  • 12
  • 26
  • For simple operations like power-cycling you generally only need to open a single port-- port 623/UDP for IPMI. Instead of Jviewer/KVM-over-lan, consider the serial-over-lan functionality. The KVM-over-LAN looks friendlier, but it can be very buggy and slow. It's behavior is inconsistent between different Supermicro servers. Jviewer triggers security warnings in Firefox, isn't possible in Chrome, etc. Supermicro needs to get with the times and replace Jviewer with an HTML5-based viewer. – Stefan Lasiewski Apr 27 '14 at 17:02