2

We all know about Spectre and Meltdown, at this point. The take away is the while Meltdown can be solved/worked around with a (complex and invasive) kernel patch (namely KAISER/PTI), Spectre requires an updated microcode with advanced branch control.

Until some days ago, Red Hat shipped an updated microcode_ctl package which, in some (but not all) cases, had the appropriate microcode to patch/update (early in the boot process) the base processor microcode.

However, it seems the updated microcode causes system instability, unexpected reboot and even unbootable systems. So Red Hat reverted the microcode_ctl package to not load the microcode update needed to fix Spectre. Now their official suggestion is "to contact their silicon vendor to get the latest microcode for their particular processor".

While understandable, this stance only move the "instability provider" down from the OS to the BIOS/firmware itself.

So, my question is: how to you feel about the microcode update? Have you applied the new BIOS/firmware to production systems? Any instability to report/comment? Finally, should I wait for a new "patch round" or you advise to immediately apply the BIOS/firmware fix?

shodanshok
  • 44,038
  • 6
  • 98
  • 162

2 Answers2

1

I don't think that's what they are actually saying. There's no mention of UEFI/BIOS updates or system vendors/motherboard vendors (although that is certainly a good option when available, and if the new microcode is working reliably).

At least to me, "Customers are advised to contact their silicon vendor to get the latest microcode for their particular processor" reads: "download and use the current microcode at your own risk, or bug Intel for a fixed version".

I also imagine that Redhat's decision is for this version with known stability issues specifically, once there is an update I would imagine that they will reevaluate (probably giving it a little more time before rolling it out to everyone).

There are other OS vendors that similarly did roll out the microcode update that have now rolled back their updates (see eg VMware's announcement).

All in all, my impression is that with the current microcode version (packaged by Intel as 20180108), it appears that there's a trade-off of "stability issues with precious little information on what triggers them" vs "possibility of spectre mitigation", and that major OS vendors seem to be taking the "stability" side while the issues are being addressed.

Håkan Lindqvist
  • 33,741
  • 5
  • 65
  • 90
  • The latest microcode can be loaded by a) the system BIOS/firmware or b) by the microcode_ctl package. If the second option is not working anymore, we are left with the first one only - a system BIOS update. Anyway, regardless the method used to load the microcode, the key question is: if it has a (real) change to bring system instability, would you risk updating your bare-metal server? What concern me is a "delayed" instability, where problems happen only days/weeks after system BIOS upgrade. – shodanshok Jan 23 '18 at 08:10
  • @shodanshok First off, I would argue that "b)" is not as narrowly defined as you make it out to be. Just because Redhat does not distribute the new microcode version (because of the cited stability issues) does not mean that the tooling to apply Intel microcode updates disappears. That said, it appears that the current microcode version has the tradeoff "potential stability issues" vs "spectre mitigation". Redhat (and other vendors) have opted for stability, while these issues are being addressed. – Håkan Lindqvist Jan 23 '18 at 08:51
  • Ok, and I wonder *what* to do in this case: should I taking stability (as done by Red Hat and others) vs security (as done by major cloud providers)? What I find frustrating is the *very* low details/info on the matter... – shodanshok Jan 23 '18 at 11:13
  • @shodanshok I think that comes down to what extent Spectre is applicable in the first place, ie what are the chances of untrusted code executing on these machines in the first place? "It's all we do" is essentially the answer for multi-tenant virtualization environments ("cloud compute" if you will), while the answer may very well be "only if things have already gone horribly wrong" for in-house server environments. – Håkan Lindqvist Jan 23 '18 at 11:41
0

Ok, it seems that multiple vendors have retired their BIOS update, so the firmware update option is almost non-existent at the moment. For example, from DELL site:

Patch Guidance (update 2018-01-22): Intel has communicated new guidance regarding "reboot issues and unpredictable system behavior" with the microcode included in the BIOS updates released to address Spectre (Variant 2), CVE-2017-5715. Dell is advising that all customers should not deploy the BIOS update for the Spectre (Variant 2) vulnerability at this time. We have removed the impacted BIOS updates from our support pages and are working with Intel on a new BIOS update that will include new microcode from Intel.

If you have already deployed the BIOS update, in order to avoid unpredictable system behavior, you can revert back to a previous BIOS version. See the tables below.

As a reminder, the Operating System patches are not impacted and still provide mitigation to Spectre (Variant 1) and Meltdown (Variant 3). The microcode update is only required for Spectre (Variant 2), CVE-2017-5715.

This is confirmed by Intel own documentation

Basically, the only method to obtain the required ucode is to manually download it from Intel site

TL;DR: I'll wait and see for the fallout of the failed microcode update to settle. Meltdown and Spectre variant n.1 can be patched by simply updating the kernel, fortunately.

shodanshok
  • 44,038
  • 6
  • 98
  • 162