12

Firmware receives very little attention when it comes to updates. Out of sight, out of mind.

Many devices: RAID controllers, NICs, chipsets, and even hard drives, get some benefit from being updated. Better features, security/bug fixes, etc.

Most SA say, "Whenever it breaks, update the firmware." But this can lead to difficulties down the road; Several times, when contacting Dell about a failed hard drive, I've been asked if my hard drive firmware is up to date. All of my servers use some type of RAID configuration. If I already have a single drive failure, should I even consider trying to upgrading the firmware on the rest of the drives or the RAID controller? I would say no. But Dell seems to have a different view.

  • What's a realistic update schedule for system firmware?
  • Do you have any best practices to share?

(I am aware that Dell has a nice utility called Server Update Utility, which checks for all new firmware on any Dell server.)

Joseph Kern
  • 9,809
  • 3
  • 31
  • 55

10 Answers10

10

I update firmware in two key instances.

  1. When staging up a server.
    • When I just get the server, I'll check the the HP web-site for the date of their latest "Firmware Update CD". If it's new enough, I'll run it against the server before bringing it up to production.
    • When I repurpose a server. Typically, this server is 2-5 years old and probably hasn't had a firmware update in that entire time. Since I'm reformatting it anyway, I'll update all the various firmware on the server.
  2. When there is a vendor identified need to do so.
    • Sometimes there are major stability problems identified, like an inability to rebuild a RAID5 array after the wrong kind of failure, or a major performance bug in the TCP-offload engine on the NIC.
    • Sometimes when calling in for support, the support tech will request I update the firmware. I will do so then.

There is a third instance that I didn't list above, because it hasn't happened yet:

  • When putting a much newer component into an older server. Sometimes the system BIOS will need updating to handle it.
sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
5

We use HP SIM (System's Insight Manager) to roll out firmware, we do it by platform - test first, then development, then integration, then reference then finally production - usually about a week or so per platform so we have a 5/6 week release to production window. Seems to work but one thing we NEVER do is roll out firmware at the same time as other updates like drivers/code etc. - saves a lot of finger-pointing.

Brett Larson
  • 854
  • 1
  • 12
  • 20
Chopper3
  • 100,240
  • 9
  • 106
  • 238
4

I might be going against the grain here, but if it's not broken don't fix it. If it's not a security issue then I leave it alone.

I've never had Dell deny service for that reason.

Perhaps you can schedule the utility once a month, then update them with other reboots?

MathewC
  • 6,877
  • 9
  • 38
  • 53
  • +1 for speaking the truth. IF there's no good reason to upgrade (like the vendor no longer supporting your kit unless you update the BIOS/firmware), then why are you doing it? – RainyRat Jun 18 '09 at 14:39
  • I don't work with Dell Support but I am still stuck on a problem that Intel won't support until I update my BIOS -- only I can't get the dang thing to update and I sometimes wonder if it's because the only available update is too many versions ahead of what my server is running. – Ben Dunlap Jun 18 '09 at 15:25
  • 1
    In principal I agree with you but we have some servers that are significantly faster and more stable now than when bought, purely due to firmware improvements. – Chopper3 Jun 18 '09 at 15:39
  • Agreed. I guess performance issues could be considered "broken" in this case. – MathewC Jun 18 '09 at 15:56
  • For the record, I recently actually had Dell deny a hard drive replacement on a PowerEdge server until I updated the raid controller firmware to the latest version. – Ryan Bolger Jun 18 '09 at 17:57
  • Interesting. Did the upgrade fix the issue? – MathewC Jun 18 '09 at 18:02
4

You should treat firmware updates as you treat patches, although these updates are typically harder to deploy so you may want to check the release notes to help deciding if it is worth the effort (but then again, you may be doing this for patches as well). Firmware updates are just as likely to contain a new nasty bug as they are to fix one.

Updating firmware should be done when commissioning (or re-commissioning the hardware) as it's an easy time to do so.

The deployment should roughly follow something along these lines:

  1. Test on lab machines
  2. Deploy to unimportant systems
  3. Wait
  4. Deploy to important/production systems

Then again, some things cannot follow this. In particular, it's pretty crucial to very quickly push Microsoft patches to Windows desktops, and testing is difficult to do quickly without significant resources.

Toto
  • 738
  • 2
  • 5
  • 11
2

I will usually update the firmware when comissioning any new infrastructure equipment. Workstations and Printers only get updated if there is a specific problem such as a required bug fix or security issue. It is less important with servers, as for the most part the firmware code either dosn't execute beyond the boot process or is so simple there little to go wrong.

I have had bad experiences with upgrading RAID firmware on servers, as many manufacturers don't guarentee the current RAID configuration will be viable if the firmware vesion changes, the same with individual disks in the array.

My tendency would be to apply a "if it ain't broke, don't fix it" policy for the most part. It is worth checking if it is possible to recover from a bad firemware upgrade (i.e. redundant ROM in HP Servers, or if you have an EEPROM flasher and are willing to use it a removable chip).

Richard Slater
  • 3,228
  • 2
  • 28
  • 42
  • How often do you check for "security issues" on all your different firmware? – Joseph Kern Jun 18 '09 at 14:12
  • Virtually all of our kit is HP, by registering the product I sign up a generic account to "notifications about firmware, drivers and security" issues, if I leave I will re-direct that account to my sucessor. – Richard Slater Jun 18 '09 at 14:52
2

I can't speak on Dell, but I was told by folks inside the IBM storage group many years ago that the first levels of their RAID firmware is NEVER the best performer. They shoot for stability first, and then crank up the performance with future rev levels. Unfortunately, few of us would think (or perhaps have the moxie) to go back later and mess with the RAID firmware of healthy system unless there were problems. So, our strategy has been to upgrade the RAID firmware levels every time we set up a new server, assuming the RAID adapter isn't a brand spankin' new model. That way we at least get the best performance levels available at the time. If it is a new mode adapter, we try to make a mental note to go back in a few months and check for updates, but we're not religious about it.

On motherboards, we DO NOT mess with them unless vendor tech support tells us. Our experience over the years has been that unless there is specific problem that needs fixing, the risks vastly outweigh the hard-to-measure benefits.

//spk

Scott Kantner
  • 161
  • 3
1

I've used the SUU and the similar products from HP and IBM with success. The only issues I have ever encountered were when a vendor driver was used to replace a Dell driver, for no technical reason, which caused the SUU upgrade to fail the OM upgrade on a Dell server. We ran updates for firmware and drivers on several hundred servers at least twice a year to keep them all current. We never ran into issues with RAID firmware.

Mitch
  • 1,127
  • 11
  • 19
  • Tell me more ... – Joseph Kern Jun 18 '09 at 14:11
  • We followed the Dell releases, quarterly I think, to schedule the changes to apply the updates. We would apply them to all Dell machines across all of the clients we supported. Perhaps we were lucky but having had very few issues I think it also came down to having a plan and keeping things up-to-date. When we took on new servers one of the first things we did was bring all the goods up to the current level. – Mitch Jun 18 '09 at 14:41
1

We update firmware every chance we get, which regrettably is probably 3-4 times per year (Major maintenance windows) for some systems.

Jason Tan
  • 2,742
  • 2
  • 17
  • 24
0

If you read the blurb that accompanies a Dell firmware upgrade you'll find it classifies the update as critical, important or whatever, and describes what the upgrade fixes (not always as clearly as one would wish!).

My view is that I will not apply a firmware upgrade without a very good reason. If it's a critical update and fixes a problem I have a reasonable chance of encountering then yes, I'll apply it even though it's a stressful business.

As you say, Dell support normally insist on applying any relevant upgrades before they will support you, and obviously in that case you don't have a lot of choice. You can understand Dell not wishing to waste engineer time, even though I have never known firmware to be the cause of a support call.

JR

John Rennie
  • 7,756
  • 1
  • 22
  • 34
0

I think it depends on type of environment you have to deal with. If you work in environment that requires high availability and you can’t reboot servers when you want. The best practice in my opinion is to implement a maintenance schedule. Select a time frame for each system, notify users and perform all necessary work including firmware updates during this allotted time frame. In this way you can be certain that all your systems are up to date and minimize impact on production at the same time. Also, as was noted in comments earlier, it is necessary to check for firmware updates when you building a new server.

user176320
  • 139
  • 2
  • 4
  • 11