14

I just RDP'd into one of my companies servers, was alerted to windows updates, so I click. Then I see 62 high priority updates, with the last update (according to update history) was installed on Thursday, January 16, 2014, more than one year ago.

What actions need to be taken here?

OpenCoderX
  • 237
  • 2
  • 7
  • 21
    Consider yourself fortunate that mfinni and others are actually answering this. It is akin to one of us coming to SO and asking "when I write code should I debug it?" – TheCleaner Feb 09 '15 at 15:21
  • 7
    @TheCleaner The answer to that question is "after you upsell the customer on your code-debugging services." – HopelessN00b Feb 09 '15 at 15:28
  • 1
    Before I read any of the answers I just want to say that you should contact the server's admin. If this person is you then brush up on the potential pitfalls of performing updates and proceed with caution. Upper-management will blame you as soon as they put their pen in the wrong drawer and wonder where it went. If it ain't broke... – MonkeyZeus Feb 09 '15 at 16:43
  • 8
    @MonkeyZeus "if it ain't broke..." in this case, you mean "if it ain't secure, don't secure it" ? –  Feb 09 '15 at 17:15
  • @AndréDaniel "If it ain't broke, don't rush to break it". Hence the entire first three sentences of my comment :) – MonkeyZeus Feb 09 '15 at 17:18
  • 5
    "If it ain't broke, don't fix it" and "If it ain't secure, don't secure it" express essentially opposite ideas. – user2338816 Feb 10 '15 at 01:06
  • 1
    @TheCleaner Beginner and non-expert questions are perfectly acceptable on the network. While the OP could have done some research first this question is useful for many other developers working in an environment that's badly managed and configured. – Lilienthal Feb 10 '15 at 09:43
  • Your best bet as a small business < 50 seats is to use WSUS to auto-approve security updates to all desktops/laptops, and simply fix/work around any resulting issues. This will be less work/cost than the testing required. Also: Antivirus is a MUST! – Ben Feb 10 '15 at 14:09
  • 7
    @Lilienthal - `"useful for many other developers"` has no bearing on this site. This site isn't designed as a helpdesk for SO users. Call it cruel if you want, I didn't make the site's scope. – TheCleaner Feb 10 '15 at 14:29
  • 2
    @Lilienthal `Beginner and non-expert questions are perfectly acceptable on the network.` Each site has it's own rules, and this is below the threshold for participation on ServerFault. ServerFault is not SO and does not have the same rules as SO. – Wesley Feb 10 '15 at 14:55
  • 3
    This question is being voted for closure because the author does not show a level of technical understanding or appropriate due diligence in researching the topic that the community judges as being a minimum barrier to participate. – Wesley Feb 10 '15 at 14:55
  • 2
    @TheCleaner Make that short for "developers in charge of maintaining their own servers". For better or worse (likely the latter) the OP apparently has the rights to *manage the business systems he's using* and as such his question is perfectly [on topic](http://serverfault.com/help/on-topic). Whether he put in enough effort before turning to the site is an entirely different but valid question. Remember that `Topicality ≠ Quality`. – Lilienthal Feb 10 '15 at 15:47

4 Answers4

31

Short answer - yes. Most of the Windows Updates are security related. Not having the patches means you're vulnerable.

Longer answer - you need a procedure that covers this sort of thing. It's more rare these days, but sometimes a patch can break things, or change behavior in such a way that it's broken as far as your company is concerned. You should be evaluating each patch when it's released (there's a monthly schedule plus some urgent ones), determine if you need the patch (probably yes), do some testing on test/staging servers to do some diligence about potential breakage, and then do the installs.

You should also exercise some care about the deployments, because OS patching often means rebooting, which often means there's service downtime, unless you've got some good HA for all your services. If you think you'll be clever and patch during the day and then postpone the reboot, that's not a great idea - some files will be updated but others won't.

Microsoft offers a free product called WSUS that can make patch management a little easier than doing approvals and deployment all one-by-one.

FYI, you should be doing this sort of thing for all classes of device you have. Network device firmware, server hardware firmware, VMware ESXi, etc. Those patches don't come out for the fun of it, almost all of them address bugs, and many of them can be security related.

Further - you should be asking someone who's more senior than you on your technical team. If you're the only admin there, you and your organization are not doing too well. Don't take that personally, we all need to start without knowing everything we should - but if this is your question, you shouldn't be the only person managing these servers.

mfinni
  • 35,711
  • 3
  • 50
  • 86
  • 14
    Fast-typing bastard. >:/ – HopelessN00b Feb 09 '15 at 15:00
  • 1
    Snow day baby. Trying to get VPN access into the office. – mfinni Feb 09 '15 at 15:03
  • I'm not managing them, I'm an app developer who happened to need to view some event viewer logs on the host, I've actually noticed update alerts before but this time I missed the little 'x' and clicked the bubble, leading me to the summary page. My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. We actually have WSUS. Up until today I just assumed that any update notice I saw would be taken care of that weekend. – OpenCoderX Feb 09 '15 at 15:41
  • Talk to management immediately. Do you have systems administrators? If you do, then they might not be doing their job, unless your company policy is "don't install updates." If you don't have systems administrators, get management to hire some or contract it out. As you can probably guess, devs don't have the same goals or skillsets as sysadmins and most can't/shouldn't play both roles. – mfinni Feb 09 '15 at 15:44
  • 10
    `"My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. "` - no dilemma, you tell your boss via email what you noticed and are concerned. There may be a legit reason, or it may simply be laziness. Either way, it isn't your fault it's not been done, but you should at least voice concern. – TheCleaner Feb 09 '15 at 15:44
  • I agree, however *rare* is a relative term, relative to $environment. I say this b/c if a shop is running non-MS software (almost *every* environment), it's more likely a patch could break something than if that environment was pure MS. – MDMoore313 Feb 10 '15 at 14:50
  • To add to your short answer, the most readily available stream of attack vectors are the monthly updates themselves. Hackers simply reverse engineer the fixes and produce a new set of attack vectors for unpatched machines every month. Think of the implications for outward facing servers. – Shiv Feb 11 '15 at 04:39
18

The generic answer is it is a good practice to keep your servers updated.

But pay attention to a few things:

  1. Updates may cause the server to be sluggish during installation, or even cause some downtime if they require reboot(s). You should plan to do them out of office work hours.

  2. Updates have some risk associated. They might break your server, or cause some incompatibility. They are usually fully uninstallable, but with 62 of them you should also consider if you have a trustworthy backup (you should, anyway).

  3. Is there a reason why you are one year late on upgrades? Is this your first log in to that server in a year, or is something else broken?

  4. Pay special attention to the infamous Excel bug that comes with some December updates of Office, if your company uses Excel macros, but this probably doesn't apply to a server that shouldn't be running Office.

  5. Many sysadmins wait a few days or weeks before installing updates, just to see if anything bad comes up on the Internet regarding those updates. When deciding if you need to wait, consider security risks of leaving the server unpatched for more time.

pgr
  • 459
  • 5
  • 14
  • What "infamous Excel bug that comes with some December updates of Office" are you talking about? – nobody Feb 11 '15 at 00:44
  • "For some users, Form Controls (FM20.dll) are no longer working as expected after installing MS14-082 Microsoft Office Security Updates for December 2014." as per Technet blog post http://blogs.technet.com/b/the_microsoft_excel_support_team_blog/archive/2014/12/11/forms-controls-stop-working-after-december-2014-updates-.aspx – Shiv Feb 11 '15 at 04:36
  • @Shiv: thanks, I edited the answer to include your link. – pgr Feb 11 '15 at 10:12
  • @pgr, Aren't there like *tons* of these infamous bugs? – Pacerier Feb 12 '15 at 10:04
  • @Pacerier: eheh, sure. Usually all you have to do is rollback the update. Not this one. Files can get "infected" with the bug, i.e., somebody opens them after the bad update, and suddenly the file stops working on a different computer. It's been a real PITA dealing with this one, and it's not over yet. Notice that the issue has got so complex (for the worst cases, when the problem travels with the file) that Microsoft is STILL working on it, and a definitive solution is still to be achieved... but, of course, each sysadmin will have his own nightmare story, this is mine... :-) – pgr Feb 12 '15 at 12:09
  • @pgr, So how did you manage to solve the nightmare eventually? – Pacerier Feb 12 '15 at 21:57
  • The Microsoft links teach the solution (actually install the update everywhere, but clean up the msforms.exd files, recompile VBA where files are broken), it's just a lot of work, and I was able to do it because I had only a handful of computers and files gone crazy. It would be really ugly if I had hundreds, or thousands. And I'm not sure everything is 100% solved, I might get complains about other files later... – pgr Feb 12 '15 at 22:39
8

I know mfinni beat me to the punch, but I'm just going to +1 for WSUS. Specifically:

Let's assume that you have multiple servers, including test and production. Let's also assume that test has similar hardware to production (which isn't a safe assumption, I know, but let's go with it--it's nice but not necessary). You could set up the following scenario in WSUS:

  1. Test servers in their own OU. Group policy says to install updates and reboot at some non-inconvenient time, like Sunday at 3am.
  2. Prod servers in a different OU or OUs. Group policy says to download and notify.
  3. Patches approved, and deadlined to install and reboot the servers during your scheduled maintenance window, several days or a week after the test/dev servers apply patches.

What this does, if it's not obvious, is it approves all critical/security patches for your servers, applies them to test first, and then applies them later to production. I've only seen an update critically break something once, but this would give you a chance to roll back the patch if it fails in test before it applies to prod.

As for the big pile of updates on the server in question, patching is a lower risk than not patching, but I would verify my backups before applying them all just in case because there are so many. If it's a VM, you might want to take a snapshot first.

Katherine Villyard
  • 18,510
  • 4
  • 36
  • 59
1

This is entirely up to your business and the policy that you have set out for updating your servers.

At the very least you should install security updates and perform any other patches like .NET framework updates in a testing environment first before updating production servers.

voretaq7
  • 79,345
  • 17
  • 128
  • 213
Vasili Syrakis
  • 4,435
  • 3
  • 21
  • 29
  • But the rest of your answer is correct - there should be policy and process for this, that weighs security against service interruption/business need/etc. – mfinni Feb 09 '15 at 15:04
  • 2
    `1.` Too slow. You got beat to the punch by two other, better answers. `2.` There's nothing opinion-based about whether to install patches/security updates or not. The only scenario I can envision where you wouldn't want to install patches would be one where you're stealing from your employer. `3.` "Patch management" is most definitely a Server Fault topic, though it might also be topical at super User. – HopelessN00b Feb 09 '15 at 15:05
  • Lots of things are opinion-based, but that doesn't mean that asking the questions about forming those opinions aren't in the domain of professional sys-adminery. Almost everything is a trade-off, and one's opinion is what matters. If this stuff was cut-and-dry, we'd just need to write one book and we wouldn't need professionals. – mfinni Feb 09 '15 at 15:07
  • 1
    If I knew my server admins were asking this on SF I'd be terrified for my infrastructure. The core of the question is "What should I do?" not something akin to "How do I manage/automate/improve?" which would fall under the category of patch management and so on. I thought this place was for professionals, maybe I am wrong about that. Just seems like it belongs on SU to me! – Vasili Syrakis Feb 09 '15 at 15:13
  • 1
    The asker is clearly fairly junior, because he/she is asking this question. They need help; that's why this site exists. Both other answers are "Yes, here's more details and nuance." – mfinni Feb 09 '15 at 15:18
  • 5
    I would be more worried about the server admins who _did not ask_ and _did not update for a year_. – Michael Hampton Feb 09 '15 at 15:41
  • I'm not managing them, I'm an app developer who happened to need to view some event viewer logs on the host, I've actually noticed update alerts before but this time I missed the little 'x' and clicked the bubble, leading me to the summary page. My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. We actually have WSUS. Up until today I just assumed that any update notice I saw would be taken care of that weekend. – OpenCoderX Feb 09 '15 at 15:43
  • I am basically auditing the existing IT manager. This is what I found and before I walk in and say hey, you no do your job, I want to know if there are legit reasons to not have this many updates installed, BTW, since I posted I've hit many more servers, and I find the same situation. – OpenCoderX Feb 09 '15 at 15:44
  • 1
    That's a different scenario altogether. As you said, the work might not be getting done, or there could be something preventing the updates from being pushed out from WSUS. Although, whoever is managing WSUS should notice if things are not getting updated. – Vasili Syrakis Feb 09 '15 at 15:46
  • 3
    It's definitely something that should be raised; there have been some fairly critical security updates in the last 12 months. – Vasili Syrakis Feb 09 '15 at 15:48
  • @shadowadmin I would avoid accusing language and say, "Hey, did you know that there are 62 unapplied updates on [Servername]? I thought WSUS took care of that." It might be some kind of accidental oversight rather than, you know, apathy (although it suggests that whomever runs WSUS doesn't have the cute little reports going to their email). – Katherine Villyard Feb 10 '15 at 15:23
  • Also, +1 for thinking patch management is on-topic. – Katherine Villyard Feb 10 '15 at 15:24