45

My company distributes a Windows Installer for a Server based product. As per best practices it is signed using a certificate. In line with Microsoft's advice we use a GlobalSign code signing certificate, which Microsoft claims is recognised by default by all Windows Server versions.

Now, this all works well unless a server has been configured with Group Policy: Computer Configuration / Administrative Templates / System / Internet Communication Management / Internet Communication settings / Turn off Automatic Root Certificate Update as Enabled.

We found that one of our early beta testers was running with this configuration resulting in the following error during installation

A file that is required cannot be installed because the cabinet file [long path to cab file] has an invalid digital signature. This may indicate that the cabinet file is corrupt.

We wrote this off as an oddity, after all no-one was able to explain why the system was configured like this. However, now that the software is available for general use, it appears that a double digit (percentage) of our customers are configured with this setting and no-one knows why. Many are reluctant to change the setting.

We have written a KB article for our customers, but we really don't want the problem to happen at all as we actually care about the customer experience.

Some things we have noticed while investigating this:

  1. A fresh Windows Server installation does not show the Globalsign cert in the list of trusted root authorities.
  2. With Windows Server not connected to the internet, installing our software works fine. At the end of the installation the Globalsign cert is present (not imported by us). In the background Windows appears to install it transparently on first use.

So, here is my question again. Why is it so common to disable updating of root certificates? What are the potential side effects of enabling updates again? I want to make sure we can provide our customers with the appropriate guidance.

Jeroen Ritmeijer
  • 717
  • 1
  • 6
  • 14
  • 14
    New root certificates appearing on all systems without warning or documentation are a concern for some security people. They simply don't trust Microsoft to fully vet new root certificates without at least doing some vetting themselves. Doesn't help matters when Microsoft does things like pushing 18 new root certificates wihout any notice. – Brian Jan 27 '16 at 16:27
  • Can't you check if the certificate is available in the system and offer to download your certificate by hand from you website if the updating is disabled ? – Falco Jan 28 '16 at 15:50
  • @falco Nop, the cert must be in place before we can run custom logic to detect things like this. That is the whole point of digitally signing installers. Also, if admins have disabled updating of root certs then they are not going to be happy to let some third party vendor do this. – Jeroen Ritmeijer Jan 28 '16 at 15:54
  • Then you could provide a website which checks for the certificate, which the users can visit prior to installing your product? Like "visit .... to check if your system is compatible" and on the website display steps to install the certificate if you detect it is not present ? – Falco Jan 28 '16 at 15:57
  • 1
    @falco Which we have (to some degree), see the link to the KB Article in my question. Also... people don't read instructions. – Jeroen Ritmeijer Jan 28 '16 at 15:59

5 Answers5

37

In late 2012 / early 2013 there was an issue with automatic root certificate updates. The interim fix was to disable the automatic updates, so partly this issue is historical.

The other cause is the Trusted Root Certificate program and Root Certificate Distribution, which (to paraphrase Microsoft)...

Root certificates are updated on Windows automatically. When a [system] encounters a new root certificate, the Windows certificate chain verification software checks the appropriate Microsoft Update location for the root certificate.

So far, so good but then...

If it finds it, it downloads it to the system. To the user, the experience is seamless. The user does not see any security dialog boxes or warnings. The download happens automatically, behind the scenes.

When this happens it can appear that certs are being automagically added to the Root store. All this makes some sysadmins nervous as you can't remove a 'bad' CA from the certificate management tools because they're not there to remove...

Actually there are ways to make windows download the full list so they can edit it as they wish but it's common to just block the updates. A great number of sysadmins don't understand encryption or security (generally) so they follow received wisdom (correct or otherwise) without question and they don't like making changes to things involving security that they don't fully understand believing it to be some black art.

James Snell
  • 761
  • 6
  • 8
  • 13
    `A great number of sysadmins [...] don't like making changes to things involving security that they don't fully understand believing it to be some black art.` Yeah. Sad, but true. – HopelessN00b Jan 27 '16 at 16:40
  • 8
    @HopelessN00b So you'd rather they *freely* make configuration changes involving security that they don't fully understand? That seems a far scarier proposition to me. – Joshua Shearer Jan 27 '16 at 23:18
  • 13
    @JoshuaShearer I'd rather they understand or stop calling themselves sysadmins. – Kevin Krumwiede Jan 28 '16 at 05:17
  • "Actually there are ways to make windows download the full list so they can edit it as they wish" Do you have any details about this? – Jeroen Ritmeijer Jan 28 '16 at 15:02
  • @Muhimbi - I don't have anything relevant to the question, but "How to obtain/install the windows trusted root certificates" sounds like an excellent question in it's own right. IMHO the solution you offer in your KB article (to import the GlobalSign certificate) is the best solution since it has the least impact. – James Snell Jan 28 '16 at 19:01
  • 3
    @JoshuaShearer Like Kevin said, if they don't understand security, they shouldn't be sysadmins, and I find it to be a scary proposition to have an administrator of anything who thinks security is some black magic or voodo. – HopelessN00b Jan 28 '16 at 19:25
  • 2
    @JoshuaShearer - since they don't understand, it's arguably moot as they won't know if what they already have is correct or not... In many small-medium sized businesses the 'admin' is `"good with computers"` because they have the latest shiny iThings rather than a genuine professional. – James Snell Jan 28 '16 at 19:34
  • @HopelessN00b I'd say the ability and willingness to admit that one doesn't understand something, then turn around and *research* it to then *gain* understanding, and then acting on that new understanding is what qualifies someone to be a sysadmin in the first place. I'm not seeing raw lack of understanding as being a big problem, it's how sysadmins react to their not understanding. The good ones gain the understanding once they see they don't yet have it. The bad ones act either recklessly or overly cautiously because they don't try to gain the understanding they lack. – Todd Wilcox Nov 28 '17 at 16:43
11

The Automatic Root Certificates Update component is designed to automatically check the list of trusted authorities on the Microsoft Windows Update Web site. Specifically, there is a list of trusted root certification authorities (CAs) stored on the local computer. When an application is presented with a certificate issued by a CA, it will check the local copy of the trusted root CA list. If the certificate is not in the list, the Automatic Root Certificates Update component will contact the Microsoft Windows Update Web site to see if an update is available. If the CA has been added to the Microsoft list of trusted CAs, its certificate will automatically be added to the trusted certificate store on the computer.

Why is it so common to disable updating of root certificates?

The short answer is probably that it's about control. If you want to control what root CAs are trusted (rather than using this feature and letting Microsoft do it for you), it's easiest and most secure to come up with a list of root CAs you want to trust, distribute them to your domain computers, and then lock that list. Since changes to the list of root CAs an organization wants to trust would be relatively rare, it makes a certain amount of sense that an administrator would want to review and approve any changes rather than allowing an automatic update.

To be completely frank, if no one knows why this setting is enabled in a given environment, that means that it shouldn't be set.

What are the potential side effects of enabling updates again?

Domain computers would be allowed to check against the list of trusted CAs on the Microsoft Windows Update Site, and potentially add new certificates into their trusted certificate store.

If this is unacceptable to your clients/customers, certificates can be distributed by GPO, and they would need to include your certificate in whatever distribution method they currently use for trusted certificates.

Or you could always suggest temporarily disabling this particular policy, to allow installation of your product.

Dessa Simpson
  • 491
  • 7
  • 25
HopelessN00b
  • 53,385
  • 32
  • 133
  • 208
4

My reason for disabling the certif.service is as follows:

I have many systems without internet connection. Also in most cases they lack display/kb/mouse because of the fact they are virtual machines on a big DatastoreServer. So in all cases when they need maintenance/modification I use Windows RDP to get to them. If you connect to a machine via RDP, Windows first checks certificate updates online. If your server/client doesn't have internet, it hangs for 10-20 seconds before continuing connection.

I make a lot of RDP connections each day. I save hours on not staring at the message: "securing remote connection":) +1 for disabling certif.service!

womble
  • 95,029
  • 29
  • 173
  • 228
Tommie84
  • 41
  • 1
  • 1
    Although I understand, that is a TERRIBLE reason :-) There are better ways to do this. I ran into a similar problem (with SharePoint checking certificates and being slow as result) years a ago. You can find several solutions and workarounds at http://blog.muhimbi.com/2009/04/new-approach-to-solve-sharepoints.html – Jeroen Ritmeijer Aug 29 '16 at 08:31
3

I would not agree that it is common to disable this. A better way to phrase it would be to ask why someone would disable it. And a better solution for your problem would be for the installer to check for the root/intermediate CA certificates and install them if not present.

The Trusted Root CA program is essential. A TON of applications would just not work as expected if it were turned off widely. Sure, there may be some organizations that disable this feature, but that's really up to the organizations, based on their requirements. It is a flawed assumption that any application that requires an external dependency (root certificate) would always work without testing it. Both developers of applications and organizations that disable this feature own the responsibility of ensuring the external dependency (root certificate) is present. That means if an organization disables this, they know to expect this issue (or will soon learn about it).

It's also worth noting that one useful purpose of the Trusted Root CA program mechanism (dynamic installation of root CA certificates) is that it isn't practical to install all or even most of the well-known/trusted root CA certificates. Some components in Windows break if there are too many certificates installed, so the only feasible practice is to install only the certificates that are needed, when they are needed.

http://blogs.technet.com/b/windowsserver/archive/2013/01/12/fix-available-for-root-certificate-update-issue-on-windows-server.aspx

"The issue is this: the SChannel security package used to send trusted certificates to clients has a limit of 16KB. Therefore, having too many certificates in the store can prevent TLS servers from sending needed certificate information; they start sending but have to stop when they reach 16KB. If clients don’t have the right certificate information, they cannot use services requiring TLS for authentication. Because the root certificate update package available in KB 931125 manually adds a large number of certificates to the store, applying it to servers results in the store exceeding the 16KB limit and the potential for failed TLS authentication. "

Greg Askew
  • 34,339
  • 3
  • 52
  • 81
  • 2
    Thanks for your answer, but based on our real world experience it IS common and I don't think any server admin would be happy for us to install a root cert if they have made the decision to not even trust Microsoft with this. Also.... our installer cannot run without the cert so chicken...egg... – Jeroen Ritmeijer Jan 27 '16 at 17:26
  • Understood, but you also learned that you own testing the external dependency, documenting this for installation, and communicating the requirement to the customer. That is real world experience. I doubt that your customer base would qualify as empirical data to support a conclusion that it is "common" that this feature is disabled. – Greg Askew Jan 27 '16 at 17:33
  • @Muhimbi: As a practical workaround, you could provide instructions for admins to manually install the required certificate, if they don't want to allow automatic root certificate updates. – Ilmari Karonen Jan 28 '16 at 11:34
  • @IlmariKaronen, we already do. For some reason it doesn't always work, even when they import it into the correct store. Perhaps it is related to many servers not being connected to the internet so they cannot verify the cert's validity. – Jeroen Ritmeijer Jan 28 '16 at 15:00
1

I know this is an older thread; however, I would like to submit an alternative solution. Use a certificate authority (ROOT CA) other than the one you are using. In other words, switch your signing certificate to one that has a much older, approved root CA.

DIGICert offers this when requesting a cert. While this might not be your default root CA within your DIGICert account, it is an option available when submitting the CSR to them. BTW, I do not work for DIGICert nor have I any gain by recommending them.. I simply feel this pain and have spent way too many hours on saving $1000 US on a cheap cert when I could have bought a more expensive cert and spent much less time dealing with the support issues. This is simply an example. There are other certificate providers offering the same thing.

99% Compatibility DigiCert Root Certificates are among the most widely-trusted authority certificates in the world. As such, they are automatically recognized by all common web browsers, mobile devices, and mail clients.

Caveat - if you select the correct root CA when making the CSR.

Edwin
  • 19
  • 2