We noticed our Automatic Deployment Rules for Software Updates failed to automatically download and apply this month's patches from Microsoft although they are correctly listed in the Catalog.
The Automatic Deployment Rules list their Last Error Code as 0X87D20417 and the Last Error Description as "Auto Deployment Rule Download failed". Re-running the rules manually reproduces this error. Deleting and recreating the Automatic Deployment Rules also reproduces the same error.
Looking at the SMS_RULE_ENGINE log shows the following errors:
Error Milestone 004 6/19/2013 3:42:21 PM SCCM.ad.example.com SMS_RULE_ENGINE 8706 Content download failed. Message: Failed to download one or more content files. Source: SMS Rule Engine.
Error Milestone 004 6/19/2013 3:42:07 PM SCCM.ad.example.com SMS_RULE_ENGINE 8706 Content download failed. Message: Failed to download one or more content files. Source: SMS Rule Engine.
Error Milestone 004 6/19/2013 2:45:44 PM SCCM.ad.example.com SMS_RULE_ENGINE 8706 Content download failed. Message: Failed to download one or more content files. Source: SMS Rule Engine.
Error Milestone 004 6/19/2013 2:43:29 PM SCCM.ad.example.com SMS_RULE_ENGINE 8706 Content download failed. Message: Failed to download one or more content files. Source: SMS Rule Engine.
If I look through the ruleengine.log (which is presumably the log file that the higher level SMS_RULE_ENGINE log within SCCM gets generated from) and coordinate the Package ID for the relevant Deployment Packages that the Automatic Deployment Rules are supposed to place these updates in I find the following:
Contents 16821586 is already present in the package "0040000F". Skipping download. SMS_RULE_ENGINE 6/19/2013 3:41:58 PM 9068 (0x236C)
Downloading contents (count = 10) for UpdateID 16829711 SMS_RULE_ENGINE 6/19/2013 3:41:58 PM 9068 (0x236C)
List of update content(s) which match the content rule criteria = {16821659,16821660,16821661,16821662,16821663,16821664,16821665,16821666,16821667,16821668} SMS_RULE_ENGINE 6/19/2013 3:41:58 PM 9068 (0x236C)
Downloading content with ID 16821659 in the package SMS_RULE_ENGINE 6/19/2013 3:41:58 PM 9068 (0x236C)
Failed to download the update from internet. Error = 4115 SMS_RULE_ENGINE 6/19/2013 3:41:58 PM 9068 (0x236C)
Failed to download ContentID 16821659 for UpdateID 16829711. Error code = 4115 SMS_RULE_ENGINE 6/19/2013 3:41:58 PM 9068 (0x236C)
At this point, I have three different errors that I believe are all generated by the same event. Of course, they may not be which is why they are all included here. I did coordinate the times in the log files and am reasonably confident they are all related to the issue with the Automatic Deployment Rules.
0X87D20417- From the SCCM Console's Automatic Deployment Rules8706- From the SCCM's Console's Monitoring SMS_RULE_ENGINE logError code = 4115- From the SCCM site server logs from [SCCMInstallationPath]\Logs\ruleengine.log
It looks like we're not able to download those updates. Apparently the place to troubleshoot that kind of thing is the PatchDownloader.log. And 'lo there is yet another error recorded there:
Trying to connect to the \\SCCM.ad.example.com\root\sms\site_REV namespace on the SCCM.ad.example.com machine. Software Updates Patch Downloader 6/19/2013 3:42:21 PM 9068 (0x236C)
Connected to \\SCCM.ad.example.com\root\sms\site_REV Software Updates Patch Downloader 6/19/2013 3:42:21 PM 9068 (0x236C)
GetContentFileInfoForDownload() failed for ContentID 16821994. hRes = 0x80041013 . Software Updates Patch Downloader 6/19/2013 3:42:21 PM 9068 (0x236C)
ERROR: DownloadContentFiles() failed with hr=0x80041013 Software Updates Patch Downloader 6/19/2013 3:42:21 PM 9068 (0x236C)
I can coordinate the Content IDs in the PatchDownloader.log back to the Error: 4115 entries recorded in the ruleengine.log so, as mentioned previously, I'm pretty sure were looking at the same event generating all of these different errors but if someone knows better please correct me.
If I use CMTrace's Error Lookup tool it tells me the following about hr=0x80041013.
Provider load failure
Source: Windows Management (WMI)
-----
And sure enough if I look at the WMI namespace the Software Updates Patch Downloader is connecting to, it doesn't look quite right:
\SCCM.ad.example.com\root\sms\site_REV
Our Site Code is actually 004 and funnily enough the first three letters of our organization start with REV. Mighty coincidental if you ask me. Furthermore, this is not the first SCCM install that existed here and it turns out the previous SCCM 2007 had the existing Boundaries, Collections and Packages migrated over to our new install. Smoking gun? Not quite. It used a different site code as well. Perhaps the REV site code was used for a temporary test installation of SCCM 2012? Perhaps not. Institutional knowledge doesn't have any record of the REV it and the migration we performed before I was hired.
However - our old PatchDownloader.log from the SCCM 2007 instance shows the Software Updates Patch Downloader connecting to the site_$SITECODE WMI namespace. Unfortunately, I don't have logs from our current 2012 install from May where I could confirm the correct WMI namespace is being referenced.
Trying to connect to the root\SMS namespace on the SCCM07.ad.example.com machine. Software Updates Patch Downloader 8/3/2011 3:18:37 PM 25128 (0x6228)
Connected to \\SCCM07.ad.example.com\root\SMS Software Updates Patch Downloader 8/3/2011 3:18:37 PM 25128 (0x6228)
Trying to connect to the \\SCCM07.ad.example.com\root\sms\site_DOR namespace on the machine. Software Updates Patch Downloader 8/3/2011 3:18:37 PM 25128 (0x6228)
Connected to \\SCCM07.ad.example.com\root\sms\site_DOR Software Updates Patch Downloader 8/3/2011 3:18:37 PM 25128 (0x6228)
Download destination = \\SCCM07.ad.example.com\WSUSContent\be128fa4-0c6b-418a-893d-3450e38c658d.1\windows-kb890830-v3.21.exe . Software Updates Patch Downloader 8/3/2011 3:18:37 PM 25128 (0x6228)
Contentsource = http://download.windowsupdate.com/msdownload/update/software/uprl/2011/07/windows-kb890830-v3.21_2aba440b72071ff17cad1ca2a39f0e40aa85c76e.exe . Software Updates Patch Downloader 8/3/2011 3:18:37 PM 25128 (0x6228)
Downloading content for ContentID = 31068, FileName = windows-kb890830-v3.21.exe. Software Updates Patch Downloader 8/3/2011 3:18:37 PM 25128 (0x6228)
OK. It really does look like an issue with WMI namespaces. Somewhere in the depths of SCCM, something is telling the Software Updates Patch Downloader to connect to \\SCCM.ad.example.com\root\sms\site_REV instead of \\SCCM.ad.example.com\root\sms\site_004.
On a WAG, I checked likely tables in SQL database for references to REV to no avail..
SELECT * FROM SysResList WHERE SiteCode = 'REV';
SELECT * FROM SiteControl WHERE SiteCode = 'REV';
SELECT * FROM SiteControlNotification WHERE SiteCode = 'REV';
SELECT * FROM Sites WHERE SiteCode = 'REV';
SELECT * FROM Sites_DATA WHERE SiteCode = 'REV';
SELECT * FROM SiteWork WHERE SiteCode = 'REV';
SELECT * FROM PkgServers WHERE sitecode = 'REV';
SELECT * FROM PkgStatus WHERE sitecode = 'REV';
To further complicate things I'm seeing multiple explanations of the 0x80041013 error.
WMI Troubleshooting Tips says that it is a failure to load a WMI provider:
WBEM_E_PROVIDER_LOAD_FAILURE - 0x80041013
The Provider Event Troubleshooting Classes are a great resource, but may be a little overwhelming. The MSFT_WmiProvider_LoadOperationFailureEvent class is one that I've found useful quite often. Most Provider Load Failures I've encountered have been the result of bad component registration (either in the registry or WMI), or permissions related.
Whereas WMI Error Constants from MSDN says that it is a permissions issue:
WBEM_E_ACCESS_DENIED 2147749891 (0x80041003) Current user does not have permission to perform the action.
The only other information I could find on the 0x80041013 error was a fellow who posted on TechNet who seems to have had the same issue as me, even down to the issue where he had a previous installation of SCCM that whose WMI namespace was being mistakenly referenced (e.g., site_REV instead of site_004). He ended up nuking the entire WMI namespace as well as the parts of the SMS_ProviderLocation. I'm not sure I want to do that.
At this point, it's been a long day, we need to get these servers patched and my head hurts. Any advice?
