5

Stay a while and listen. This is going to be a long one. We've been having a problem the last couple of months with Updates on very specific Microsoft Server 2003 R2 Machines. This problem has confounded me and our resident server administrator to no end as we can not make heads or tails of what exactly is causing the problem.

Here are the details:

Our Network: This network is a medium/small sized organization with approximately (500) computers and 20+ servers. The primary DC and and critical services are all on Server 2012 (r1) while some of the older application/services run on 2008 R2. Additionally, there are a smattering of Server 2003 R2 machines running mostly legacy applications and services which will either be replaced or become obsolete next year.

The issue: The issue is that among these few 2003 R2 servers, three of them are exhibiting strange behavior. We were first alerted to this behavior when they suddenly (in the same Update cycle) failed to properly install updates and we were alerted through our monitoring software.

Windows Update: On the problem machines, we are able to connect to the Windows Update URL just fine, but any attempt to actually check for updates will result in a

Error number: 0x80244016

in the Windows Update page. This error code corresponds to an invalid URL. As we are going through WSUS the URL should be and (we have checked on all the servers) to the FQDN and port of our WSUS server \NOC3.sub.domain.com:8888 . We found that the traffic was getting blocked at this point through a winHTTP proxy setting. So, when this problem occurs, we can use

proxycfg -h

and clear the proxy settings. Run windows updates and it will successfully connect.

This is where it gets interesting: Now, this would be all good and fine, but it gets better. So the next windows update comes around last month, and lo-and-behold, the exact same issue with the exact same servers occurs again. Now, we are a little suspect about why the same proxy settings have reapplied themselves. Naturally, we believed it may have been an old policy that was not properly disabled or possibly just incorrectly applied. So we both scoured all applicable GPO's for the servers in GPM on the Domain controller looking for any applied policies that may have anything to do with this strange, now reoccurring, policy(?) that is causing our servers to somehow revert to this (wrong) proxy server setting. There were no policies applied that have any mention of proxy servers. Then, I ran a RSOP on the machine.

It gets stranger: So after running the RSOP, everything seemed to have loaded fine. Until I attempted to open the Local Machine --> Administrative Templates folder. First of all, it took an inordinately long time to open the folder. Then when it finally did run, error messages associated with AER_####.adm would appear. Not only errors, but you would have policy settings in multiple languages displayed in the same folder. Now, I do not read Arabic, Chinese, Italian, or Japanese very well, but I am going to assume that these are duplicates of the policy setting in different languages. I found that by clearing the .adm files from the %root%\system\inf folder would, at the very least, allow for the RSOP to load the Administrative templates without error. *Note: later research showed that the AER####.adm files are associated with error reporting; which is strange when you consider what happens next:

First attempt at resolution: So, on a hunch, I moved the extraneous .adm files to another folder; then I ran a

gpupdate /force

on the affected server. *note: this server still had the 'stuck' proxy server setting in place. After a restart, the proxy settings seemed to have stayed with the no proxy setting which we want.

But alas, the joy was temporary: During this recent update cycle, we were alerted that the servers were once again not getting updates. The same problem, the same errant proxy server setting. *Note: the proxy server setting is pointed to an internal server which we once used as a web proxy so we are not thinking "security risk" atm. I also checked the RSOP once again, and the extraneous .adm files were back in place. So at this point, I am convinced that the problem lies on whatever is modifying our policy such that these new AER####.adm files are being generated (a policy or some external device/program which is making changes to our policy). I have found no references to anything like this anywhere on the web, and I do not understand the internal mechanics of Group Policy as they are applied to server 2003 R2 os to delve any deeper at this point. So I ask you, help!

Conclusion: So here is what I know for sure:

  1. The problem is only on 2003 R2 Servers
  2. The problem does not affect ALL 2003 R2 Servers
  3. There are no policies that are applied (or even active) that have any reference to this current proxy setting which seems to reapply itself at irregular intervals
  4. Updates will work fine after manually disabling proxy setting on the local servers
  5. Once again, it will not stay disabled

Any advice or insight into this issue would be great. Thanks for your time!

Get-HomeByFiveOClock
  • 448
  • 5
  • 13
  • 1
    At the very least, the console thing is a [known issue](http://support.microsoft.com/kb/969250) with 2003, so there's that. The proxy settings are controlled by a registry entry ..do you have *any* GPOs or central management (ie, SCCM) that could change this setting? – Nathan C Jul 09 '14 at 17:56
  • Also, in regards to your comment @Nathan C ; Not to mention, that running the gpupdate /force will actually *fix* the problem. So I fathom it's safe to say that the correct policies are running as correctly in these instances. What else do you think can change registry keys on the local machine without explicit admin creds? – Get-HomeByFiveOClock Jul 09 '14 at 19:37
  • 1
    Nothing I can think of off the top of my head. Have you tried setting a GPO to force the "direct" setting instead? – Nathan C Jul 09 '14 at 19:47
  • @Nathan C. damn good idea, I like you already! Nonetheless, I'm still curious as to the **cause** of the problem, but honestly, the priority to actually fix this thing is low as we are already able to manually run updates and the 03' servers are all planned to be off of the network by EOL for Server 2003 R2 – Get-HomeByFiveOClock Jul 09 '14 at 19:51
  • It is not clear if you checked the local security policy in addition to the domain policy. Can you speak to that. – Bin Jul 09 '14 at 19:51
  • @bin, no i did not. I was not aware that server '03 had any local security policies which would affect winHttP proxy settings, I'll look into that too. – Get-HomeByFiveOClock Jul 09 '14 at 19:54
  • I would say the only way you'd figure out what's resetting it would be to babysit the machine with ProcMon and watching for that key to change, but it's probably not worth the trouble and the massive log file. – Nathan C Jul 09 '14 at 20:01
  • 1
    @Yehaw IIRC just run gpedit.msc and it will default to the local security policy. Local Computer Policy > Computer Configuration > Administrative Templates > Windows Components > Windows Update > Specify intranet Microsoft update service location – Bin Jul 09 '14 at 20:02
  • I've thought about that too. Once again, everybody else has more-or less determined that to fix this problem would be more trouble than it's worth as we've moved pretty much all the critical/important services off those machines a couple years ago. At this point, this is almost more of a personal vendetta. :P – Get-HomeByFiveOClock Jul 09 '14 at 20:03
  • @Bin, checked it. All clear. Good suggestion though. – Get-HomeByFiveOClock Jul 09 '14 at 20:22
  • 1
    @yehaw You could try putting "updates.microsoft.com" for the service location in the local policy. I once had to do this to override a bad policy that was being forced on my servers. – Bin Jul 09 '14 at 20:35
  • @Bin, both your suggestions are tremendously helpful. As you suggested, a direct local policy to override what may be an errant policy resetting the proxy may work. Or, I was thinking something along the lines of a scheduled task to manually clear proxy settings at ~1hour intervals would do the trick as well. Once again, not quite a fix, but it'll get the job done. I'll let this question stick around for a few more days, if I get no more answers and your suggestions solve the problem I'll mark your suggestion as the "answer" AFAIC, Thanks again folks – Get-HomeByFiveOClock Jul 11 '14 at 18:33

0 Answers0