4

I have a webserver with Windows 2012 R2, running only one website, on ASP.NET 4.5.1, IIS 8.5. It does not have, and has never had, the AD role installed.

Last Sunday, the process lsass.exe, suddenly started using 20-40% CPU. We did not change anything on the server, or in the surrounding network. (No windows update happened at this point)

Since then, the website gradually deteriorates, having longer and longer response times, until the server is rebooted, then everything is back to normal, but it slowly goes downhill from there. The server now has to be rebooted every 4-12h, in order to keep the site up and running. The CPU usage on the server never reaches 100%, and there is also spare RAM available.

From New Relic, which is used to monitor the site and server, it looks like it is Request Queueing that is causing the poor performance, increasing from ~100ms after a reboot, to 10s+ after a few hours.

I have other servers, with the same website code, running in the same environment, with more load than this one, which do not show this behaviour. (Though they are on WS 2012 (not R2))

I have monitored network traffic, and see nothing strange there. The one thing that does stand out, and seem correlated, in Perfmon, is "Security per process statistics\Credential Handles", which keeps increasing for the lsass process, it starts at only a handful after reboot, and reaches 100.000+ after a few hours in operations. No other process is doing this, and lsass.exe on other servers, only have a few hundred "Credential Handles"

See graph over lsass.exe process over the last month: enter image description here

Graph of website response times, from New relic. Each peak demanded a reboot, server was also rebooted on smaller peaks, not very visible on this chart: enter image description here

Any advice on what might be wrong, and how to fix?

Edit: Snapshot of TCP/IP panel of Process Explorer on lsass.exe enter image description here

Snapshot of a typical thread spawned from lsass.exe, these pop up about every second in Process Explorer: enter image description here

Oskar Melen
  • 43
  • 1
  • 1
  • 3
  • Using Process Explorer, i see threads like: 5056 6.81 1,683,161,716 ntdll.dll!RtlFreeUnicodeString+0x1370 – Oskar Melen Mar 09 '15 at 14:57
  • Thread stack from Process explorer:ntoskrnl.exe!KeSynchronizeExecution+0x2246 ntoskrnl.exe!KeWaitForMultipleObjects+0x123e ntoskrnl.exe!KeWaitForMultipleObjects+0xcb9 ntoskrnl.exe!KeWaitForMutexObject+0x2c0 ntoskrnl.exe!_misaligned_access+0x7c ntoskrnl.exe!_misaligned_access+0xca9 ntoskrnl.exe!KeSynchronizeExecution+0x4133 KERNELBASE.dll!RegOpenKeyExW+0x3fe KERNELBASE.dll!RegQueryValueExW+0xe9 DNSAPI.dll!Reg_GetValueEx+0x177 DNSAPI.dll!DnsQueryConfigAllocEx+0xcc DNSAPI.dll!HostsFile_Close+0x141d DNSAPI.dll!Socket_RecvFrom+0x3cd DNSAPI.dll!DnsQuery_UTF8+0x46 netlogon.DLL!DsrGetDcNameEx2+.... – Oskar Melen Mar 09 '15 at 14:59
  • UPDATE: Never found a solution. Solved it by installing a new (VM) server and throwing out the old one. New server with exactly the same signature, does not have any problems. – Oskar Melen May 06 '15 at 13:33

2 Answers2

6

After enabling TLS 1.2 (also 1.0 and 1.2) on Windows 2012 R2 and 2008 R2 SP1, process lsass.exe began to overload CPU. I used the IISCrypto40, click "Best Practices", and in "Key Exchanges Enabled" left enabled only PKCS. After it, the load on the cpu returned to normal (5 - 10%). p.s. sorry for the poor english IISCrypto preferences

Алексей Книгин
  • 61
  • 1
  • 2
1

Check for malware. I've caused this by migrating meterpreter to lsass because it grants me more permissions.

I'd also look at process explorer, and see if you can see what threads are causing the high cpu.

Steve Butler
  • 1,016
  • 9
  • 19
  • I have run Spybot and 2 AV scanners with no results. Also have Kaspersky installed since server creation, so hopefully i am clean. Using Process Explorer, i see Threads popping up all the time; starting in "ntdll.dll", Start Adress: "RtlFreeUnicodeString" Not sure how to interpret that. – Oskar Melen Mar 09 '15 at 15:09
  • Under TCP/IP does it only show connections going back to itself? i.e. machine:12345 machine:0 – Steve Butler Mar 09 '15 at 21:23
  • Yes. I have a two occasional from the internal DNS, but otherwise, i have 2 on TCP, one on UDP, and two on TCPv6, from the same server. (Internal DNS, meaning: we have a big internal network, with native IP's and stuff. These webservers are in the DMZ between the two networks) – Oskar Melen Mar 09 '15 at 22:09