15

In 3 separate systems, the following event is being logged many times (between 30 to 4,000 times a day depending on the system) on the domain controller server:

An account failed to log on.

Subject:
    Security ID:        SYSTEM
    Account Name:       %domainControllerHostname%$
    Account Domain:     %NetBIOSDomainName%
    Logon ID:       0x3E7

Logon Type:         3

Account For Which Logon Failed:
    Security ID:        NULL SID
    Account Name:       
    Account Domain:     

Failure Information:
    Failure Reason:     Unknown user name or bad password.
    Status:         0xc000006d
    Sub Status:     0xc0000064

Process Information:
    Caller Process ID:  0x1ec
    Caller Process Name:    C:\Windows\System32\lsass.exe

Network Information:
    Workstation Name:   %domainControllerHostname%
    Source Network Address: -
    Source Port:        -

Detailed Authentication Information:
    Logon Process:      Schannel
    Authentication Package: Kerberos
    Transited Services: -
    Package Name (NTLM only):   -
    Key Length:     0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
    - Transited services indicate which intermediate services have participated in this logon request.
    - Package name indicates which sub-protocol was used among the NTLM protocols.
    - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

This event is slightly different to all of the others that I've found during research but I have determined the following:

  1. Event ID: 4625. "An account failed to log on".
  2. Logon Type: 3. "Network (i.e. connection to shared folder on this computer from elsewhere on network)".
  3. Security ID: NULL SID. "A valid account was not identified".
  4. Sub Status: 0xC0000064. "User name does not exist".
  5. Caller Process Name: C:\Windows\System32\lsass.exe. Local Security Authority Subsystem Service (LSASS), is a process in Microsoft Windows operating systems that is responsible for enforcing the security policy on the system. It verifies users logging on to a Windows computer or server, handles password changes, and creates access tokens. It also writes to the Windows Security Log.
  6. Workstation Name: SERVERNAME. The authentication request is being submitted by or via the domain controller itself.

Affected systems' similarities:

  1. Server Operating System: Windows Small Business Server 2011 or Windows Server 2012 R2 Essentials
  2. Desktop Operating System: Windows 7 Professional (generally)

Affected systems' differences:

  1. Antivirus
  2. Active Directory-integrated Internet filtering
  3. Desktop cached logons
  4. Roles (Exchange, backup, etc)

Some interesting things I've noticed in the most severely affected system:

  1. We recently started synchronizing Active Directory and Office 365 user account passwords via Windows Server 2012 R2 Essentials' Office 365 integration. The integration requires an Office 365 administrator's password and the security policy to be escalated. The synchronization requires each user account to be assigned to the corresponding Microsoft online account which requires the account's password to be changed on next logon. We also added their primary email domain as a UPN suffix in Active Directory Domains and Trusts and changed all user accounts' UPN to their email domain. Effectively, this allowed them to logon to the domain and Office 365 using their email address and password. However, since doing this the number of events logged per day has increased from ~900 to ~3,900. Note: none of the administrative or job-based (backup, scanner, etc) user accounts have been modified and no users are having issues accessing any parts of the system.
  2. The bulk of the events seem to be logged at regular intervals usually every 30 or 60 minutes except for ~09:00 which is when the users arrive at work: 2015/07/02 18:55
    2015/07/02 19:25
    2015/07/02 19:54
    2015/07/02 20:25
    2015/07/02 20:54
    2015/07/02 21:25
    2015/07/02 22:24
    2015/07/02 23:25
    2015/07/03 00:25
    2015/07/03 01:24
    2015/07/03 01:55
    2015/07/03 02:24
    2015/07/03 02:55
    2015/07/03 03:55
    2015/07/03 04:55
    2015/07/03 05:54
    2015/07/03 06:25
    2015/07/03 07:25
    2015/07/03 08:24
    2015/07/03 08:27
    2015/07/03 08:49
    2015/07/03 08:52
    2015/07/03 08:54
    2015/07/03 08:56
    2015/07/03 08:57
    2015/07/03 09:00
    2015/07/03 09:01
    2015/07/03 09:03
    2015/07/03 09:06
    2015/07/03 09:08
    2015/07/03 09:10
    2015/07/03 09:12
    2015/07/03 09:13
    2015/07/03 09:17
    2015/07/03 09:13
    2015/07/03 09:25
    2015/07/03 10:24
    2015/07/03 11:25

  3. The following event is logged on the terminal / remote desktop services server though nowhere near as many times:

    An account failed to log on.

Subject:
    Security ID:        NULL SID
    Account Name:       -
    Account Domain:     -
    Logon ID:       0x0

Logon Type:         3

Account For Which Logon Failed:
    Security ID:        NULL SID
    Account Name:       %terminalServerHostname%
    Account Domain:     %NetBIOSDomainName%

Failure Information:
    Failure Reason:     Unknown user name or bad password.
    Status:         0xC000006D
    Sub Status:     0xC0000064

Process Information:
    Caller Process ID:  0x0
    Caller Process Name:    -

Network Information:
    Workstation Name:   %terminalServerHostname%
    Source Network Address: %terminalServerIPv6Address%
    Source Port:        %randomHighNumber%

Detailed Authentication Information:
    Logon Process:      NtLmSsp 
    Authentication Package: NTLM
    Transited Services: -
    Package Name (NTLM only):   -
    Key Length:     0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
    - Transited services indicate which intermediate services have participated in this logon request.
    - Package name indicates which sub-protocol was used among the NTLM protocols.
    - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

So, in summary, it definitely seems to be related to network access from desktop computers using staff user accounts but I can't see how.

Update 2015/08/25 08:48:

In the most severely affected system I have done the following to isolate the issue and after each reverted the change:

  1. Shut down the terminal / remote desktop services server and the generic failed logons did continue.
  2. Disconnected the domain controller server from the network and the generic failed logons did continue.
  3. Rebooted the server into Safe Mode with no networking and the generic failed logons did not continue.
  4. Stopped and disabled all "unnecessary" services (monitoring agent, backup, network filtering integration, TeamViewer, antivirus, etc) and the generic failed logons did continue.
  5. Stopped and disabled Windows Server Essentials services (WseComputerBackupSvc, WseEmailSvc, WseHealthSvc, WseMediaSvc, WseMgmtSvc, and WseNtfSvc) and the generic failed logons did not continue.
  6. Eventually, stopped and disabled the Windows Server Essentials Management Service (WseMgmtSvc) and the generic failed logons did not continue.

I have double-checked that the Windows Server Essentials Management Service (WseMgmtSvc) is responsible for these generic failed logons by disabling it for a few days and there were no generic failed logons and enabling it for a few days and there were thousands of generic failed logons.

Update 2015/10/08 09:06:

On 2015/10/07 at 16:42 I found the following scheduled task:

  • Name: "Alert Evaluations"
  • Location: "\Microsoft\Windows\Windows Server Essentials"
  • Author: "Microsoft Corporation"
  • Description: "This task periodically evaluates the health of the computer."
  • Account: "SYSTEM"
  • Triggers: "At 08:54 on 28/10/2014 - After triggered, repeat every 30 minutes indefinitely"
  • Actions: "Start a program: C:\Windows\System32\Essentials\RunTask.exe /asm:"C:\Windows\Microsoft.Net\assembly\GAC_MSIL\AlertFramework\v4.0_6.3.0.0__31bf3856ad364e35\AlertFramework.dll" /class:Microsoft.WindowsServerSolutions.NetworkHealth.AlertFramework.HealthScheduledTask /method:EvaluateAlertsTaskAction /task:"Alert Evaluations""

This timeframe almost exactly matches the behaviour above so I disabled it to see if it affects the issue.

On 2015/10/08 at 08:57 I found that only 47 of these generic failed logons were logged since at irregular intervals.

So, I have narrowed it down even further.

mythofechelon
  • 877
  • 3
  • 22
  • 38
  • What method did you use to setup your win7 machines? – strange walker Apr 30 '15 at 09:15
  • @strange walker It's likely that, in each of the 3 affected environments, the batch of initial PCs were setup as follows: a single PC was configured (drivers, software, etc), an image of the PC was created, the remaining PCs were imaged using the configured image, and then each PC was renamed and added to the domain via the Connector wizard. – mythofechelon Apr 30 '15 at 09:24
  • To be honest I would just ignore these events. Windows creates a myriad of security events, and this particular event is definitely not harmful. – Lucky Luke Apr 30 '15 at 13:16
  • @Lucky Luke Unfortunately, our monitoring system can't differentiate between failed logon events so we can't really increase the check's threshold in case we miss an actual issue. – mythofechelon Apr 30 '15 at 15:14
  • So you can't exclude/ignore events which contain "NULL SID" for example? What monitoring system do you use? – Lucky Luke Apr 30 '15 at 16:04
  • @Lucky Luke I don't believe MAXfocus' Automated Monitoring does, no. It seems to just check if the number of Audit Failure logs has surpassed the defined threshold. – mythofechelon May 01 '15 at 07:48
  • That's surprising, I would consider using a different monitoring solution (if possible), I'm actually in disbelief that this is not possible. You could also limit auditing, but you may then miss actual critical audit failures. But then again, you will probably miss them anyways since they will be masked by the nonsense failures you are getting on a regular basis. – Lucky Luke May 02 '15 at 03:16
  • 1
    @Lucky Luke We are considering it but that's a while off and it doesn't resolve the root cause, unfortunately, so I still need an answer to this. – mythofechelon May 05 '15 at 15:16
  • You may want to check if someone is trying to hack into your server if it's public facing – rboy Oct 16 '20 at 18:08

2 Answers2

7

This Event is usually caused by a stale hidden credential. Try this from the system giving the error:

From a command prompt run: psexec -i -s -d cmd.exe
From the new cmd window run: rundll32 keymgr.dll,KRShowKeyMgr

Remove any items that appear in the list of Stored User Names and Passwords. Restart the computer.

Mark Henderson
  • 68,316
  • 31
  • 175
  • 255
zea62
  • 89
  • 1
  • 2
  • There are no entries. Also, isn't that the same as Credential Manager? – mythofechelon Oct 08 '15 at 15:09
  • 1
    @mythofechelon - Yes, technically that is the "Credential Manager", but Credential Manager stores credentials on a per-user basis. Using psexec to open a SYSTEM cmd window and then run Credential Manager runs Credential Manager as the SYSTEM user, which is the local computer account. – Thomas Nov 05 '19 at 18:32
1

It seems that the problem was caused by the scheduled task "Alert Evaluations".

mythofechelon
  • 877
  • 3
  • 22
  • 38