4

Team,

Exchange environment is all 2016, no mix. Parent and child domains exist, but the functional level of each domain and forest is 2012R2. All domain controllers have been 2012R2 until recently. The AD team (different from me) have introduced Server 2016 domain controllers and I think it has caused a problem. Here are the symptoms:

Mail gets stuck on servers in a parent domain with the destination to the child domain. Error on the queue is "454 4.7.0 Temporary Authentication error" and the correlating events in the event log are Event ID 2017, Source: MSExchangeTransport, the message is:

"Outbound authentication failed with error KdcUnknownEncryptionType for Send connector Intra-Organization SMTP Send Connector. The authentication mechanism is ExchangeAuth. The target is SMTPSVC/server fqdn."

Mail will flow from child domain to parent domain with no problem. Like I said earlier, I think the most significant change that has happened is simply introducting Server 2016 DCs. To fix it temporarily, I can reboot the server that the messages are stuck on and it will work for a while. This really seems like a Kerberos issue.

EDIT: We also have an ASA set up, but the supported encryption types between all my exchange servers, DCs, and this ASA are all "28".

Google searches suggest time sync issues, but I've checked both the Exchange servers and the domain controllers and things seem to be exactly on sync. I've also checked replication health and there doesn't appear to be any issues with that. I've also checked for duplicate or malformed SPNs and didn't find anything. Is there more that I can look into about the SPNs though? Like what kind of encryption they are requesting/using or something? I don't know much about Kerberos.

EDIT: As another note, using a GPO, I did remove the RC4-HMAC from the destination server. As a result, the klist tickets command did show the correct "AES...", but then my powershell was broken... "Possible causes" were:

  • "Username or password invalid"
  • "Kerberos used when no authentication method and no user name are specified"
  • "Kerberos accepts domain usernames, but not local user names."
  • "SPN for remote computer name and port doesn't exist"
  • "client and remote computers are in different domains and there is no trust."

Then it suggests to try adding destination computer in the WinRM TrustedHosts list.

Joseph
  • 208
  • 2
  • 10

2 Answers2

2

I think you may first check the SPN, you could run “SetSPN -L <ExchangeServerName>” command to check the SPN configuration. It should contain: 

SMTP/<ExchangeServerName>

SMTP/<ExchangeServerName>.example.com

SMTPSVC/<ExchangeServerName>

SMTPSVC/<ExchangeServerName>.example.com

If some missed, you could run "setspn -a <data>" to add.

And then, on the client side, run “klist tickets” in CMD to check the ticket type. Usually it should be “AES-256-CTS-xxx”.

Here is a related blog about ticket types: https://blogs.msdn.microsoft.com/openspecification/2011/05/30/windows-configurations-for-kerberos-supported-encryption-type/

In addition, according my research, beginning with Windows Server 2016, KDCs can support the PKInit freshness extension. Maybe you could also check this point.

Shaw
  • 339
  • 1
  • 4
  • Well, the `klist tickets` command is showing me something new. I have run the other setspn commands suggested and nothing is amiss there. The first ticket (which is the krbtgt) on the source server (parent domain) is the "AES..." you mentioned, but on the destination server (child domain), it is "RSADSI RC4-HMAC". My guess is that this has something to do with it? – Joseph Jan 02 '19 at 10:34
  • Also, to the link you provided, The GPO settings for this are the same on both domains and it shows up the same on both servers when I look in SECPOL.msc – Joseph Jan 02 '19 at 12:13
  • Don't just look in secpol, use gpresult.exe to output the resultant set of policy for the computer. You should have your AD team verify this on the 2016 DCs also. – twconnell Jan 02 '19 at 23:51
  • Gpresult shows the same configurations as well. – Joseph Jan 04 '19 at 04:52
1

It sounds like RC4 was an allowed Kerberos encryption type on the 2012 DCs, and your AD team introduced 2016 DCs with RC4 disabled. I say this with some confidence, because it is the recommended security setting on Server 2016. The goal is to remove RC4 from the environment, but not without first updating your mission critical applications to support AES. My suggestion would be to ensure the allowed Kerberos encryption types are configured the same on all DCs and then enable Kerberos service ticket success auditing on all Domain Controllers to see which applications are still trying to use RC4. It will show up in EventId 4769 with an encryption type of 0x17. If you see this, then RC4 is being used. You need to make sure all involved user accounts, computer accounts, and the Kerberos policy on each server (including the DCs) is configured properly for AES to work.

twconnell
  • 764
  • 4
  • 12
  • There are no 4769 events on the DCs. Sorry it took me a while to respond. We still have all our 2012R2 DCs as well, so it seems like it shouldn't stop mail flow. – Joseph Jan 04 '19 at 04:29
  • We do have an event ID of 16 logged multiple times, that reads something like this: "While processing TGS request for HTTP/, the account did not have a suitable key for generating a Kerberos ticket (the missing key has an ID of 8). The requested etypes were 18 17. The accounts available etypes were 18 17 23 -133 -128 24 -135. Changing or resetting the password for will generate a proper key." I don't see why it is requesting a key with an ID of 8. According to the link above about Kerberos types, that is "AES-128..." That's supposed to be supported. – Joseph Jan 04 '19 at 04:52
  • You need to enable Kerberos service ticket success/failure auditing on your Domain Controllers to see these events. This is the event you're looking for: https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4769 – twconnell Jan 05 '19 at 09:23
  • Enabling Kerberos auditing today, will let you know the results. – Joseph Jan 08 '19 at 02:51
  • Yeah, audit of event 4769 shows my Exchange servers on the child domain requesting RC4 encryption type for the tickets. Now I just need to figure out how to get it not to do that? – Joseph Jan 08 '19 at 04:01
  • Your issue isn't that they are using RC4 (that's another issue for later). The issue is your AD team introduced new Domain Controllers into the environment that are rejecting RC4 without first verifying if any applications would be affected. There's probably more than just Exchange. They should leave RC4 enabled on all DCs until the applications support AES. You will need to verify the supported encryption types attribute on all Exchange computer/service accounts is set to support AES. I'm very annoyed that this isn't on by default for all accounts. – twconnell Jan 12 '19 at 11:10
  • Yeah, I checked the DCs on the child domain (where mail would not flow to) and it was indeed not there. It was there on all the 2016 DCs on the parent domain, just not the child domain. You da man. Thanks. – Joseph Jan 14 '19 at 17:27
  • 1
    Glad to hear you got it squared away! – twconnell Jan 15 '19 at 23:41