I've got an issue with a PDC for a tree domain in our AD forest. It cannot replicate to the other DC's and it's clients are losing trust with the domain and cannot be reset using Test-computersecurechannel -repair and I cannot join/rejoin clients the domain.
This is a Server 2016 Forest with sites created as Tree Domains. The Forest is xyz.den.lcl and the other trees are xyz.atl.lcl (the broken one), xyz.sea.lcl, xyz.cle.lcl, xyz.qucy.lcl, and xyz.sat.lcl. All DC's are windows 2016 STD and most are VM's on 2016 Data center hyper-v clusters. CLEDC01 is the one excption and is a physical server.
I've kind of vomited a bunch of Info here but I suspect that all these results are because SChannel/Machine Password is Broken.
I believe this issue was initially caused by an AV Firewall which i have since uninstalled trying to fix this issue.Everything was good before installing A/V. I've since verfied that the RPC services are listening and reachable from/to all of the DC's using Portqry and RPCping.
I also fixed some missing SRV records that were missing for the Domain (_kerberos and _Kpassword were missing for the xyz.atl.lcl _msdcs directory). As far as i can tell DNS is good. I can resolve the DC (ATLVMDC01) by all it's server records, by it's CNAME GUID in the forest _msdcs zone, rDNS resolves correctly, its got the right priority on it's SRV records. I ran through everything at http://go.microsoft.com/fwlink/?linkid=5171
I've also tried Netdom resetpwd on the DC to reset the machine PW.
When I try to use nltest to get the DC for xyz.atl.lcl from any other Server/PC on any of the domains I see this.
C:\Windows\system32>nltest /dsgetdc:xyz.atl.lcl
Getting DC name failed: Status = 1355 0x54b ERROR_NO_SUCH_DOMAIN
From ATLVMDC01
C:\Windows\system32>nltest /dsgetdc:xyz.atl.lcl
DC: \\ATLVMDC01.xyz.atl.lcl
Address: \\<server ip>
Dom Guid: <GUID>
Dom Name: xyz.atl.lcl
Forest Name: xyz.den.lcl
Dc Site Name: ATLANTA
Our Site Name: ATLANTA
Flags: PDC GC DS LDAP KDC TIMESERV WRITABLE DNS_DC DNS_DOMAIN DNS_FOREST CLOSE_SITE FULL_SECRET WS DS_8 DS_9 DS_10
The command completed successfully
Network config for ATLVMDC01. All the other DC's are close to this config. Their primary DNS is their own IP's and Secondary DNS servers are the same 2 listed here.
Windows IP Configuration
Host Name . . . . . . . . . . . . : ATLVMDC01
Primary Dns Suffix . . . . . . . : xyz.atl.lcl
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : xyz.atl.lcl
xyz.den.lcl
xyz.qucy.lcl
xyz.cle.lcl
xyz.sat.lcl
Ethernet adapter Ethernet 3:
Connection-specific DNS Suffix . : xyz.atl.lcl
Description . . . . . . . . . . . : Microsoft Hyper-V Network Adapter #3
Physical Address. . . . . . . . . : 00-15-5D-2C-6F-15
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::a174:d654:693c:79c6%5(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.1.245(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.1.1
DHCPv6 IAID . . . . . . . . . . . : 134223197
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-23-AB-58-66-00-15-5D-2C-6F-05
DNS Servers . . . . . . . . . . . : 192.168.1.245
172.18.32.165
172.18.32.166
NetBIOS over Tcpip. . . . . . . . : Enabled
RPLADMIN /REPLSUM
On the problem DC "ATLVMDC01"
Source DSA largest delta fails/total %% error
ATLVMDC01 32d.00h:12m:06s 32 / 32 100 (1722) The RPC server is unavailable.
CLEDC01 12m:51s 0 / 32 0
DENVMDC01 25m:33s 0 / 26 0
DENVMDC02 32m:45s 0 / 42 0
QCYDC001 11m:51s 0 / 32 0
SATVMCAM01 25m:00s 0 / 42 0
SATVMDC01 32m:49s 0 / 10 0
From any other another DC
Source DSA largest delta fails/total %% error
ATLVMDC01 32d.00h:12m:06s 32 / 32 100 (1722) The RPC server is unavailable.
CLEDC01 12m:51s 0 / 32 0
DENVMDC01 25m:33s 0 / 26 0
DENVMDC02 32m:45s 0 / 42 0
QCYDC001 11m:51s 0 / 32 0
SATVMCAM01 25m:00s 0 / 42 0
SATVMDC01 32m:49s 0 / 10 0
Experienced the following operational errors trying to retrieve replication information:
8341 - ATLVMDC01.xyz.atl.lcl
On the Problem DC (ATLVMDC01) DCDIAG /TEST:DNS and DCDIAG /TEST:DNS /E all pass.
C:\Windows\system32>dcdiag /test:DNS
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = ATLVMDC01
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: ATLANTA\ATLVMDC01
Starting test: Connectivity
......................... ATLVMDC01 passed test Connectivity
Doing primary tests
Testing server: ATLANTA\ATLVMDC01
Starting test: DNS
DNS Tests are running and not hung. Please wait a few minutes...
......................... ATLVMDC01 passed test DNS
Running partition tests on : DomainDnsZones
Running partition tests on : xyz
Running partition tests on : ForestDnsZones
Running partition tests on : Schema
Running partition tests on : Configuration
Running enterprise tests on : xyz.den.lcl
Starting test: DNS
......................... xyz.den.lcl passed test DNS
On all other DC's I see this for DCDIAG /TEST:DNS /E
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = DENVMDC01
* Identified AD Forest.
[ATLVMDC01] LDAP bind failed with error 8341,
A directory service error has occurred..
Got error while checking if the DC is using FRS or DFSR. Error: A directory service error has occurred.The
VerifyReferences, FrsEvent and DfsrEvent tests might fail because of this error.
Done gathering initial info.
Doing initial required tests
Testing server: DENVER\DENVMDC01
Starting test: Connectivity
......................... DENVMDC01 passed test Connectivity
Testing server: DENVER\DENVMDC02
Starting test: Connectivity
......................... DENVMDC02 passed test Connectivity
Testing server: QUINCY\QCYDC001
Starting test: Connectivity
......................... QCYDC001 passed test Connectivity
Testing server: ATLANTA\ATLVMDC01
Starting test: Connectivity
Got error while checking LDAP and RPC connectivity. Please check your firewall settings.
......................... ATLVMDC01 failed test Connectivity
Testing server: CLEVELAND\CLEDC01
Starting test: Connectivity
......................... CLEDC01 passed test Connectivity
Testing server: SANANTONIO\SATVMDC01
Starting test: Connectivity
......................... SATVMDC01 passed test Connectivity
Testing server: SANANTONIO\SATVMCAM01
Starting test: Connectivity
......................... SATVMCAM01 passed test Connectivity
Doing primary tests
Testing server: DENVER\DENVMDC01
Testing server: DENVER\DENVMDC02
Testing server: QUINCY\QCYDC001
Testing server: ATLANTA\ATLVMDC01
Testing server: CLEVELAND\CLEDC01
Testing server: SANANTONIO\SATVMDC01
Testing server: SANANTONIO\SATVMCAM01
Starting test: DNS
Starting test: DNS
Starting test: DNS
Starting test: DNS
Starting test: DNS
Starting test: DNS
Starting test: DNS Starting test: DNS
DNS Tests are running and not hung. Please waita few minutes...
ATLVMDC01 failed test DNS
SATVMDC01 passed test DNS
DENVMDC01 passed test DNS
SATVMCAM01 passed test DNS
CLEDC01 passed test DNS
DENVMDC02 passed test DNS
QCYDC001 passed test DNS
Running partition tests on : DomainDnsZones
Running partition tests on : ForestDnsZones
Running partition tests on : Schema
Running partition tests on : Configuration
Running partition tests on : xyz
Running partition tests on : DomainDnsZones
Running partition tests on : xyz
Running partition tests on : DomainDnsZones
Running partition tests on : xyz
Running partition tests on : DomainDnsZones
Running partition tests on : xyz
Running partition tests on : DomainDnsZones
Running partition tests on : xyz
Running partition tests on : DomainDnsZones
Running partition tests on : xyz
Running enterprise tests on : xyz.den.lcl
Starting test: DNS
Test results for domain controllers:
DC: ATLVMDC01.xyz.atl.lcl
Domain: xyz.atl.lcl
TEST: Authentication (Auth)
Error: Authentication failed with specified credentials
TEST: Basic (Basc)
Error: No LDAP connectivity
Error: No WMI connectivity
No host records (A or AAAA) were found for this DC
DC: QCYDC001.xyz.qucy.lcl
Domain: xyz.qucy.lcl
TEST: Basic (Basc)
Warning: adapter [00000003] QLogic BCM57800 10 Gigabit Ethernet (NDIS VBD Client) has invalid DNS
server: <server ip> (<name unavailable>)
Warning: Adapter B0:83:FE:D2:39:FB has dynamic IP address (can be a misconfiguration)
Warning: Adapter B0:83:FE:D2:39:F9 has dynamic IP address (can be a misconfiguration)
DNS server: 8.8.8.8 (<name unavailable>)
3 test failure on this DNS server
Name resolution is not functional. _ldap._tcp.xyz.den.lcl. failed on the DNS server 8.8.8.8
DNS server: <server ip> (<name unavailable>)
1 test failure on this DNS server
PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server <server ip> Name resolution is not functional. _ldap._tcp.xyz.den.lcl. failed on the DNS server 192.168.130.3
Summary of DNS test results:
Auth Basc Forw Del Dyn RReg Ext
_________________________________________________________________
Domain: xyz.den.lcl
ATLVMDC01 FAIL FAIL n/a n/a n/a n/a n/a
QCYDC001 PASS WARN PASS PASS PASS PASS n/a
......................... xyz.den.lcl failed test DNS
Ran DCDIAG /test:checksecurityerror and recived the following.
C:\Windows\system32>dcdiag /test:checksecurityerror
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = DENVMDC01
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: DENVER\DENVMDC01
Starting test: Connectivity
......................... DENVMDC01 passed test Connectivity
Doing primary tests
Testing server: DENVER\DENVMDC01
Starting test: CheckSecurityError
Source DC ATLVMDC01 has possible security error (1722). Diagnosing...
No KDC found for domain xyz.atl.lcl in site ATLANTA (1355, NULL)
[ATLVMDC01] Unable to contact this DC. Cannot continue diagnosing errors with this DC.
......................... DENVMDC01 failed test CheckSecurityError
Running partition tests on : DomainDnsZones
Running partition tests on : ForestDnsZones
Running partition tests on : Schema
Running partition tests on : Configuration
Running partition tests on : xyz
Running enterprise tests on : xyz.den.lcl
