All DCs fail VerifyEnterpriseReferences and DNS RReg Test - Everything else works including replication to a brand new DC

Question

So, we recently added a new DC to our domain (Win 2008 R2 Enterprise) with the idea to replace our Win 2008 R2 Standard DC with a second Enterprise one - which will give us the 2 DCs on 2008 R2 Enterprise.

While adding this DC we also ended up raising the Forest and Domain level from 2003 to 2008.

Everything has replicated fine as far as I can tell. No issues with AD, SYSVOL or anything else.

I am making sure everything is good and tight before demoting the 2008 R2 Standard box.

All DCs are failing the VerifyEnterpriseReferences test with the following output:

   Starting test: VerifyEnterpriseReferences
     The following problems were found while verifying various important DN
     references.  Note, that  these problems can be reported because of
     latency in replication.  So follow up to resolve the following
     problems, only if the same problem is reported on all DCs for a given
     domain or if  the problem persists after replication has had
     reasonable time to replicate changes. 
        [1] Problem: Missing Expected Value
         Base Object: CN=DC2008S-0,OU=Domain Controllers,DC=domain,DC=com
         Base Object Description: "DC Account Object"
         Value Object Attribute Name: msDFSR-ComputerReferenceBL
         Value Object Description: "SYSVOL FRS Member Object"
         Recommended Action: See Knowledge Base Article: Q312862

        [2] Problem: Missing Expected Value
         Base Object:
        CN=DC2008E-0,OU=Domain Controllers,DC=domain,DC=com
         Base Object Description: "DC Account Object"
         Value Object Attribute Name: msDFSR-ComputerReferenceBL
         Value Object Description: "SYSVOL FRS Member Object"
         Recommended Action: See Knowledge Base Article: Q312862

        [3] Problem: Missing Expected Value
         Base Object:
        CN=DC2008E-1,OU=Domain Controllers,DC=domain,DC=com
         Base Object Description: "DC Account Object"
         Value Object Attribute Name: msDFSR-ComputerReferenceBL
         Value Object Description: "SYSVOL FRS Member Object"
         Recommended Action: See Knowledge Base Article: Q312862

        LDAP Error 0x20 (32) - No Such Object. 
     ......................... DC2008S-0 failed test VerifyEnterpriseReferences

Additionally, the DNS RReg test fails - I haven't looked into this yet in as much detail, but it is included in the dcdiag report so I figure I'll add it in here for now

     Summary of DNS test results:


                                        Auth Basc Forw Del  Dyn  RReg Ext
        _________________________________________________________________
        Domain: domain.com

           DC2008S-0                    PASS PASS PASS PASS PASS FAIL n/a  
           DC2008E-0                    PASS PASS PASS PASS PASS FAIL n/a  
           DC2008E-1                    PASS PASS PASS PASS PASS FAIL n/a  

     Total Time taken to test all the DCs:2 min. 52 sec.

     ......................... domain.com failed test DNS

The error points me to a KB article for 2003 server https://support.microsoft.com/en-us/help/312862/recovering-missing-frs-objects-and-frs-attributes-in-active-directory

Which I still tried to follow along with, just to see what I'd find.

Server-Reference appears to be filled out on all our of DCs. (ASDIEdit, root domain, default naming context, CN=System, CN=File Replication Service, CN=Domain System Volume (SYSVOL share), all 3 DCs are listed as a nTFRSMember, and the attributes have details filled in in serverReference.

It does not match completely what I pulled out, but I'm not 100% sure I'm looking in exactly the right places:

CN=NTDS Site Settings,CN=SITE_NAME,CN=Sites,CN=Configuration,DC=DOMAIN_NAME,DC=com

CN=NTDS Settings,CN=DC2008S-0,CN=Servers,CN=SITE_NAME,CN=Sites,CN=Configuration,DC=DOMAIN_NAME,DC=com

But the second value is true (with different DC names) for all 3 DCs.

If I run ntfrsutl ds I do get the (null) output however:

NTFRS CONFIGURATION IN THE DS
SUBSTITUTE DCINFO FOR DC
   FRS  DomainControllerName: (null)
   Computer Name            : DC2008E-0
   Computer DNS Name        : DC2008E-0.domain.com

And that output is true on all 3 DCs as well.

Again - as far as I can tell everything else seems to be working great. We floated the new DC and updated the functional levels 5 days ago now. I'm not sure why I'm getting these failures and would like to get it cleaned up before continuing with the decommissioning.

Additional details:

I ran the script "Testing SYSVOL Replication Latency/Convergence Through PowerShell" and everything seems to have come up roses:

Name                      PDC   Site Name  DS Type    IP Address OS Version
----                      ---   ---------  -------    ---------- ----------
DC2008S-0.domain.com      FALSE sitename   Read/Write 10.1.1.3   Windows Server 2008 R2 Standard
DC2008E-0.domain.com      TRUE  sitename   Read/Write 10.1.1.27  Windows Server 2008 R2 Enterprise
DC2008E-1.domain.com      FALSE sitename   Read/Write 10.1.1.28  Windows Server 2008 R2 Enterprise

Which is all correct and the report spat out a positive result!

  ====================== CHECK 6 ======================

  REMARK: Each DC In The List Below Must Be At Least Accessible Through SMB Over TCP (445)

  * Contacting DC In AD domain ...[DC2008E-1.domain.COM [SOURCE RWDC]]...
     - DC Is Reachable...
     - Object [sysvolReplTempObject20180926163805.txt] Exists In The NetLogon Share

  * Contacting DC In AD domain ...[DC2008S-0.domain.COM]...
     - DC Is Reachable...
     - Object [sysvolReplTempObject20180926163805.txt] Now Does Exist In The NetLogon Share

  * Contacting DC In AD domain ...[DC2008E-0.domain.COM]...
     - DC Is Reachable...
     - Object [sysvolReplTempObject20180926163805.txt] Now Does Exist In The NetLogon Share

  Start Time......: 2018-09-26 16:38:05
  End Time........: 2018-09-26 16:38:11
  Duration........: 6.20 Seconds

  Deleting Temp Text File...

  Temp Text File [sysvolReplTempObject20180926163805.txt] Has Been Deleted On The Target RWDC!


Name                                    Site Name  Time
----                                    ---------  ----
DC2008E-1.domain.COM [SOURCE RWDC]      sitename    0
DC2008S-0.domain.com                    sitename    6.17
DC2008E-0.domain.com                    sitename    6.20

More details:

I also ran the script "Testing Active Directory Replication Latency/Convergence Through PowerShell" to verify AD replication

Name                      Domain        GC   FSMO                Site Name  DS Type    IP Address OS Version
----                      ------        --   ----                ---------  -------    ---------- ----------
DC2008S-0.domain.com      domain.com   TRUE  .....               sitename Read/Write 10.1.1.3   Windows Server 2008 R2 Standard
DC2008E-0.domain.com      domain.com   TRUE  SCH/DNM/PDC/RID/INF sitename Read/Write 10.1.1.27  Windows Server 2008 R2 Enterprise
DC2008E-1.domain.com      domain.com   TRUE  .....               sitename Read/Write 10.1.1.28  Windows Server 2008 R2 Enterprise

All the DCs come up correct in the Forest and then the Domain check (domain output listed above. Sees they all have a global catalog and DC2008E-0 has all our FSMO roles)

====================== CHECK 15 ======================

  REMARK: Each DC In The List Below Must Be At Least Accessible Through LDAP Over TCP (389)
  REMARK: Each GC In The List Below Must Be At Least Accessible Through LDAP-GC Over TCP (3268)

  * Contacting DC In AD domain ...[DC2008E-1.domain.COM [SOURCE RWDC]]...
     - DC Is Reachable...
     - Object [CN=adReplTempObject20180926164916,CN=Users,DC=domain,DC=com] Exists In The Database

  * Contacting DC In AD domain ...[DC2008S-0.domain.COM]...
     - DC Is Reachable...
     - Object [CN=adReplTempObject20180926164916,CN=Users,DC=domain,DC=com] Now Does Exist In The Database

  * Contacting DC In AD domain ...[DC2008E-0.domain.COM]...
     - DC Is Reachable...
     - Object [CN=adReplTempObject20180926164916,CN=Users,DC=domain,DC=com] Now Does Exist In The Database

  Start Time......: 2018-09-26 16:49:16
  End Time........: 2018-09-26 16:49:32
  Duration........: 15.59 Seconds

  Deleting Temp Contact Object...

  Temp Contact Object [CN=adReplTempObject20180926164916,CN=Users,DC=domain,DC=com] Has Been Deleted On The Target RWDC!


Name                                    Domain        GC   Site Name   Time
----                                    ------        --   ---------   ----
DC2008E-1.domain.COM [SOURCE RWDC]     domain.com    TRUE sitename     0
DC2008E-0.domain.com                   domain.com    TRUE sitename    15.59
DC2008S-0.domain.com                   domain.com    TRUE sitename    2.20

Again, everything looks like it is replicating well. Or is 15 seconds considered too long? Is that delay what is causing me agita on the dcdiag test?

Another update!

I've verified that the SOA Serial number in each zone on each DC matches.

I also went through all the subdirectories and records in the _msdcs zone and everything there matches 100% as well.

Anything less than a minute on those tests is just fine for a domain with a single local site. When you have multiple sites across network links the times can be longer depending on configuration and speeds. I believe the default for a cross-site link is 15 minutes. — duct_tape_coder, Sep 26 '18 at 21:18

duct_tape_coder · Accepted Answer · 2018-09-26T21:27:19.203

2

Sounds like you're having a SYSVOL replication issue, I'd recommend using the following Powershell tool to test the SYSVOL replication: https://gallery.technet.microsoft.com/Testing-SYSVOL-Replication-c3e9dc68

Once you've confirmed that SYSVOL replication has an issue, you can resolve by performing a non-authoritative SYSVOL restore. https://support.microsoft.com/en-us/help/290762/using-the-burflags-registry-key-to-reinitialize-file-replication-servi

If non-authoritative doesn't work, you can use an authoritative restore but be careful as that's more dangerous.

Once you've resolved your SYSVOL replication, I would recommend doing a migration from FRS replication to DFS-R replication. https://blogs.technet.microsoft.com/filecab/2014/06/25/streamlined-migration-of-frs-to-dfsr-sysvol/

For reference, there is also a DFS-R set of resyncronization steps if needed: https://support.microsoft.com/en-us/help/2218556/how-to-force-an-authoritative-and-non-authoritative-synchronization-fo

Jorge also provides an ADDS replication check powershell script that I would recommend running: https://gallery.technet.microsoft.com/Testing-Active-Directory-94e61e3e

Folks often forget that Active Directory replication is made of two separate halves, ADDS and SYSVOL. If either side fails, you'll end up with big problems. On top of that, FRS and DFS-R are infamous for failing silently with disastrous consequences to GPO. The ADDS side of GPOs will match between DCs but the SYSVOL side ( which contains the actual instructions for GPO) will not.

Not sure about the RREG issue but I would confirm that your reverse DNS look up zones are set up and replicating correctly. You can confirm the serial numbers of the zone between the two DCs for convergence. Additionally, review the _msdcs forward zone for errors including incorrect NS entries.

Upon further discussion with OP, I found this link: https://social.technet.microsoft.com/Forums/windowsserver/en-US/2ce07c3f-9956-4bec-ae46-055f311c5d96/dcdiag-test-failed-on-verifyenterprisereferences?forum=winserverDS

It seems his original error "failed test VerifyEnterpriseReferences" is a known and expected error after migrating a domain from 2003 level to 2008 level but before migrating from FRS to DFS-R SYSVOL replication. The error can be safely ignored but best practices involve completing the DFS-R migration.

edited Sep 26 '18 at 21:27

answered Sep 26 '18 at 19:03

duct_tape_coder

755
3
12

This is interesting. When I started here, only DC2008S-0 was really working. DC2008E-0 was online, and replicating AD, but GPOs were messed up and once I dug into it I found it was not replicating SYSVOL. I did exactly what you mention, and actually had to do the authoritative restore (which wasn't as bad as one might think, since we really only had 1 fully valid DC at that point). So, I will work through this from the top. In the meantime, do you think its safe to cut DC2008S-0 loose right now, or should I fix everything first? – Sam K Sep 26 '18 at 20:20
Ok, after reviewing the script I ran it, and it seems to run just fine and report everything is good (I updated the top post). Now I'm at even more of an impasse about where to proceed from here. – Sam K Sep 26 '18 at 20:46
1

@SamK Since that's not the PDC, it's probably safe to cut that DC loose but I would confirm with the ADDS script that everything has replicated first. Next, I would take a look through Event Viewer for any kind of errors. I found this link https://social.technet.microsoft.com/Forums/windowsserver/en-US/2ce07c3f-9956-4bec-ae46-055f311c5d96/dcdiag-test-failed-on-verifyenterprisereferences?forum=winserverDS which also recommends doing a DFS-R migration as I suggested in my original answer. – duct_tape_coder Sep 26 '18 at 21:02
1

@SamK Reading further in that post, it seems your error is a known expected error and can be safely ignored but the DFS-R migration is the resolution. – duct_tape_coder Sep 26 '18 at 21:04
1

Oh, fantastic! I'd just finished updating the top post, but yes the SYSVOL and ADDS scripts both passed, and the DNS zones all look good. Thank you very much for the help, I will likely proceed with migrating to DFS-R and cutting the DC loose after reviewing the link you posted - thanks again! – Sam K Sep 26 '18 at 21:13
Just to add - yep, we are not mid DFSR migration, the global state is that the migration is not yet initialized. I wouldn't be shocked if a previous had admin started that, never finished it, and no one noticed. But I do see my exact scenario in the known errors list now. I am going to link the KB that lists all the known DCDIAG errors, just to be thorough: https://support.microsoft.com/en-us/help/2512643/dcdiag-exe-e-or-a-or-c-expected-errors The real depressing part is I was in that same technet post earlier and I took a breadcrumb from higher up and ran with it, without reading down :( – Sam K Sep 26 '18 at 21:27
Don't be too hard on yourself or your predecessors. DFS-R is only available after migrating to domain level 2008 or higher. FRS was the only option for 2003. Migrating to DFS-R is supposed to be one of the steps in the domain level upgrade process but it may not be in official documentation as it's not technically necessary. P.S. Get started on reading up for that 2016 migration! "On January 14, 2020, Microsoft will be officially ending its support for Windows Server 2008 R2 editions." – duct_tape_coder Sep 26 '18 at 21:33
LOL - well thanks. OH! I see, so I created the error in upgrading the functional domain level! Wonderful :D No wonder - I thought all of this was working so great when I did the initial authoritative restore. It really knocked me off my high horse when I went to start cutting the old DC loose and this stuff started popping up. – Sam K Sep 26 '18 at 21:41

All DCs fail VerifyEnterpriseReferences and DNS RReg Test - Everything else works including replication to a brand new DC

1 Answers1