1

Last Saturday I had one of the Citrix servers in my cluster start complaining about losing the trust relationship with the domain. While this is not really common, it's not unheard of. Usually, the process is reset AD computer account, remove from the domain, re-join to the domain. Easy peasy.

With this server, however, the tables have turned. I followed my normal process of resetting the computer account and removing from the domain using a local admin account, but when I rebooted the server was still looking for the domain. Tried again thinking I had messed up somehow and received the same result.

I then tried restoring the server from a backup the night before (when I originally looked I had misread the backup logs and thought that particular server hadn't been backed up for a while; not uncommon with terminal servers) and experienced the same results. I even tried sysprepping the machine and after rebooting it came back up to the domain login prompt.

A friend suggested I try creating a new local admin account and removing from the domain that way. I created the account, rebooted, then the account was gone. I created it again and just logged out, then back in as the new admin and that got me in. I removed from the domain and rebooted with predictable results. No trace of the user account I had created and successfully logged in with. No user directory, nothing in the registry for profile list, nothing. After this happened I ran chkdsk and sfc /scannow just for giggles and both came up with no errors.

This is a Windows Server 2012 R2 running on VMware 6.5. I wouldn't put this much effort in normally when I could clone/sysprep/rejoin one of the other servers, but this server has a particular set of users with specific requirements. I repurposed another server for them in the meantime, but ideally, I'd like to get this one back into service.

When the event initially happened I did check the system time and the logs and found nothing out of the ordinary. The server was dead on with both of the domain controllers.

Sqrl
  • 11
  • 3
  • 1
    Sounds like the storage might be broken, or perhaps is shadowing writes and discarding them? Have you tried cloning to a new disk? – John Keates Jul 25 '18 at 01:55
  • I have not specifically, no. I did a Veeam complete restore with it which (I believe) writes it into a new disk. A good thing to check though, I can write some files to the desktop or something and see if they persist. – Sqrl Jul 25 '18 at 02:37
  • Yep, that is exactly what is happening. Going to shut down the server and clone it onto a different LUN to see if that has any affect. – Sqrl Jul 25 '18 at 02:49
  • Just checked another Citrix server on that same LUN and changes are persisting, so *hopefully* it is just a fouled vmdk or something. – Sqrl Jul 25 '18 at 03:00
  • Freshly cloned server on a new LUN does the same thing. Going to poke through the VMware settings (doubt this is the problem, I'm the only admin) and see if I can see anything there. – Sqrl Jul 25 '18 at 03:26

3 Answers3

0

Rather than go through all of that, try resetting the trust with Powershell, like this:

Reset-ComputerMachinePassword -Server domaincontroller -Credential Domain\Administrator

Or with netdom, like this:

netdom.exe resetpwd /s:domaincontroller /ud:Domain\Administrator /pd:domain administrator password

So... restore the server from backup again, and reset the trust with either of the two methods above.

joeqwerty
  • 108,377
  • 6
  • 80
  • 171
  • Thank you for that info. Regrettably, it did not work for either command. Rebooting the computer and trying to log in produced the same trust relationship error. – Sqrl Jul 25 '18 at 00:03
0

It seems that you are using a non persistent disk or, you enabled autoRevert on the VM.

You should check this link, about the advanced configuration parameter "autoRevert" and try to disable it.

Swisstone
  • 6,357
  • 7
  • 21
  • 32
0

In this instance, the problem stemmed from a registry key somehow losing a critical portion which caused a part of the Citrix XenApp VDA to hold onto changes then discard them at reboot. This article on Reddit has provided the answer:

https://www.reddit.com/r/sysadmin/comments/7dikt0/server_2012_r2_vm_keeps_reverting_back_after/

I had to reset the VM 3 times during boot to get to the recovery screen, enter into the command prompt, run RegEdit, load the HKLM/System hive from the D: (installed) drive, then change this key:

HKLM/System/ControlSet001/Control/Session Manager/BootExecute

to have these values

autocheck autochk *
PvsVmBoot

When I first opened it, only the line autocheck autochk * was there. I checked in ControlSet002 as well and that one was correct. I rebooted the machine and reset the trust relationship again and this time was able to log in.

Sqrl
  • 11
  • 3