10

We're planning to automate the creation of VMs for our build infrastructure so that we can:

  1. Scale the build resources based on demand, e.g. by adding more build agents when required and removing them when not required
  2. Recreate all or part of the build environment if / when machines die
  3. Duplicate the build environment when we need a test set up

One of the steps in this process is automating the creation of the VM base images (in our case using Hyper-V). For that we have a script that:

  1. Creates new VHDX from ISO with the Convert-WindowsImage script. We're currently using Windows 2012R2 but will be looking to get started with 2016 as soon as it is available.
  2. Adds an unattend script to the new VHDX with all the base configuration we need
  3. Updates the VHDX with the latest windows patches using the Apply-WindowsUpdate script
  4. Creates a new Hyper-V VM based on the VHDX and starts it
  5. Waits for the VM to boot and wait for the WinRM service to be ready to accept remote connections
  6. Waits for windows to complete the initial configuration and the configuration of the new patches
  7. Applies any further patches
  8. Reboots to complete the configuration of the latest patches
  9. Waits for windows to complete configuring the patches
  10. Pushes a sysprep script to machine and invokes that script. This runs sysprep and then turns the machine off
  11. Deletes the VM but keeps the VHDX
  12. Removes sysprep and unattend files from the VHDX and then compacts the VHDX
  13. Moves VHDX to template location and mark as read-only

The problem we're experiencing is in steps 6 and 9. Ideally we wait for all the configuration to be complete before we reboot / shut down the machine but there does not seem to be a way to detect that windows has finished the configuration stage.

When going through the UI it is very clear when either step is done because the log-in UI doesn't show up until the process is ready. However when using WinRM to remotely connect to the machine this is less clear because WinRM provides access to the machine before it is done with the configuration work.

So the question is what is the most fool proof way to detect over a remote connection that Windows has finished configuring updates etc. so that we can reboot / shut down the machine without causing issues later on.

------ EDIT -----

In the end we're using a modified version of Katherine's answer in that our script also waits for windeploy and ngen to complete. Given that ngen doesn't complete until well after the OS has finished initializing that works, and as a bonus the final VHDX will have all of the .NET framework ngen-ed which means we don't have to deal with that when we create new VMs of the template disk. Both script that we use to create the VHDX template and the scripts to create the local test environment are on github in case anybody is interested.

Petrik
  • 203
  • 2
  • 9

3 Answers3

6

This might sound like kind of a weird answer, but...

There's a PowerShell script for checking to see if there are available updates for Nagios. You could probably use this script or a variant for your purposes, without Nagios.

As for whether they're in progress, check for whether or not Wuauclt and TrustedInstaller are running. Microsoft's advice about updates on Server Core might help here:

Depending on the updates that are installed, you might need to restart the computer, although the system will not notify you of this. To determine if the installation process has completed, use Task Manager to verify that the Wuauclt or Trusted Installer processes are not actively running. You can also use the methods in the “Viewing installed updates” section to check the list of installed updates.

You can probably pull that information with something like Get-Process -Computername YourImage TrustedInstaller.exe. After both the Wuauclt and TrustedInstaller processes have finished, it should be safe to reboot.

Katherine Villyard
  • 18,510
  • 4
  • 36
  • 59
  • That script solves the problem of getting updates and detecting if a reboot is required, which is another problem we may need to solve, but the script does not deal with waiting for a reboot to complete to the point where the machine is ready to do work. – Petrik Jan 21 '16 at 21:25
  • You commented while I was editing. I added some info about detecting that state in Server Core, which is close to doing it remotely. – Katherine Villyard Jan 21 '16 at 21:33
  • 1
    I was being too comment happy. I will have a look at searching for Wuauclt or TrustedInstaller. – Petrik Jan 21 '16 at 21:34
  • I was being a little "hit post" happy myself. :) – Katherine Villyard Jan 21 '16 at 21:35
  • 1
    This approach almost worked for me, except that both TrustedInstaller and Wuauclt terminate before the initialization is done. After adding windeploy and ngen the script waits for the machine to complete all initialization (probably because ngen doesn't complete until well after the machine has finished initializing). – Petrik Feb 06 '16 at 10:41
3

Each Windows update patch will write several events in the Setup event log.

  • Event ID 1 - Initiating changes for package KB####
  • Event ID 4 - A reboot is necessary before package KB#### can be changed to installed state
  • Event ID 2 - Package KB#### was successfully changed to the Installed state

One way to determine all patches had been applied would be to loop a check on Event ID 4. Compare the time of that event to the current time. If no event ID 4's had been written for 5 or 10 minutes, then all pataches are probably done, and ready to reboot.

I'm not clear if you want to do the first reboot when patches are done installing (event4), or the second reboot after they are done configuring (event 2). This code does the former. Simply change the filterHashTable to event id 2 for the other reboot before your step 10.

$target = "bart"
$found = $false
while (-not $found) {
    $lastEvent4 = (get-winevent -comp $target -maxEvents 1 -filterHashTable @{ Logname='Setup'; id = '4';}).timeCreated
    if (((get-date) - $lastEvent4).totalMinutes -gt 10) {
        "do reboot"
        restart-computer -comp -$target
        $found = $true
    } else {
        "wait"
        start-sleep 60
    }
}
Clayton
  • 4,483
  • 16
  • 24
  • Wouldn't it work to write down the KB IDs of all packages that start installing, and assume completion only when no more updates are in-flight? – Simon Richter Jan 27 '16 at 11:29
  • The list of patches is going to change each month on patch Tuesday. Updating the process each month to use a new list would be an ongoing maintenance point... I thought what I suggested would be simpler. – Clayton Jan 27 '16 at 14:05
  • 1
    I meant as an extension to your answer: when the update starts installing (event 1), it is added to the list, and removed when it reports done (event 4). With some tweaking (failed updates, list reset during reboot?) it should be possible to determine whether there is still an installation going on. – Simon Richter Jan 27 '16 at 18:17
  • So for a fresh install there are no entries in the eventlog unfortunately. I haven't tried installing after the machine has initialized but I assume that this approach will work fine in that case. – Petrik Feb 06 '16 at 10:39
  • Fresh install? Confused. You've got a 13 step process here. You asked for assistance with steps 6 and 9. Your question was for "a way to detect that windows has finished the configuration stage" not a way to begin deployment of patches. – Clayton Feb 08 '16 at 14:36
  • That is correct. Step 6 is determining when the initial configuration of the machine is done, i.e. on first boot after creating a VHDX image from the windows install ISO (step 1), i.e. step 6 is the first time a VM is booted from the newly created VHDX. Step 9 is determining when the machine is done after the first reboot. Technically that's where your solution comes in but after trying it I don't see any entries in the eventlog, so I'm guessing those entries are not written when the machine first initializes even if we add offline patches to the VHDX before first boot. – Petrik Feb 09 '16 at 04:04
0

I have had good success with the following approach: Wait until Windows changes the startup type of the Windows Module Installer Service (aka TrustedInstaller) to Manual (Demand Start) - after a reboot. At that point , updates have completed installing.

The Trusted Installer process sometimes continues to run after patches have been installed ? However the service start up type is still reset to Manual.

You can verify for yourself, if above observation is consistent/correct, by looking at previous event log messages and correlating events between the System and Setup logs.

The start-up change to Windows Module Installer is logged as a 7040 System event , and it correlates to the last Event 2 in the Setup log , after a reboot.

I think, that when updates are first installed , this service is set to 'Auto Start' in case a reboot is needed. It is set back to 'Manual' when the last patch is installed (regardless of whether a reboot was required).

On some servers , I have noticed that trusted Installer startup is quickly toggled from Manual to Automatic and back, and this may occur every hour or so. I suspect it is some application that is regularly checking for updates. But, me experience, is that it generally seems safe to assume, if the start-up is Manual, then no patching is occurring.

ydrol
  • 1
  • 1