Automating Virtual Machine File Server Updates and Reboots

This post is mainly to share a few scripts I have written which automate the Windows Update of my Server 2016 Core File Server which hosts all of the VHDX files for my Hyper-V cluster. As you might be aware restarting the server hosting the hard drives of running VMs can be pretty painful and usually not all that easy (have to pause or shutdown all the VMs and make sure you don’t break the Hyper-V cluster).

So the basic setup is:
File Server: FS01
Hyper-V Cluster: HVC01
Workstation: Windows 10 Pro x64

The first thing we need to do is install the remote WSUS tools onto the File Server and on the Workstation. Those can be found here:
Windows Update PowerShell Module

The easiest way to install these is to simply open up PowerShell as an Administrator and type the following:
Install-Module PSWindowsUpdate

Use Remote PowerShell to install it onto the File Server or Invoke it
Invoke-Command -ComputerName FS01 -ScriptBlock { Install-Module PSWindowsUpdate }

Now we need a script to do the following:
#1. Tell FS01 to check for updates, but do not restart
#2. Wait for FS01 to finish installing updates
#3. Check if a restart is required

So let’s get to it:

Function Update-FS01
{   
    param([switch]$OptimizeOnReboot)

    Write-Output "Starting FS01 update process."
    $Script = {Import-Module PSWindowsUpdate; Get-WUInstall -AcceptAll -IgnoreReboot -IgnoreUserInput | Out-File C:TempPSWindowsUpdate.log -Force;}
    Invoke-WUInstall -ComputerName FS01 -Script $Script -Confirm:$false
    
    Write-Output "Waiting for update task to complete."
    Start-Sleep -Seconds 30

    While ((Invoke-Command -ComputerName FS01 -ScriptBlock {Get-ScheduledTask | Where-Object { $_.TaskName -eq "PSWindowsUpdate" }}).State -match "Running|4")
    {
        Write-Output "Task still running. Waiting 30 seconds..."
        Start-Sleep -Seconds 30
    }

    Write-Output "FS01 Update task completed."
    if(Get-WURebootStatus -ComputerName FS01 -Silent)
    {
        Write-Output "Reboot Required"
        if ($OptimizeOnReboot)
        {
            Reboot-FS01 -Optimize
        }
        else
        {
            Reboot-FS01
        }
    }

    & "C:Program FilesNotepad++notepad++.exe" "\FS01C`$TempPSWindowsUpdate.log"
}

For now let’s ignore the Optimize parameter, I’ll come back to it. We start by creating a script block that we will send to FS01. That script block is going to start a scheduled task on FS01 which will run immediately. The options I am using here are:
AcceptAll: Do not ask for confirmation updates. Install all available updates.
IgnoreReboot: Do not ask for reboot if it needed, but do not reboot automaticaly.
IgnoreUserInput: Finds updates that the installation or uninstallation of an update can’t prompt for user input.

We’re then using Invoke-WUInstall to FS01 with our script and telling it don’t ask for confirmation. Which again, creates a scheduled task on FS01 called “PSWindowsUpdate” that runs immediately.

Now we’re going to wait for it to finish with the while command, which checks the status of the scheduled task every 30 seconds until it is complete.

Once complete we use “Get-WURebootStatus” to determine if a restart is required from the update. If it is we’ll launch another script that reboots the server, and optionally performs an optimization of the VHDX files prior to restarting FS01.

Finally, when the reboot is complete we’ll launch notepad++ to load the results of the Winodws Update Process. Note that C:Temp should exist on the File Server, if it doesn’t you should create it or choose a different location in the script block to save to. Also if you don’t have notepad++ just change the whole “C:Program FilesNotepad++notepad++.exe” to simply “notepad”

But wait, where’s the reboot and optimization script? Here:

Function Reboot-FS01
{
    Param
    (
        [Switch]$Optimize
    )
    
    # Additional time to wait after FS01 reboots for stability and cluster health
    $FSWWait = 30
    $VMStartWait = 30
    
    Workflow Stop-RunningVirtualMachines
    {
        param($VirtualMachines)
        ForEach -Parallel($VM in $VirtualMachines)
        {
            InlineScript
            {
                Invoke-Command -ComputerName $Using:VM[1] -ScriptBlock {
                    param($VMName)
                    Stop-VM -Name $VMName | Out-Null
                } -ArgumentList $Using:VM[0]
            }
        }
    }
    
    Workflow Start-RunningVirtualMachines
    {
        param($VirtualMachines)
        ForEach -Parallel($VM in $VirtualMachines)
        {
            InlineScript
            {
                Invoke-Command -ComputerName $Using:VM[1] -ScriptBlock {
                    param($VMName)
                    Start-VM -Name $VMName | Out-Null
                } -ArgumentList $Using:VM[0]
            }
        }
    }
    
    WorkFlow Optimize-VHDs
    {
        param($VirtualMachines)
        ForEach -Parallel($VM in $VirtualMachines)
        {
            InlineScript
            {
                Invoke-Command -ComputerName $Using:VM[1] -ScriptBlock {
                    param($VMname)
                    ForEach($VHD in ((Get-VMHardDiskDrive -VMName $VMname).Path)){
                        Mount-VHD -Path $VHD -NoDriveLetter -ReadOnly
                        Optimize-VHD -Path $VHD -Mode Full
                        Dismount-VHD -Path $VHD
                    }
                } -ArgumentList $Using:VM[0]
            }
        }
    }
    
    # Getting All Virtual Machines
    $AllVirtualMachines = New-Object System.Collections.ArrayList
    Get-ClusterResource -Cluster HVC01 | Where-Object {$_.ResourceType -eq "Virtual Machine"} | ForEach-Object { $AllVirtualMachines.Add(@($_.OwnerGroup.Name,$_.OwnerNode.Name,$_.State)) | Out-Null }
    
    # Selecting Running Virtual Machines
    $RunningVirtualMachines = New-Object System.Collections.ArrayList
    $AllVirtualMachines | Where-Object { $_[2] -eq "Online" } | ForEach-Object { $RunningVirtualMachines.Add(@($_[0],$_[1])) | Out-Null }
    
    Write-Output "Stopping Running VMs"
    Stop-RunningVirtualMachines $RunningVirtualMachines
    
    if ($Optimize)
    {
        Write-Output "Optimizing VHDs of all Virtual Machines"
        Optimize-VHDs $AllVirtualMachines
        Write-Output "Finished with Optimizations"
    }
    
    Write-Output "Stopping File Share Witness"
    $FSW = Get-ClusterResource -Cluster HVC01 -Name "File Share Witness"
    $FSW | Stop-ClusterResource | Out-Null
    
    Write-Output "`nRebooting FS01`n"
    Restart-Computer -ComputerName FS01 -Force -Wait
    
    Write-Output "FS01 Reboot Complete. Waiting $FSWWait seconds to bring File Share Witness Online"
    Start-Sleep -Seconds $FSWWait
    
    Write-Output "Bringing File Share Witness Online"
    $FSW | Start-ClusterResource | Out-Null
    
    Write-Output "Waiting an additional $VMStartWait seconds to start previously running Virtual Machines"
    Start-Sleep -Seconds $VMStartWait
    
    Write-Output "Starting Previously Running VMs"
    Start-RunningVirtualMachines $RunningVirtualMachines
    
    Write-Output "`nDone"
}

Now this script is a bit more complicated because it’s using workflows to make the starting, stopping, and optimization tasks parallel (waiting for these one at a time sucks if you have more than a couple of VMs…)

So the workflows should be pretty self explanatory:
Stop-RunningVirtualMachines: Takes an array of Virtual Machines ( “VM Name”, “VM Host” ) and issues the Stop-VM command on that VM’s current host
Start-RunningVirtualMachines: Takes an array of Virtual Machines ( “VM Name”, “VM Host” ) and issues the Start-VM command on that VM’s current host
Optimize-VHDs: Takes an array of Virtual Machines ( “VM Name”, “VM Host” ) and then mounts each of the HDDs for that VM on that VM’s current host and then runs a VHDX optimization task.

If you’re asking why not just issue the commands on the cluster, well I mostly did it to spread the load. When you issue commands to a cluster (HVC01) it all goes to the cluster master.

Now to the code:

First we get all of the Virtual Machines in the cluster, then we get a list of Running Virtual Machines. Then we pass the Running Virtual Machines list into the Stop-RunningVirtualMachines process.

If the -Optimize switch has been used we’ll then optimize all the Virtual Machine hard drives. This can take a while depending on the sizes of the VHDX files and how many there are, but it will get done in parallel, so expect a lot of disk IO

Next we’ll stop the File Share Witness (I use the File Server as a File Share Witness for the cluster quorum).

Now that all VMs are off, the VHDX files have (or haven’t) been optimized and the File Share Witness is offline we simply reboot FS01 and wait for it to come back.

Once the File Server has rebooted we start the File Share Witness, pause, then we start all of the previously running VMs with Start-RunningVirtualMachines

At this point the script would return to the Update-FS01 script to open the update log.