The environment that I support has close to 1,000 virtual guests with many of them running out of date VMware Tools. The out of date status was more of a cosmetic issue for a while until a few guests had unexpected crashes, which were later diagnosed by VMware support as to be related to the old version of tools. The tools crashes weren’t happening often, but they were occurring. Updated tools were in need.

I tried a few methods to update all of our Windows-based guests during our scheduled once a month maintenance weekend. The first plan was to update the tools the Friday night before out Saturday morning Windows updates/reboot. I wrote a simple PowerCLI script to initiate the tools upgrade from vCenter. I threw in the no reboot switch to delay the reboot until morning. Just over half of the VMs updated. The remaining that failed, were in an “updating tools” state on vCenter and it couldn’t be cancelled. I was able to kill the process by logging into the hosts and killing it from there. I attempted to narrow down the cause as to why many didn’t upgrade and vCenter’s initiation failed. I tried a few “solutions” that I came up with from other users but none of them worked. I was also unable to find any correlation as to why some updated and some didn’t. There were guests that were identical to other guests (load balanced web servers) that failed when some succeeded. Our team thought up of a few more ways we could push updated tools but all required too much work and planning.

One of us came across the “Check and upgrade Tools during power cycling” checkbox within the guest’s properties. This seemed optimal but we weren’t sure if the Windows updates would interfere with the Tools update when the server began the power cycle. We decided to go with it as there are other random times that our servers are rebooted for various reasons. In the end, they would take care of themselves. Perfect! Now to tick that checkbox on close to 1,000 guests. We could call in the interns…. They’d probably miss a few. There had to be a way to do it in PowerCLI! My co-worker came up with the script initially and we worked through some kinks to finally make it work. We kicked it off in our DR facility to say we performed “testing” before we started changing things in our production datacenter. No issues. In the end, the script took less than 20 minutes to tick the checkbox for all Windows guests.

The work done by the script is shown below:

 #Get Windows Guest machines in datacenter from vCenter server
$vmguests = Get-Datacenter $datacenter | Get-VM | %{Get-View $_.ID} | where {$_.Guest.GuestFamily -match "windowsGuest"}
 
#Create new Virtual Machince Config Spec
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
 
#Write new config options to VMs
foreach ($vm in $vmguests){
 
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.tools = New-Object VMware.Vim.ToolsConfigInfo
$vmConfigSpec.tools.toolsUpgradePolicy = "upgradeAtPowerCycle"
 
$vm.ReconfigVM_Task($vmConfigSpec)
}

After running the script in DR and against our production guests, it completed in around 20 minutes total. The results were great after the first maintenance weekend. Many environments were exempt from our last maintenance weekend but just over half of our Windows guests now have updated tools because of this script.

The script can be downloaded from here.

————————————-

Caution! This background process has been altered in vSphere5 and is described here.