Eric and I are deploying a distributed installation of vRealize Automation 6.2.2 with the help of a VMware architect on-site. We have progressed nicely through less than two weeks with the exception of some load balancing issues. Today we were deploying a VM into our environment and testing out different functions within the vRA interface. After a bit of testing, we were unable to expire or power on a vRA managed machine. Here’s where we ran into an issue.

A VM had been deployed by vRA and was online. I set the VM to expire. We checked the Requests tab to see if the request had successfully processed. It said it did but the VM never powered down. Also when viewing the VM within the list of Items in vRA, the status still reflected “On”.

Time to troubleshoot! we checked the Log under Infrastructure > Monitoring > Log. The following error was shown:

Workflow ‘FireVirtualMachineEventRequest’ failed with the following exception: The HTTP request is unauthorized with client authentication scheme ‘Anonymous’. The authentication header received from the server was ‘NTLM,Negotiate’. Inner Exception: The remote server returned an error: (401) Unauthorized.

After a bit of digging, the “Negotiate,NTLM” bit in the error was the key. We checked the Web server’s IIS Windows Authentication Providers. Negotiate was listed above NTLM, which was the incorrect order as shown.

After moving NTLM to top provider as shown,

make sure you restart IIS with “iisreset” in the command line. We then tested expiring the VM. It was successful!

But….

Later when I attempted to power on the VM, I received the same error and the VM was never powered on. The status wasn’t expired, it was just powered off.

I then logged into my DEM Orchestrator servers and checked the same setting with the providers in IIS. Sure enough, Negotiate was listed above NTLM. I moved NTLM to the top and restarted the DEM-Orchestrator services.

Success! The VM powered on successfully!

The NTLM provider should have been ahead of Negotiate as we ran Brian Graf’s vRA 6.2 Pre-requisite Script, but for some reason they weren’t configured correctly.