Bits and Bytes of Virtualization

April 19, 2018
by zach

vRA 7.4 Upgrade Issue

VMware released the latest revision of vRealize Automation last week. I found some time to perform an upgrade to my homelab environment. At the time, 7.3.0 was the running version. vRAI planned to skip past 7.3.1 and go directly to 7.4. I downloaded the vRA 7.4 ISO file, attached it to the appliance’s CD-ROM drive and clicked check updates from the CD-ROM. Unfortunately, the error “No update found on 1 CD drive(s)” was given. I soon decided to skip that and let the appliance upgrade to 7.3.1 first. That upgrade went smoothly without any issues.

The Issue

Next up was the vRA 7.4 upgrade. I took another round of snapshots and went back into the appliance management and initiated the 7.4 install. The vRA appliance upgraded to 7.4 and asked for a reboot. The appliance rebooted and came back online. After waiting a very long time for the IaaS components to begin their upgrade I noticed an issue with some appliance services. The vCO service did not have any status while the following services were “UNAVAILABLE“:


Services Unavailable

I dug into some logs and found WARN events surrounding the unavailable services. In those events, I noticed the following error: “Unable to establish a connection to vCenter Orchestrator server.” Therefore, I needed to figure out why the vCO service was not starting. Once I could get it to start, the others would register successfully. I checked the logs for the vCO services and found the following error:

 2018-04-14 18:39:16.702+0000 [serverHealthMonitorScheduler-1] WARN {} [LdapCenterImpl] Unable to fetch element "vsphere.local\vcoadmins" from ldap : Error...:[404 ][javax.naming.NamingException]
2018-04-14 18:39:16.702+0000 [serverHealthMonitorScheduler-1] ERROR {} [AuthenticationHealth] Unable to find the server administrative group: vsphere.local\vcoadmins in the authentication provider.

The Resolution

This is an immediate smoking gun for my configuration. I set up the vRO admin group to use a group within my Active Directory. Therefore, the local group, vcoadmins, was not present and prevented the vCO service from registering with vRA. I changed the vRO admin group to my AD group and rebooted the appliance.

vRO Admin Group

All of the services registered successfully and the IaaS upgrade process began. The vRA 7.4 upgrade completed shortly after that without any further issues.

Upgrade Complete

However, I don’t know why the vRO admin group was changed to vsphere.local/vcoadmins during the 7.3.1->7.4 upgrade. Luckily it wasn’t too big of an issue to fix but annoying to say the least.

April 11, 2018
by zach

Import Python Modules for use in an Azure Function

Azure Functions is a “serverless” compute service that supports multiple programming languages. Some languages are officially supported, while others are in preview. Azure FunctionsI have numerous python scripts that I could push into the cloud to help me learn how to use Azure Functions. Unfortunately, the previewed languages do not have very much documentation out there.  The biggest hurdle was importing python modules for use in an Azure Function.

Azure Functions uses the App Service on the back-end which allows you to customize your application environment with the help of Kudu. I found some documentation across multiple sites that had aged a bit. Not a single how-to post or guide had all of the answers. The inaccuracy of the guides I found may be from the preview nature of the language support. This is not surprising as Python is in preview. After lots of trial and error, I found a method that worked for me.

Create a Function App

First, create a new Function App. 

Create a New Function App

Confirm the function app is up and running.  Then click the + sign next to functions to add a function to the app. 

Create a New Function

The center pane will ask for a scenario and language to assist with a premade function. Since we are using python for our language, a custom function must be selected to proceed.

Create Custom Function

The next screen provides templates to use to get started. However, to use python, the “Experimental Language Support” switch needs to be enabled.

Enable Experimental Languages

After selecting Python, only two options (HTTP trigger and Queue trigger) can be selected. For this demo, I will select HTTP trigger. I left the defaults for this example. 

HTTP Trigger

Update Python Version

Now that we have a function in the app, the python version needs to be updated. The python version that is installed is old and conflicted with my scripts. This may not be the case for your scripts but if you need to update to a specific version of python, this will assist in that process. My scripts were written for Python 2.7. I need to fix my scripts to support Python 3.6 but that will come at a later time. To get started, We need to access the Kudu tool. Click the Function App name on the left, then Platform features at the top, and then “Advanced tools (Kudu)” near the bottom of the center pane.

Azure App's Kudu

To update the Python version, click the Site extensions at the top.

Click the Gallery tab. Then type in Python in the search. The results will provide multiple versions of Python available to be installed. Pick your desired version. 

I need Python version 2.7.14 x64. Click the + sign to install the extension into your environment. The install icon will show an animated loading icon while it is installing. Once it is finished, a X icon will be present in the upper right of the tile. Take not of the path where this version of python is installed. It will be needed later.

Now that our desired version of Python has been installed, the Handler Mappings need to be updated. Go back to the Function App’s Platform Features page. Then select “Application settings.”

Application Settings

A new tab is shown in the center pane. Scroll to the bottom to the Handler Mappings section. A new mapping needs to be added. Click “Add new handler mapping” and enter the relevant settings for “fastCgi” handler mapping for the version of Python you installed. The path is shown on the tile when you installed the different version. My handler settings were as followed:

fastCgi ->D:\home\python27\python.exe -> D:\home\python27\ 

Python fastCgi Handler Mapping

Scroll to the top of the Application Settings page and click Save.

You can test the version of Python being used by replacing the code in the file with the following code:

import os
import json
import platform
postreqdata = json.loads(open(os.environ['req']).read())
response = open(os.environ['res'], 'w')
response.write("Python version: {0}".format(platform.python_version()))

When the above code is run, the output returns the Python version. My example returns the correct version from the site extension I installed.


Create Virtual Environment

Next, a virtual environment needs to be created. This is where the Python modules will be installed. Head back to the Kudu tool and click the “Debug console” dropdown and click CMD.

Kudu Powershell

At the top, you will see a directory structure that can be used for navigation. First, the virtual environment module needs to be installed as it does not seem to come with the updated version of Python that was installed previously with the site extension addition. Run the following command: “python -m pip install virtualenv”.

Install Virtual Env

Now that the virtualenv module is installed, it is time to create a new virtual environment. Navigate to the following directory: “D:\home\site\wwwroot\{yourFunctionName}”. Then in the console type the following: “python -m virtualenv yourenv” where ‘yourenv’ will be the name of the virtual environment that you create.

Create Virtual Env

Once the virtual environment has been created navigate to “yourenv\scripts” and run activate.bat. This will activate your virtual environment and place your active console in it. You can see if it is active as it the environment name precedes the path as shown below.

Enter Virtual Env

You now have access to run python commands that allow you to install modules and configure your Python environment to your needs. 

Install Python Modules

Installing modules through PIP is recommended. However, I ran into an issue where PIP would not install a couple modules I needed.  I recommend attempting to install using PIP first, as I did with ‘lxml’ below.

lxml Install

I have received an error while installing modules that indicates it needs the vcvarsall.bat file that is included within the Microsoft Visual C++ 9.0 package. If you do get this error, you can manually download the “wheels” that contain the module you need to install. The best site that I found that can direct you to the official wheel files is From there, you can find the module you need. Select the correct version of wheel that is specific to your environment (2.7, 3.6, x86, x64, etc.). You also need to install the wheel module before you import these wheel files (python -m pip install wheel). 

Now that wheel is installed and you have downloaded the correct .whl file for your module, you can simply drag and drop the .whl file from your desktop into the following folder: “D:\home\site\wwwroot\{yourFunctionName}\{yourenv}\Lib\site-packages.” It will unpack the .whl file automatically and make it available. 

Once you have installed all of your modules, run “pip freeze” to discover the modules that are installed. I installed bs4, lxml, and requests. They naturally installed a few other modules as dependencies.

List Installed Python Modules

Import Modules Within Your Script

I know this has been long, but you’re almost done! The last thing to do is let Python know where your modules reside so it can correctly import them into your scripts for use. At the top of you script(s), enter the following code:

import sys, os.path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname( __file__ ), 'yourenv/Lib/site-packages')))

Ensure you replace “yourenv” with whatever you chose to name your virtual environment. 

After that, your script will be able to import any Python module it needs and complete successfully.

June 1, 2017
by zach

Invalid Username or Password When Logging Into Embedded vRO

Granting user authentication from vRealize Automation (vRA) 7.2 to vRealize Orchestrator (vRO) is not as easy as it should be. I received an “Invalid Username or Password” error when logging into vRO, as shown below. vRO with an invalid username or password error.My vRA environment was configured to use my home lab Active Directory (AD) domain without any issue. Next I wanted to get my vRO appliance configured. I logged into the vRO Control Center to configure the authentication and other items. Since I am using the embedded vRO, the Authentication Provider is automatically set to vRealize Automation. The custom tenant was set and I was able to populate the AD groups from the dropdown without any issues. Since I could see my AD groups, I didn’t think vRO would have any issues with authenticating any user within my selected AD group. I was mistaken.

A quick search across blogs and forums did not provide much help. I went to the vExpert Slack channel and hit another roadblock. A couple members told me to follow a blog post from vCOTeam to correctly configure the domain login. I had already tried this without any success. The channel said that is how it works now and could not see much else from the logs I provided. With a bit more searching, I found a blog post on Spas Kaloferov’s blog that was my key to finding the solution to this problem, twice.

Solution 1

The solution that worked in my home lab was referenced under his misconfiguration of the Identity Provider in vRA. He mentions changing the IdP Hostname to the vRA Load Balancer address. Unfortunately, my vRA environment contains zero load balancers. I did notice that my IdP Hostname was not the vRA FQDN. It was set as the hostname with no domain suffix. After changing the IdP Hostname to the correct vRA FQDN, I was able to login with my AD user account.

IdP Hostname with the correct FQDN causes vRO to not authenticate.

Solution 2

While working with a client, I ran into this issue again. Immediately, I checked the IdP Hostname. This time, the IdP Hostname had the correct FQDN configured. Later, we accidentally discovered that the certificate that was generated by one of their team members had a misspelled FQDN for the vRA appliance and lacked another Subject Alternative Name (SAN). A new certificate was generated with all of the correct FQDNs and SANs required for our deployment. This proved to be the solution for their version of this issue.

In Conclusion

VMware needs to address this finicky configuration between vRA and vRO. There are too many variables that may cause this issue.

With the release of vRA/vRO 7.3, they changed the back-end authentication again and will probably eliminate this issue. However, they will create a new issue. They always do.


May 31, 2017
by zach
1 Comment

Errors Deploying Infoblox Appliances to vCenter 6.5

I want to learn more about integrating with Infoblox’s IPAM solution as it is the number one IPAM solution for most medium to large companies. Getting experience should be as simple as deploying their DDI appliance into my home lab. I discovered a couple issues while trying to deploy their OVA into my environment. First, I am running vCenter and vSphere 6.5. When I requested an evaluation of their product, I was provided a link to an older vNIOS version (7.3.9). The following error was presented when I tried to deploy it:

Infoblox OVA failed checksum verification

If you look closely at the error, you will see it is failing a checksum for the Xen Server .ovf file. I can confirm I was downloading the VMware version. The MD5 hash to matched up with Infoblox’s MD5 hash. I tried deploying through the vCenter Web Client from a local source first. I also tried allowing vCenter to reach out to Infoblox to download the package by the URL. After trying too many methods, I reached out for support and was told to download a newer version, specifically version 8.0.6. No reason was provided why this package failed.

Lets Try a Newer Infoblox Version!

With a newer version in hand, a new deployment began. This time, a new error presented itself.

Infoblox-Known Issue

Issues detected with selected template. Details: – 17:3:SECTION_RESTRICTION: Section Product Section (Information about the installed software) not allowed on envelope.

But there is a workaround..

Infoblox’s support confirms that this issue is a known issue and will be fixed in version 8.2.0. The workaround to this issue is to not deploy the appliance through vCenter but rather directly to a host. I can confirm this does work and I have Infoblox up and running in my environment. Even though this is a known issue, I did a quick search and was unable to find this error in any results. Infoblox’s Knowledge Base is also lacking this error. Therefore, I’m putting it out there just in case anyone else comes across this issue.

April 24, 2017
by zach

vRA 7 Server Deployment Fails After VM is Deployed From Template

Recently, I purchased enough equipment to complete a homelab environment. Everything went well until the last step of deploying a new VM through vRA 7.2. I asked a couple colleagues what they thought and they hadn’t seen it before. I searched VMTN and google didn’t find the exact cause of the issue so I decided to get this out there just in case someone else ran into it.


To set the stage, I have a small deployment of vRA 7.2 running in a nested environment. My first catalog item is a Windows Server 2012 R2 VM. The template is prepped and a customization specification ready to be applied. Using just vCenter, I could deploy a VM from the template and use the customization specification to customize the guest successfully. However, when I attempted this process through vRA, I received the following error right after the clone completed.

The following component requests failed: vSphere_Machine_1. Request failed: Machine “servername”: 
CustomizeVM: Error getting property ‘info’ from managed object CustomizationSpecManager.


I also received the following error in vCenter.

Set virtual machine custom value: A specified parameter was not correct: key

vcenter-cSpec-DeployErrorI tried a few different things to resolve the issue like creating a new customization spec but everything I did always pointed back to vRA trying to initiate the next step after the VM was deployed from the template.


As I searched blogs and VMTN for answers, I discovered the following thread. It isn’t the smoking gun but did get me pointed in the right direction. It describes a permissions issue causing the error Danny saw, which happens to be the same error I experienced. Next, I took a look at the permissions granted to my svc_vRA account. It had full admin privileges, at the data center level. Since this is my homelab, there’s no reason I can’t grant it more access. I granted it admin privileges at the vCenter level. This change allows it access to the customization specifications, which are above the data center access. I kicked off a new deployment and received a successful deployment of a base Windows Server 2012 R2 VM.

Make sure the account you are using within vRA has enough permissions. Then ensure they are granted in the correct location!

April 21, 2017
by zach

Moving My Career AHEAD

As many of you already know, I joined AHEAD a couple months ago. I started as a Senior Technical Architect on February 7th. This is my jump out of the customer space into consulting. Moving My Career AHEADI felt this was the best time in my career and in my personal life to make this move. I had become bored with the day-to-day activities within a customer environment. My last company had plenty of technology to work with but it was just advancements of the same old stuff I had been using for years. Therefore, a change was needed before I was completely burnt out of IT in general.

I had been in contact with AHEAD for some time but the time was not quite right. They reached out to me in January and the ball was in full motion to get me onboard. They wasted zero time getting me engaged with clients as I was on-site with a client in my first two weeks. In the two months that have passed since joining, I have been busy the entire time. Not only learning how the consulting side works, but also learning new methods and new technology has been on my agenda.

I am excited about my future career at AHEAD. Initially when searching for companies I was willing to work for, AHEAD stood out because of the talent on staff. Two months in, the talent at AHEAD has surprised me even more. The best part is everyone is willing to assist me in whatever way. It is definitely a team atmosphere. I’m definitely glad to be here! I’m ready for the challenges ahead!

I will also be paying more attention to this blog. I already have four posts in queue resulting from issues or experiences that I have come across in the past two months.


April 12, 2017
by zach

vRA 7.2 Active Directory Policy Failing to Create New Computer Object

I love the new Active Directory Policy feature within vRealize Automation (vRA) 7.2. It allows easy management of Active Directory (AD) objects, like computer objects when a new VM is provisioned. I like this integration much better than the CCC plugin that was created for vRA 6.x a couple years ago. The flexibility of Active Directory Policies within vRA is highly desirable for most admins. It can also be fairly dynamic when paired with its custom property.

The Issue

Without much work, the Active Directory Policy configuration is quick and simple. However, I encountered a problem when the workflow within vRealize Orchestrator (vRO) could not create a new computer object during an event subscription lifecycle state. The error isn’t very descriptive unfortunately. 

AD Object Creation Failure

With not much to go on, I decided to perform the same operation but with the regular AD workflows within the AD plugin in vRO’s library. I received the same error when using those workflows. Choosing a different OU to deploy to also resulted in an error.

The Solution

I changed the service account I used to a domain admin account and was met with a successful creation of an AD computer object. At that moment, I realized I used a service account that did not have proper rights to the OU I was trying to create/delete computer objects in. It is an easy fix but without much of an error, it can be frustrating to troubleshoot.

Other than this user error, the Active Directory Policy integration works very well and is a must have for environments with Active Directory.


July 13, 2016
by zach

vRA Could Not Create a SSL/TLS Secure Channel


At the end of Monday, I noticed our vRA implementation was not provisioning new servers. A failure of a new machine request was reported two minutes after the submission/approval. I looked through the logs and found an error stating that the request could not create SSL/TLS secure channel. Therefore, I performed my proper engineer duties and hit the interwebs for a solution.

vRA Couldn't Create SSL/TLS Secure Channel

vRA Couldn’t Create SSL/TLS Secure Channel

Solution? Not So Fast.

Great! I found a VMware KB article (2123455) that describes my error in verbatim. Scrolling down to the resolution, I find it is a communication issue between the DEM-Worker servers and vRO. VMware references a specific Microsoft patch (3061518)that would have been installed on the DEM-W servers that needs to be removed. Therefore, I logged onto our DEM-W servers and found the patch was indeed installed on the servers. Unfortunately, I noticed it had been installed on the servers since August 9, 2015, which happened to be the day the servers were stood up initially. I was then not sold on the idea that they had worked for 11 months and then all of the sudden quit working because of this patch.


I opened a case with VMware to look into it. A vRA support log bundle was generated and sent off for review. The support engineer asked me to remove the patch even though it had been working properly for 11 months.  I found I could not directly remove it as it wasn’t shown in the list of Windows updates that could be uninstalled. So I wait for another solution….

The next day I was provided an update showing there was a roll-up update from Microsoft that may be the culprit. This time it was KB3161606. Sure enough, this patch was installed on both DEM-W servers over the weekend. I uninstalled it and rebooted both servers. Success! IaaS server provisioning is now completing without issues. The patch was pushed out in June by Microsoft. Hopefully, VMware gets around to updating their KB article to include KB3161606 alongside KB3061518.

April 15, 2016
by zach

Custom Property is Not Displayed Correctly in vRA 6.2

I just ran into a quirk where a custom property is not displayed correctly in vRA 6.2. I had created a new custom property with a DropDown control type. Added the property to a build profile and attached the build profile to a blueprint. However, when I opened the catalog item to view it, there was no dropdown. I’ve included some screenshots to show an example.

As an example, I created a custom property in my property dictionary named “Test.Dropdown”.Custom Property

I added a few property attributes to be displayed in the dropdown.Custom Property Attributes

Added the custom property to a build profile.
Build Profile
Added the build profile to the desired blueprint.
Then I went to the catalog item related to the blueprint I added the dropdown to and WTF?

 Originally when I was trying to figure out why a text field was displayed instead of a dropdown, I double checked everything. Copied and pasted the name of the custom property to ensure I didn’t type something in wrong or had a trailing space… I then remembered I had a property layout attached to this blueprint. Once I added the property to the layout, the dropdown was displayed as expected.

I guess this is what you call a “feature.” It pulled the name of the custom property from the build profile, displayed the property definition name but went no further.

April 5, 2016
by zach

Busy, but definitely not forgotten

It may seem I forgot this blog was here, but it is always in the back of my mind. Along with non-stop day-to-day requirements and projects at work, the last year of life outside of work has also been busy. Last summer, my wife and I went to Europe for just over two weeks. We did “the beer tour”, Belgium-Germany-Ireland. Definitely a trip of a lifetime and would go back in a heartbeat. However, my wife and I’s master plan has went perfectly as she is due with our first baby in mid-September. With that, house projects on our 100 year old house have been accelerated. The critical things are mostly done. Year after year, new experiences. None bigger than 2016!

Busy with…. Analytics?

Also last fall, I was fortunate to begin working with Dave Bartoo over at to provide statistical analysis for upcoming college football games. I have mentioned this is “my profession meets my obsession.” Throughout the season, it was on and off but then was provided a truckload of data for the entire year for Clemson and Alabama for the National Championship. With the help oAlways busy with new experiencesf Python and Splunk, I was able to come up with some telling deficiencies/strengths on both sides of the ball. Dave attempted to provide the stats to his contacts on both sides two days before the big game. Both sides denied that stats unfortunately. All was not lost though. Our stats were gladly accepted by the TV commentators on the national broadcast. Four of my stats were mentioned on air. Another one of my stats that was not used but the opposite was said by Kirk HerbstreitĀ on-air, proved to be true in favor of Alabama. Normally, my wife and I would have watched the game as we are huge football fans in general but really couldĀ have cared less considering the two teams. But since we knew there was a possibility that the stats could be used on-air it was one of the most interesting games to watch considering the statistical analysis that I had performed for the game. Who knows how the upcoming 2016 season will turn out and what stats I can come up with.

I know 2016 will be full of new experiences in and out of work which will no doubt keep me busy. Now to document some of those experiences on here so it is not forgotten.