Get-VM

Bits and Bytes of Virtualization

July 13, 2016
by zach
0 comments

vRA Could Not Create a SSL/TLS Secure Channel

Problem!

At the end of Monday, I noticed our vRA implementation was not provisioning new servers. A failure of a new machine request was reported two minutes after the submission/approval. I looked through the logs and found an error stating that the request could not create SSL/TLS secure channel. Therefore, I performed my proper engineer duties and hit the interwebs for a solution.

vRA Couldn't Create SSL/TLS Secure Channel

vRA Couldn’t Create SSL/TLS Secure Channel

Solution? Not So Fast.

Great! I found a VMware KB article (2123455) that describes my error in verbatim. Scrolling down to the resolution, I find it is a communication issue between the DEM-Worker servers and vRO. VMware references a specific Microsoft patch (3061518)that would have been installed on the DEM-W servers that needs to be removed. Therefore, I logged onto our DEM-W servers and found the patch was indeed installed on the servers. Unfortunately, I noticed it had been installed on the servers since August 9, 2015, which happened to be the day the servers were stood up initially. I was then not sold on the idea that they had worked for 11 months and then all of the sudden quit working because of this patch.

Fixed!

I opened a case with VMware to look into it. A vRA support log bundle was generated and sent off for review. The support engineer asked me to remove the patch even though it had been working properly for 11 months.  I found I could not directly remove it as it wasn’t shown in the list of Windows updates that could be uninstalled. So I wait for another solution….

The next day I was provided an update showing there was a roll-up update from Microsoft that may be the culprit. This time it was KB3161606. Sure enough, this patch was installed on both DEM-W servers over the weekend. I uninstalled it and rebooted both servers. Success! IaaS server provisioning is now completing without issues. The patch was pushed out in June by Microsoft. Hopefully, VMware gets around to updating their KB article to include KB3161606 alongside KB3061518.

April 15, 2016
by zach
0 comments

Custom Property is Not Displayed Correctly in vRA 6.2

I just ran into a quirk where a custom property is not displayed correctly in vRA 6.2. I had created a new custom property with a DropDown control type. Added the property to a build profile and attached the build profile to a blueprint. However, when I opened the catalog item to view it, there was no dropdown. I’ve included some screenshots to show an example.

As an example, I created a custom property in my property dictionary named “Test.Dropdown”.Custom Property

I added a few property attributes to be displayed in the dropdown.Custom Property Attributes

Added the custom property to a build profile.
Build Profile
Added the build profile to the desired blueprint.
Blueprint
Then I went to the catalog item related to the blueprint I added the dropdown to and WTF?
IncorrectDisplay

 Originally when I was trying to figure out why a text field was displayed instead of a dropdown, I double checked everything. Copied and pasted the name of the custom property to ensure I didn’t type something in wrong or had a trailing space… I then remembered I had a property layout attached to this blueprint. Once I added the property to the layout, the dropdown was displayed as expected.

I guess this is what you call a “feature.” It pulled the name of the custom property from the build profile, displayed the property definition name but went no further.

April 5, 2016
by zach
0 comments

Busy, but definitely not forgotten

It may seem I forgot this blog was here, but it is always in the back of my mind. Along with non-stop day-to-day requirements and projects at work, the last year of life outside of work has also been busy. Last summer, my wife and I went to Europe for just over two weeks. We did “the beer tour”, Belgium-Germany-Ireland. Definitely a trip of a lifetime and would go back in a heartbeat. However, my wife and I’s master plan has went perfectly as she is due with our first baby in mid-September. With that, house projects on our 100 year old house have been accelerated. The critical things are mostly done. Year after year, new experiences. None bigger than 2016!

Busy with…. Analytics?

Also last fall, I was fortunate to begin working with Dave Bartoo over at CFBMatrix.com to provide statistical analysis for upcoming college football games. I have mentioned this is “my profession meets my obsession.” Throughout the season, it was on and off but then was provided a truckload of data for the entire year for Clemson and Alabama for the National Championship. With the help oAlways busy with new experiencesf Python and Splunk, I was able to come up with some telling deficiencies/strengths on both sides of the ball. Dave attempted to provide the stats to his contacts on both sides two days before the big game. Both sides denied that stats unfortunately. All was not lost though. Our stats were gladly accepted by the TV commentators on the national broadcast. Four of my stats were mentioned on air. Another one of my stats that was not used but the opposite was said by Kirk Herbstreit on-air, proved to be true in favor of Alabama. Normally, my wife and I would have watched the game as we are huge football fans in general but really could have cared less considering the two teams. But since we knew there was a possibility that the stats could be used on-air it was one of the most interesting games to watch considering the statistical analysis that I had performed for the game. Who knows how the upcoming 2016 season will turn out and what stats I can come up with.

I know 2016 will be full of new experiences in and out of work which will no doubt keep me busy. Now to document some of those experiences on here so it is not forgotten.

 

 

 

November 30, 2015
by zach
0 comments

Commitmas is Almost Here

Last year Matt Brender (@mjbrender) started a little movement called Commitmas. As we approach the end of the year, Commitmas is almost here! At the heart, it is all about learning and sharing with the community. In the past, GitHub was an application developer’s playground. As infrastructure is becoming more and more managed by code, revision control is a must. GitHub or some other revision control system should be at the top of all IT Pro’s list of skills to learn.holiday-octocat

Commitmas was only twelve days last year but this year, a couple of the vBrownBag crew (Jonathan Frappier & Rob Nelson) has expanded it tremendously. This year, it is the entire month of December. Community engagement has expanded with the addition of an entire series of vBrownBags (sign up here), a twitter account (@commitmas), and a new Commitmas repository for 2015.

I didn’t join in on Commitmas last year as I didn’t see it until it was almost over. As I started to learn Python in late 2014 into early 2015, I used GitHub to keep track of my progress as well as learn GitHub at the same time. Unfortunately, I haven’t used GitHub much since then except for sharing a few PowerCLI scripts and vRO workflows. I’m not sure what I will be committing this Commitmas but I plan to make it through the entire 30 days!

Get signed up on GitHub, join the vBrownBag events, and be social while you learn a new skill! I urge you to join the challenge with the community!

 

August 20, 2015
by zach
0 comments

Unable to Expire or Power On vRA Managed Machine

Eric and I are deploying a distributed installation of vRealize Automation 6.2.2 with the help of a VMware architect on-site. We have progressed nicely through less than two weeks with the exception of some load balancing issues. Today we were deploying a VM into our environment and testing out different functions within the vRA interface. After a bit of testing, we were unable to expire or power on a vRA managed machine. Here’s where we ran into an issue.

A VM had been deployed by vRA and was online. I set the VM to expire. We checked the Requests tab to see if the request had successfully processed. It said it did but the VM never powered down. Also when viewing the VM within the list of Items in vRA, the status still reflected “On”.

Time to troubleshoot! we checked the Log under Infrastructure > Monitoring > Log. The following error was shown:

vRA-Expire-Failed-Error

Workflow ‘FireVirtualMachineEventRequest’ failed with the following exception: The HTTP request is unauthorized with client authentication scheme ‘Anonymous’. The authentication header received from the server was ‘NTLM,Negotiate’. Inner Exception: The remote server returned an error: (401) Unauthorized.

After a bit of digging, the “Negotiate,NTLM” bit in the error was the key. We checked the Web server’s IIS Windows Authentication Providers. Negotiate was listed above NTLM, which was the incorrect order as shown.

vRA-Expire-Failed-NTLM-Wrong

After moving NTLM to top provider as shown,

vRA-Expire-Failed-NTLM-Right

make sure you restart IIS with “iisreset” in the command line. We then tested expiring the VM. It was successful!

But….

Later when I attempted to power on the VM, I received the same error and the VM was never powered on. The status wasn’t expired, it was just powered off.

I then logged into my DEM Orchestrator servers and checked the same setting with the providers in IIS. Sure enough, Negotiate was listed above NTLM. I moved NTLM to the top and restarted the DEM-Orchestrator services.

Success! The VM powered on successfully!

The NTLM provider should have been ahead of Negotiate as we ran Brian Graf’s vRA 6.2 Pre-requisite Script, but for some reason they weren’t configured correctly.

March 6, 2015
by eric
2 Comments

Asynchronously remove datastores via vCO! (Updated)

Anyone with more than 3 hosts absolutely dreads removing data volumes from the VMware environment.  It is a mind-blowing tedious and redundant process that VMware has yet to fully address.  First you must unmount the volume(s) from all the hosts.  This part, thankfully, is easy, it just requires you to select the proper datastore, right click, and select ‘Unmount’.  A nice little wizard comes up and runs the appropriate checks to make sure the datastore can indeed be unmounted.  Just hit next and select the hosts you wish to unmount from and VMware kicks off the unmount procedure for that datastore on the selected hosts.

Well if you thought you were done and ready to unpresent that datastore, you are mistaken.  vSphere still sees that LUN and if you simply unpresent it from the hosts, they will really not like you one bit until you reboot them.  You must go to each host’s configuration page for storage adapters, find the correct LUN, right click and detach.  Here is one of VMware’s KB articles for those that need more information on the process.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004605

Imagine the time it takes to go through 10 hosts, or how about 50 hosts without automation?

So…let’s fix that and automate the entire process via vCenter Orchestrator!  Here is a quick run-down of what the workflow does.  First thing you need to do when running the workflow is select the cluster the datastore is presented to.

Unmount-Article001

After selecting the proper cluster and hitting next, you are presented with a dialog to select your datastore or datastores you wish to unmounts and detach from the hosts in the selected cluster.

Unmount-Article002

After selecting the datastores, just hit “submit” and away it goes.  So what does it do?  Here is what the schema looks like for the workflow.

 Unmount-Article003

The workflow starts off by getting all the hosts of the cluster you select.  It then grabs the needed information from the datastore(s) and stores it in a couple of arrays to be used later.  Take a quick look at the actual scripting behind this.

Unmount-Article004

It grabs the UUIDs needed for the unmount procedure and the Canonical NAA name for the detach sequence.  Who knows why VMware doesn’t allow these procedures to be done by just using just one of these variables, or at the very least fully documents the process, but this works…for now.

*Note:  you might need to adjust the SLICE number in your environment to grab the correct UUID.  14 is what works in my environment.

So after the workflow has the necessary info, it can proceed to the unmount loop.  We set the host to work with within the host array to the counter, then we kick off the unmount procedure that loops through each datastore in the datastore array that you selected and unmounts it on that host.  Here is the scripting code for that workflow.

Unmount-Article005

After it has looped through all the hosts, kicked off the unmounts and they finish, the workflow exits the unmount loop, resets the counter, and then drops into the detach loop.  The detach loop has the same setup as the unmount loop, except it launches the detach workflow for each host instead of the unmount workflow.  Take a look at its scripting code.

Unmount-Article006

Once the detach loop is complete and all detach operations have finished, the workflow exits the detach loop, kicks off a rescan for datastores on the hosts in the cluster to clean up the LUN paths, and then exits.

That is pretty much it, all this is done asynchronously on the hosts to save even more time.  Let me know what you think or if you have any questions.  Have fun tailoring this workflow for your needs!

You can find this workflow package on either Github or Flowgrab.

Github:  https://github.com/get-vm/vCO_Packages

Flowgrab: https://flowgrab.com/project/view.xhtml?id=0752edc8-a0fd-45fe-a675-4842732c4bad

April 21, 2015 Update:  Updated workflow to 2.1.0 based on Jason’s feedback.  There is now a sleep timer of 15 seconds and an initial counter reset before the unmount.  The updated workflows were pushed to the links above.

March 1, 2015
by eric
0 comments

Howdy all!

Thanks for the intro Zach.  I am both nervous and excited to start blogging.  I feel that it is time for me to make my appearance on the world-wide web in a more productive manner.  I have quite a few lofty goals this year, both personally and professionally, that could provide good writing opportunities as well as some comedic gold I am sure.  I tend to be light-hearted, but also don’t beat around the bush.  I am not afraid to call people, products, or companies out when they do questionable or flat-out dumb things.  So with that all said, let’s do this.  Head to the about page to read about me professionally and I will soon have a new post up that I hope you like.

February 27, 2015
by zach
0 comments

.Net 3.5 Feature Install Fails on Windows 2012

Recently, I ran into an issue where the .Net 3.5 Feature install fails on Windows 2012. Many search engine searches, blog posts, and message board posts later, I found a solution.

As you may know Windows Server 2012 comes with the .Net Framework 4.5 feature preinstalled. It does not have the .Net Framework 3.5 feature installed. Normally, it is an easy process to add the feature – Server Manager->Add Feature->Check the .Net Framework 3.5 feature->Install. But what if the server you are attempting to install .Net 3.5 onto is not allowed to connect to the Internet? If it is a VM, quickly attach a Windows Server 2012 .iso and specify an alternate source path and point it to “[DVD Drive Letter]:\sources\sxs” and it installs, right?

Well not every time. Most of the servers I have come across, usually fresh builds, attaching the ISO and specifying it as a the alternate source path does the trick. But I have found a couple servers that will fail indicating that the correct files are not in the attached .ISO, even though they are. The specific error I received was 0x800F081F. .Net 3.5 Feature install fails on Windows 2012

 

 

 

 

 

 

 

 

 

 

I finally found the correct KB article that outlined the correct issue with a resolution. I found numerous other reasons why .Net 3.5 wouldn’t install but this was the cause. I also found that if any language packs were installed prior to trying to install the feature, it would also fail. Any language pack installed, needs to be uninstalled and then the feature enabled, then the language pack(s) can be reinstalled.

The KB article points out that if either KB2966827 or KB2966828 were installed on the system, the .Net Framework 3.5 feature installation would fail, regardless of where the source of the files were. I downloaded the fix and installed it on the server (no reboot required!), and the feature was enabled without issue.

February 27, 2015
by zach
0 comments

vCO 5.5 Appliance Access Permissions

I hadn’t worked too much with the “Copy file from vCO to guest” workflow until the past six months. I quickly ran into issues with the default vCO 5.5 appliance access permissions settings. When I first tried to use it, I created a folder named “vcofiles” in the /opt/ directory on the vCO appliance based on a guide I was following. The had copied the file I wanted to transfer to multiple guests up to the /opt/vcofiles/ directory on the vCO server and gave root 777 rights to the vcofiles directory and the individual files. I kicked off the workflow and received the following error:

vCO No Permissions!

So I went back and checked to ensure I gave it full 777 access. I had. I then researched a bit more and found that the js-io-right.conf file needs to be edited to allow vCO rights to the new directory I created. Nick Colyer had a good post on what needed to be done over here and there was also a VMware KB article about it. If you check out the KB, you will notice that this applies for version 4.2.x and 5.1.x, but not 5.5.x. Of course, I was using the 5.5 appliance. The meat and potatoes of both articles still hold true. The only difference I have found is the new location of the js-io-rights.conf file in the 5.5.x appliance.

5.1.x and older location: /opt/vmo/app-server/server/conf/
5.5.x+ location: /etc/vco/app-server/

I added read, write, and execute permissions (+rwx) permissions to my new directory. After I finished, here’s what my settings looked like:

vCO 5.5 Appliance Access Permissions

 

 

 

 

As you can guess, this is done to ensure that the application that the users are accessing from the vCO client can only access directories that are specifically defined by the vCO admin.

February 24, 2015
by zach
0 comments

Welcome Eric!

A past co-worker of mine, Eric TeKrony, wanted to not only jump into vCO more after I left but he also wanted to contribute back to the community. So far we have combined forces on the Get-VM GitHub organization and have uploaded a few of our vCO workflows and actions. Along with uploading resources to GitHub, he may be on here from time to time releasing resources or just documenting an issue he found and resolved.

Along with vCO, he has extensive knowledge in many other realms of IT. I’m excited to see what he can bring to the community through this blog and other avenues.

So welcome Eric!