VMware All Flash VSAN Configuration and Setup

tier-whatIt seems each year after VMworld I come home excited about some new piece of hardware or software that was announced at the show, or something at a vendor’s booth tht caught my eye. Well as in years passed, this year was no different. After sitting through and seeing a few sessions around an all flash (AF) VMware VSAN configuration I thought I would give it a shot in the home lab.

For the last six or eight months or so I have been running a “hybrid” VSAN configuration in my lab once I was able to get my hands on the bits. While the setup and configuration was easy, the performance (from my configuration) was less then stellar. For the hybrid configuration the flash tier (either SSD or PCIe based devices) are used to provide a read cache/write buffering tier in front of traditional magnetic disks. In my lab the HDD’s are my limiting factor for performance. Using a pair of Western Digital REDs in each host was an economical way (that is cheap) to try out VSAN and get it running, the 5400 RPM drives where just not up to the task for anything “performance” related.

Enter the AF VSAN configuration. After getting home I jumped over to NewEgg.com to look for some good cost per GB SSD drives for the lab. I knew from conversations with Jim Millard ( blog / twitter ) that he has good experience with the 512GB drives from Transcender, and the price was right. I ordered up six drives and drive carriers for my ESXi hosts. After waiting a few days the new toys arrived and I set out on getting my AF VSAN up and running.

Setting up an AF VSAN works pretty much the same way as the hybrid model, except for one catch. We have to flag the flash devices (in my case SSD’s) to not only be identified for use as the read caching/write buffering tier, but also to be used for data disks or for the capacity tier. There are two ways to accomplish this task, the hard way (read that as manual way) or the easy way (read that as automated way).

The Hard Way

Just like days back at school, the following is the “show work” process of configuring the local host SSD drives to be used as the capacity tier for my AF VSAN. As VSAN can only be configured via the vSphere Web Client, the screen shot below presents my starting point. I have three ESXi hosts with two SSD drives ready for use:


By clicking the “Create Diskgroup” with host esx01.lab.local selected we see that the two SSD devices (one 512GB SSD and a 128GB SSD) are present, but can be only configured for the write cache/read buffer setting. Nothing is listed in the “Data Disks” section:


This leads us to the manual steps to flag the 512GB SSD as a “capacity flash” to allow us to select the drive for a data disk. After enabling SSH on host esx01.lab.local is issued the vdq –q command to provide a listing of drives eligible for use by VSAN. In the screen shot below, the two drives highlighted in red are my eligible SSD drives. Make note of two settings:

  • IsSSD
  • IsCapacityFlash

The IsCapacityFlash is the setting we are concerned with and need to change.


Using the device “Name” provided via the vdq –q command above, we will use an esxcli command to flag the device as “capacityFlash:


After adding the capacityFlash tag to the drive, I re-ran the vdq –q command to verify the drive settings. Highlighted in red below, we see that the setting for IsCapcityFlash is marked as 1:


Switching back over to the vSphere Web Client I again clicked on the “Create Disk Group” for esx01.lab.local. I now see the 128GB SSD as the option for the write cache/read buffer, but now can also select the 512GB SSD to be used as a “Data Disk”:


While the process is pretty easy and straightforward (and with only three hosts in my lab, a matter of minutes to configure), in a large scale AF VSAN deployment making the needed changes across multiple drives and multiple hosts would be come quite tedious. This brings us to the easy way!

The Easy Way

So, working through the manual process is straightforward and it is always good to know how something works behind the scenes. But again, in a decent size deployment the manual process could become quite laborsome. Enter the VMware Virtual SAN All-Flash Configuration UtilityThis utility was co-developed by Brian Graf ( blog / twitter ) and Rawlinson Rivera ( blog / twitter ) and as you can guess, used to automate the above procedure.

After downloading the tool and configuring the need pre-requisites (see the link above to Brian’s blog for details) I launched the configuration utility. After filling out the top line information (vCenter, User, Password, ESXi user, ESXi password) I clicked on “Connect”:


With a successful connection made, the utility identified the two ESXi clusters and hosts in my lab environment. More specifically the hosts that will be configured for VSAN are under my “Compute Cluster” heading and highlighted in the screen shot below. The utility also provides the current status of the SSH service. This is important as the tool leverages SSH to execute the needed esxcli commands to flag the drives (as seen in the manual process). With each of the three hosts selected I clicked the “Enable SSH” button:


After waiting a couple of seconds the SSH service is up and running. Again with the three hosts selected, click the “Select” button to begin flagging the needed drives:


With a poll of each of the three hosts each of the host devices are listed. We are looking for the 512GB SSD for each host and checking the box to select them. Note under the “State” column you see the “Eligible for use by VSAN” status. With each drive selected, click the “Set Capacity Flash” button in the bottom right hand corner:


Once the process completes the selected drives will be marked as “IsCapacityFlash” as shown in the screenshot below:


With the drives properly flagged, again we will jump over to the vSphere Web Client and select
“Create Disk Group”. As expected we can no see the two SSD drives listed, the 128GB SSD for the read cache/write buffer and the 512GB drive listed as a “Data Disk”:


Final Product

Either using the manual process or the configuration utility, below is the results of the work. You can see that each of the three hosts consists of a single Disk Group and two disks are in use per group. With esx01.lab.local selected, in the bottom portion of the screen you can see the two disk devices listed:


Switching to the “General” section under “Virtual SAN” in the vSphere Web Client we can see the configuration/sizing of the VSAN Datastore and disk format version:


Thanks for reading and any questions or comments be sure to post them below.