Deploying a 2-Node ROBO #vSAN Cluster

While Hyper-Converged Infrastructures such as Nutanix and VMware’s vSAN are popular topics in changing the dynamics of how compute and storage resources are consumed in a primary datacenter, one use case that sometimes gets overlooked is organizations that have or support a remote or branch office (referred to as ROBO going forward).

VMware addressed this customer need in the v6.1 release of vSAN supporting 2-Node + Witness configurations and has continued to introduce new enhancement/features since. Most recently the ability to “Direct Connect” the nodes in the ROBO location bypassing the need for a switch (at least for vSAN connectivity) to be deployed in v6.5.

While setting up vSAN via the vSphere Web Client is straightforward, there is a bit of “plumbing” that needs to be accomplished (both on the physical networking and ESXi networking side) to really get his use case up and running. Let’s see how it done!

Networking

Before we get started on want to provide a quick overview of my current network environment. I have a “Production Site” configured that will be used to run the required vSAN Witness Appliance (TUK-WTN02) and I have dedicated VLANS for the needed network traffic; Management, vSAN Data, and vSAN Witness. On the “Branch Side” I have already deployed my two ESXi hosts, RO-ESX01 and RO-ESX02, with a very limited configuration. The dedicated VLANS at this location are for Management and vSAN Data.

Providing proper communication/routing between the two sites is SonicWall firewall device (not listed in the diagram).

Deploying the vSAN Witness Appliance

The vSAN Witness appliance is provided by VMware and is an OVA deployment of specialized ESXi build for this exact purpose. You will need to log into your “My VMware” account to access and download the OVF template. Currently a single witness appliance is needed per ROBO vSAN deployment, so if you have three branch offices you would need to deploy three separate appliances. For this reason, I have created a Datacenter object in my vCenter inventory to place the appliance(s).

Now, I will have assumed you have logged into your vSphere Web Client with an Administrative account.

  • To Deploy the OFV Template, Right click on the vSphere Cluster and Select the option for Deploy OVF Template:

  • Click the Browse button to navigate to OVF down load location. Click the Next button to continue:

  • Provide a Name for the vSAN Witness Appliance and select the Datacenter/Folder location. Click the Next button to continue:

  • Select the Host or Cluster to house the virtual machine, for my lab I will be using my Management Cluster. Click the Next button to continue:

  • Review details for the deployment and Click the Next button to continue:

  • Read the License Agreement and Click the Accept and Next buttons to continue:

  • From the Dropdown menu select the Configuration size for your Witness Appliance. The Appliance comes in three sizing options:
    • Tiny – 10 VMs or Fewer
      • 2 vCPUS
      • 8GB vRAM
      • 1 x 12GB ESXi Boot Disk
      • 1 x 15GB Magnetic Disk (used for vSAN capacity tier)
      • 1 x 10GB Solid State Disk (used for vSAN cache tier)
      • Maximum of 750 Components
    • Medium – Up to 500 VMs
      • 2 vCPUs
      • 16GB vRAM
      • 1 x 12GB ESXi Boot Disk
      • 1 x 350GB Magnetic Disk (used for vSAN capacity tier)
      • 1 x 10GB Solid State Disk (used for vSAN cache tier)
      • Maximum of 22K Components
    • Large – More than 500 VMs
      • 2 vCPUs
      • 32GB vRAM
      • 1 x 12GB ESXi Boot Disk
      • 2 x 350GB Magnetic Disks (used for vSAN capacity tier)
      • 1 x 10GB Solid State Disk (used for vSAN cache tier)
      • Maximum of 45K Components

    For my lab deployment, I have select the Tiny configuration. Click the Next button to continue:

  • Select the Datastore that will be used to house the virtual machine. Click the Next button to continue:

  • Select the associated Port Groups for the Management and Witness Network traffic. Click the Next button to continue:

  • Provide a root account Password. Click the Next button to continue:

  • On the Ready to Complete screen, review the settings and if no changes/edits Click the Finish button to deploy the template:

With that, we have deployed the vSAN Witness Appliance virtual machine.

Configuring the vSAN Witness ESXi Host

With the witness appliance now deployed in your environment, power on the virtual machine and wait till it boots into ESXi. Configure the standard ESXi properties via the DCUI (I will spare you the screen shots here as this should be an easy one):

  • IPv4 Configuration
  • IPv6 Configuration (if applicable)
  • DNS Configuration
  • Custom DNS Suffixes

With the base ESXi settings completed, join the vSAN witness appliance to your vCenter Server instance. For a logical grouping, I created a new Datacenter object titled vSAN_Witness to hold this and additional witness appliances I will be deploying. Again, for the sake of saving screen grabs this process is no different than joining any other ESXi host to a vCenter. The only slight difference is on the Assign License screen. The vSAN Witness appliance comes with its own license and does not consume a “traditional” per socket ESXi license:

Once completed with the “Add Host” wizard you will see the appliance listed in your vCenter inventory. Note the blue color of the ESXi host icon:

  • Select the host, expanding the Networking node from the Configure tab and highlight VMkernel Adapters. Finally, highlight vmk1 and Click the Pencil icon to edits it properties. (Note – See that the IP address for the interface is not set and the Port Group name is “witnessPg”)

  • By default, the Virtual SAN service box will be Checked:

  • On the right-hand side Select the IPv4 Settings node. Provide the Static IPv4 Settings for the interface. Click the OK button when completed:

  • With that, the needed VMKernel interfaces have been configured and we are ready to move onto the next step, configuring the needed static routes to tie this all together.

Configuring Static Routes

Ok so far so good for the deployment, nothing too complicated. But setting the static routes is really the key part about successfully deploying the 2 Node ROBO configuration (not considering the true “plumbing” that must be done by your network team). For the configuration to work properly the vSAN Witness appliance needs to be able to communicate with the ROBO site nodes and vice versa. This is accomplished by configuring static routes on each of the three ESXi hosts. These routes are required as ESXi can only be configured with a single default gateway (usually assigned to your Management VMKernel interface) and if following vSAN networking best practices, it should be isolated onto its own isolate network.

So, taking my previous logical network design for my deployment the following communication is needed:

  • Production Site – Witness Appliance on VLAN31 needs to be able to contact RO-ESX01 and RO-ESX02 on VLAN230 at the Branch Site
  • Branch Site – RO-ESX01 and RO-ESX02 on VLAN230 needs to be able to contact the Witness Appliance at the Production Site

Let me visualize this a bit in a chart:

Host Name

VLAN

IP Address

Route

TUK-WTN02

31

10.1.31.32

10.1.31.1

RO-ESX01

230

10.2.30.51

10.2.30.1

RO-ESX02

230

10.2.30.52

10.2.30.1

Easily accomplished by using your SSH client of choice and the esxcli network ip route ipv4 command set. I am going to start on the Branch site with host RO-ESX01. The numbered steps below describe the corresponding items in the screenshot:

  1. Pinging the Witness Appliance vSAN enabled VMKernel interface. As expected it should fail
  2. Showing the current routing table on the host via esxcli network ip route ipv4 list. The only route defined is the default gateway for the Management VMKernel interface
  3. Using the esxcli network ip route ipv4 add command I am specifying the static route needed for traffic to reach the 10.1.31.0/24 network (-n switch) via the gateway of 10.2.230.1 (-g switch)
  4. Verifying that the route statement was added via the esxcli network ip route ipv4 list command

With the RO-ESX01 configured it is time to switch over to the Witness Appliance, TUK-WTN02. Again, leveraging SSH I have contacted to the system to specify the needed static route. The numbered steps below describe the corresponding items in the screenshot:

  1. Pinging RO-ESX01 vSAN enabled VMKernel interface. As expected it should fail
  2. Showing the current routing table on the host via esxcli network ip route ipv4 list. The only route defined is the default gateway for the Management VMKernel interface
  3. Using the esxcli network ip route ipv4 add command I am specifying the static route needed for traffic to reach the 10.2.30.0/24 network (-n switch) via the gateway of 10.1.31.1 (-g switch)
  4. Verifying that the route statement was added via the esxcli network ip route ipv4 list command
  5. Reissuing the ping command to RO-ESX01, with SUCCESS!

No let’s verify that RO-ESX01 can reach TUK-WTN02:

All is looking good at this point! From here I will complete the same steps on the second Branch ESXi host, RO-ESX02. I will spare you the steps.

Creating the vSAN Cluster

We can now finally get down to what we are trying to do, create the vSAN cluster! At this point I will assume you are still logged into the vSphere Web Client with Administrative privileges.

  1. Select the RemoteOffice cluster object. In the right-hand pane under the Configure tab select the General node under Virtual SAN. Next Click the Configure button to enable Virtual SAN:

  1. As I am using an All Flash configuration at my branch site I have elected to enable Deduplication and Compression. Regardless if you are doing an All Flash or Hybrid 2-Node ROBO configuration, you will need to Select the option for Configure two host Virtual SAN cluster. Click Next to continue:

  1. The configuration utility will check/verify the VMKernel port configuration on each of the ESXi branch hosts. As we can see we are good to go with vSAN Enabled on the given adapters. Click Next to continue:

  1. At this point vSAN will Claim Disks needed for the Cache and Capacity Tiers. In the screenshot below I am using a 4GB cache tier and 8GB capacity tier for each of the two hosts. Click Next to continue:

  1. Now we need to select the Witness Host for the cluster. I have drilled down our freshly deployed TUK-WTN02 host in my vSANWitness Datacenter object. Click Next to continue:

  1. Just like for the branch offices hosts, vSAN needs to claim the disks that will be used for the Cache and Capacity tiers. Since I select the Tiny appliance configuration the default settings are for a 10GB cache tier and a 15GB capacity tier. With the disks claimed, click Next to continue:

  1. On the last screen, review your settings and if all looks good click Next to Finish the configuration wizard and implement your vSAN cluster:

  1. After a few minutes (your mileage may vary) your vSAN configuration will be complete and you can see the corresponding Disk Groups in the Disk Management node. Or have a peek at the Capacity by going to the Monitor tab and Selecting the Virtual SAN option:

Closing Thoughts

While a lot of screenshots to get up and running, really setting up a 2-Node ROBO vSAN cluster is straight forward and quite simple. VMware continues to invent and incorporate “wizards” for setting up vSAN (and other features) and wouldn’t be surprised to see in the future the addition of static routes to be automated to make it even easier to implement.

Even if you haven’t jumped to vSAN/HCI at your main datacenter, if you are looking at a straightforward way to setup and mange clusters in remote or branch offices you might want to give it a look.

Thank you for reading!

-Jason

%d bloggers like this: