Notes from the Field: VSAN Design–Networking

NoSANA few weeks back I published a post related to a VMware VSAN design I was working on for a customer (located here). That post focused mostly around the key area that VSAN addresses, storage. While the storage piece is where VSAN shines and has the most moving parts to understand from a design and implementation perspective, you can’t forget about the network. With the scale out nature of  VMware VSAN, the network connectivity between hosts to carry replica VM storage traffic becomes increasingly important.

As this post and the previous post are based on a customers design  leveraging  VSAN in a net new infrastructure we are implementing 10Gb Ethernet connectivity for the ESXi host connectivity. Two factors played into this decision, first was the fact that 10Gb Ethernet over the last few years has come down in pricing allowing for a greater adoption rate. Second, as we are deploying VSAN, VMware recommends using 10Gbe to provide the needed network throughput/bandwidth to handle the storage traffic.

Since we are “building” our own VSAN nodes as mentioned in the storage post, it was off to the VMware HCL I/O section to confirm/verify supported 10Gb Ethernet NICs to be used with our Dell R720 servers. We will be using copper based 10GBase-T switches  for connectivity so the servers will be configured with redundant Dell OEM’ed Intel Ethernet 10G 2P x540-t adapters. For the initial deployment we will be using one port from each card to provide redundancy and availability.

Someone Mixed VDS with My VSAN

While VSAN brings along with it some cool technology related to storage, one piece that is overlooked (or hasn’t received enough attention in my opinion) is that when licensing VSAN, VMware bundles in the ability to utilize the Virtual Distributed Switch (VDS). This feature is normally reserved for deployments involving VMware’s Cadillac version of licensing, Enterprise Plus. Leveraging the VDS along with the Network I/O Control (NIOC), a feature that is only available on VDS, allows for a streamlined installation/configuration of the vSphere environment. Additionally deploying the VDS in 10Gbe VSAN environment is preferred by VMware. The below quote is taken from page 7 of the VMware Virtual SAN Design and Sizing Guide:

Virtual SAN provides support for both vSphere standard switch and VMware vSphere Distributed Switch™, with either 1GbE or 10GbE network uplinks. Although both vSphere switch types and network speeds work with Virtual SAN, VMware recommends the use of the vSphere Distributed Switch with 10GbE network uplinks.”

If you are not familiar with VDS or NIOC, Frank Denneman has a great primer post on the feature and functionality, that post can be viewed here. Also, though a bit dated, VMware has an excellent whitepaper around VDS design and implementation. VMware vSphere Distributed Switch Best Practices  is available here. The diagram below provides an overview of how the hosts will configured and will communicate at both the physical layer as well as the VDS/portgroup layer.

image

For sake of simplicity the diagram above only show the use of five portgroups that will need to be created on the VDS for our deployment. The traffic type and VDS teaming policy for each portgroup is listed in the table below:

Traffic Type

Port Group

Teaming Option

Active Uplink

Standby Uplink

Management

Mgmt

LBT

vmnic0/vmnic2

N/A

vMotion

vMotion-1

Explicit Failover

vmnic0

vmnic2

vMotion

vMotion-2

Explicit Failover

vmnic2

vmnic0

VSAN

VSAN

Explicit Failover

vmnic0

vmnic2

Virtual Machine

Virtual Machine

LBT

vmnic0/vminc2

N/A

 

Virtual SAN Networking Requirements and Best Practices

VMware has published a guideline for VSAN requirements and deployment best practices. Below is the listing from VMware vSphere 5.5 Documentation Center located here.

  • Vitual SAN does not support IPv6
  • Virtual SAN requires a private 1Gb network. As a best practice, use 10Gb network.
  • On each host, dedicate at minimum a single physical 1Gb Ethernet NIC to Virtual SAN. You can also provision one additional physical NIC as a failover NIC.
  • You can use vSphere standard switches on each host, or you can configure your environment with a vSphere Distributed Switch.
  • For each network that you use for Virtual SAN, configure a VMkernel port group with the Virtual SAN port property activated.
  • Use the same Virtual SAN Network label for each port group and ensure that the labels are consistent across all hosts.
  • Use Jumbo Frames for best performance.
  • Virtual SAN supports IP-hash load balancing, but cannot guarantee improvement in performance for all configurations. You can benefit from IP-hash when Virtual SAN is among its many consumers. In this case, IP-hash performs the load balancing. However, if Virtual SAN is the only consumer, you might not notice changes. This specifically applies to 1G environments. For example, if you use four 1G physical adapters with IP-hash for Virtual SAN, you might not be able to use more than 1G. This also applies to all NIC teaming policies that we currently support. For more information on NIC teaming, see the Networking Policies section of the vSphere Networking Guide.
  • Virtual SAN does not support multiple VMkernel adapters on the same subnet for load balancing. Multiple VMkernel adapters on different networks, such as VLAN or separate physical fabric, are supported.
  • You should connect all hosts participating in Virtual SAN to a single L2 network, which has multicast (IGMP snooping) enabled. If the hosts participating in Virtual SAN span across multiple switches or even across L3 boundaries, you must ensure that your network is configured correctly to enable multicast connectivity. You can change multicast addresses from the defaults if your network environment requires, or if you are running multiple Virtual SAN clusters on the same L2 network.
    I hope this post as well as the original post is helpful in designing/implementing your VSAN environment.

-Jason