Lets get started….

First step is to log into you ESXi host with you favorite SSH/Terminal software (using Putty in the screenshots) and run esxcfg-scsidevs –m. The command will list the currently mounted VMFS volumes and there vmhba names. Make note of the naa number for the device you wish  to mask, we will be using it later. In the example screen shot below the device ID is naa.5000144f88249906.

Next up is issuing the esxcfg-mpath –L command. This will provide the next piece of information we need, the number of paths to the device and the vmhba that is in use. For the example below I have a single iSCSI LUN (naa.5000144f88249906) that has two paths. One path going down vmhba37:C1:T0:L0 and the other down vmhba37:C0:T0:L0. Take note of this information as we will be using it further on to create our “rule”

Finally we will get a list of the current claim rules that are in place and to be sure we do not reuse an existing rule number. Issue the esxcli storage core claimrule list command:

From the screenshot above we can see the rules that are currently in place. For my example rules I am going to use rule numbers 200 and 201. Using the information we gathered from the esxcfg-mpath –L above will allow us to complete the needed esxcli command:

Next, verify that your new rules have been created by running another esxcli storage core claimrule list command. Note the new rules, rules 200 and 201:

With the new claim rule added the next step is to “reload” the claim rules via the esxcli storage core claimrule load command:

Note – This is a pretty uneventful screenshot.

With the claim rules reloaded, again run the esxcli storage core claimrule list command and notice the “runtime” entries that have been added:

Now, with the claim rules present we can actually reclaim the LUN or “Mask” it from the ESXi host. This is accomplished with the esxcli storage core claiming reclaim command and use the naa. identifier we collected above:

At this point you have successfully masked the LUN from the ESXi host. If you issue an esxcfg-scsidevs –m you will notice that the device is not longer listed (compared to the screen shots above) or if you do a rescan of the storage adapters from within the vSphere client:

Any questions or comments please let me know, and thank you for reading.

-Jason

Last weekend I received Larry Loucks book “Critical VMware Mistakes You Should Avoid” and finished it up the other night. The book was released in September of 2011 and I have had my eye on it for a few months hoping it would be released as Kindle version, to no avail. The book is a quick read, partially to Larry’s writing style and the other to the fact the book is 108 pages cover to cover.

In the book, Larry covers common misunderstandings and misconfigurations he has encountered in his years as a consultant and working with customers, either troubleshooting an existing environment or setting up a new one. The book consists of eight chapters covering  his adventures with half of the chapters based on the core four (cpu, memory, networking, and storage), and accounting for the lion share of the content. The remaining chapters cover VMware vSphere concepts, physical to virtual migrations, your path to virtualization and a “catch all” chapter discussing free hypervisors, monitoring and affinity/non-affinity rules. As these chapters are rather short it covers enough of the foundations to get you moving in the right directions.

Each of the core four chapters as well as the chapter on P2V migrations begin in the same manner, giving a brief overview of how that feature/resource is used or schedule in an ESX(i) environment. These concepts are then put on display via examples and scenarios of “What NOT to do” and “What TO do”. The examples/scenarios are covered first as the “What NOT to do” and are explained as why someone may configure or design something in that fashion. The “What TO do” sections flip the scenarios to the correct way and again offers the reasons to the why.

For me, I found the chapter on CPU usage and scheduling the most beneficial. Here Larry covers how ESX/ESXi schedules CPU resources for virtual machines across physical CPU cores  with single and multi vCPU virtual machines. Along with scheduling, “CPU Ready Time” is covered in scenarios where a physical host may be oversubscribed for CPU resources.

One drawback I have with the book is that it appears to be based on vSphere 4 or possibly 4.1. It does get into some discussions around “ESX Classic” that  going forward with vSphere 5 are no longer true or valid since VMware has standardized on ESXi.

With the information provided you get both VMware design elements as well as troubleshooting knowledge that comes from someone who has years of in the field experience and depth of knowledge. Overall I enjoyed reading the book and think Larry has put together a book that can be useful to someone just starting down the path of using VMware virtualization or even someone with a few years of experience under their belt.

-Jason

Once I got the device home, unboxed it, loaded up Google Chrome I thought what is next? So I reached out on Twitter and asked “What are the apps to get”. In a few short minutes I received replies from Matt Cowger (blog/twitter) and Andrew Miller (blog/twitter) letting me know they would send some lists over via email.

Since then I have received a few messages from folks wanting a copy of what Andrew and Matt provided me, so listed below is a combined list of each of there emails. Hopefully you will find them as helpful as I did.

Browser and Multimedia

Google Chrome – Web browser
Google Notifier – Access to your gmail and gcal from your menu bar
GrowlVoice – For Google Voice in menu bar
Adium – Multiprotocol IM client
VLC – Media player that plays must multimedia file types
PandaBar – Play Pandora from your menu bar
Echofon – Twitter client
Handbrake – Open-source video transcorder
Amazon Kindle App – Access your Amazon Kindle library
Netnewswire – RSS reader/weblog publisher
msgfiler – Keyboard based email filing application
ScreenFlow – Video capture software
Perian (codecs for Mac, lets Quicktime play anything)
Perian – Adds additional video formats to Quicktime
Flip4Mac – Play Windows WMV files in Quicktime

Productivity

Alfred – Application popup launcher
iTerm 2 – Mac OS Terminal replacement
The Unarchiver – Decompression tool
CrashPlan – System state  backups (free) or online backups (not free)
RazorSQL – SQL client
CyberDuck – FTP/SFTP client
Remoter – VNX+RDP client
Viscosity – OpenVPN client
CoRD – RDP client
Transmit – File transfer client on Mac (SCP, SFTP, FTP, S3)
TextMate – Text editor
TextWrangler – Text editor
Fusion 4 – Run a Windows machine on your Mac

System Tools

iStats Menu – Places CPU, Memory, Network, etc usage into menu bar
SmartReporter – Helps warn of hard drive disk failures
Unplugged – Power notification
SmartSleep – Lets you toggle the sleep modes
Dockstar – Gives multiple mail notification badges
TinkerTool – Access to additional preference settings

-Jason

When it comes to protecting VMware vCenter you are limited to just two VMware supported options (from my understanding, if this is incorrect please let me know), using VMware High Availability (HA) or VMware vCenter Server Heartbeat. Now I know some might bring up the VMware KB article 1024051 “Supported VirtualCenter high availability options” that calls out using MSCS and VCS. But for me the kicker is this nugget information in the second paragraph:

VMware does not certify these third party solutions. VMware will offer best effort support for any issues encountered with an environment that uses third party solutions for protecting against VMware VirtualCenter downtime.

So you tell me, if your vCenter is down and you are using one the above mentioned clustering solutions do you want to do the vendor support fingering pointing while your management is asking you why is it down and how long it will take to bring it back up? Exactly.

What if VMware HA doesn’t provide the resiliency or up time you require for your vCenter implementation? What if vCenter needs to be treated as a “Tier 1″ application? Enter vCenter Server Heartbeat (vCSHB going forward). vCSHB allows for the protection of vCenter, but also can be used to protect MS SQL as well several other VMware products (VMware Update Manager, Converter, Orchestrator, etc). vCSHB can be deployed to protect vCenter in two network configurations, LAN for local high availability (which is outlined in this guide), and WAN deployments for disaster recovery. Included with the network configuration options are the supported options on how you choose to run/protect vCenter. Available options are physical to physical, physical to virtual, and virtual to virtual (which is outlined in this guide).

For further reading and additional information on vCSHB view the links below:

VMware vCSHB product page

VMware vCSHB Administrators Guide

VMware vCSHB Installation Guide

VMware vCSHB Quick Start Guide

Now, onto the installation!

Creating the Primary Node

For this guide I am using the “Virtual to Virtual” pre-cloning process supported by vCSHB when both the primary and secondary will be running in a virtual machine. The primary system was configured as follows:

• Virtual machine was built with 2 vCPU’s, 4GB of memory, and 2 NIC’s (each NIC on a separate vSwitch (and VLAN in my config) – pg 22 of the Installation Guide
• Windows Server 2008R2 Enterprise with SP1 and latest patches
• VMware Tools
• Labeled one network interface as “Public” and the other as “VMware Channel”
• On each interface clear the “Register this connection’s address in DNS” check box – pg 30 of the Installation Guide
• List the Principal (Public) network adapter as the first adapter in the Network Connections Binding Order – pg 24 of the Installation Guide
• Joined the server to my domain
• Added a SQL service account (svc_sql) to the Local Administrators group – pg 29 of the Installation Guide
• Installed SQL Server 2008 R2
• Installed vCenter Server 5
• Installed vCenter Update Manager 5 (configured using FQDN) – pg 20 of the Installation Guide
• Set the following services to manual startup type – pg 30 of the Installation Guide:
  • VMware VirtualCenter Server
  • VMware vSphere Profile-Drive Storage
  • vCenter Inventory Service
  • VMware VirtualCenter Management Webservices
• Create vCSHB folder on root of C: and copied the vCSHB-v6-4-6643.exe file to it

After completing the above steps I powered down the VM and via the vSphere Client I created a clone of the VM to be used as the secondary node (appended Secondary to the VM name). Once the clone task had finished I powered on the primary node to being the vCSHB installation.

vCSHB Installation on Primary Node

Once the VM was powered on I logged onto the server with administrative credentials and launched the executable from the C:vCSHB folder I created:

After the package has been fully extracted the “Setup Type” screen will be displayed. Select “Install VMware vCenter Server Heartbeat” and click “Next”:

For “Physical Hardware Identity” select “Primary” and click “Next”:

On the licensing page, read the license agreement if you choose, select “I accept terms of the License Agreement” and click “Next”:

Now if this was a true install you can either “Add” or “Import” an existing license key:

Adding a license key if you have one

Since this is just a demo install there is nothing to add or import, just click “Next” to continue:

For the “Select Topology” page select “LAN” as we are only doing a HA environment on the local network:

On the “Deployment Option” screen select “Secondary Server is Virtual” and click “Next”:

For the “Installation Paths” if you don’t want use the default click the “Browse” button and choose a new path. Also choose whether or not to “Create icons on Desktop”. Click “Next” to continue:

On the “Channel Adapter Identification” screen select the interface labeled “VMware Channel” and click “Next”:

For the “VMware Channel IP Configuration” screen click “Add”:

Enter the IP addresses being used on both the Primary and Secondary nodes for the VMware Channel interface, click “OK”:

You will receive an error message stating that the setup is unable to communicate with the Secondary node. This is fine as we have not turned on the Secondary node. Click “No” to continue with the setup:

With the “VMware Channel IP’s” configured for both the Primary and Secondary nodes click “Next”:

On the “Public Adapter Identification” screen select “Public” and click “Next”:

For the “Public IP Address Configuration” click “Add”:

Select the correct IP address from the drop down, in my case 10.20.0.81. Click “OK”:

Leave the default “Use same IP addresses for Secondary (Recommended for HA secondary)” selected and click “Next”:

On the “Client Configuration” leave the default port at 52267 unless you have a need to change it. Click “Next”

For “Application Protection” in this demo we are protecting both vCenter and SQL Server. Select “Protect vCenter Server and SQL Server” and enter valid vCenter Login credentials. Click “Next” to continue:

On the “Pre-synchronization Data Configuration” screen, in the “Folder” dialog enter the full path to the vCSHB folder we created on the root of the C: drive. Note – Use the VMware Channel IP address as this is how the Secondary node will connect and import the configuration information during its setup. Click “Next” to continue:

The “Installation Summary” will be displayed. After reviewing the settings click “Next”:

If the “Pre-Install Checks” run successfully click “Next” to continue:

Once the “Install” has completed click “Next”:

Once the “Packet Filter Installation” has completed click “Next”

On the “Primary Installation Complete” screen click “Finish”

Once the installation has finished the “Server Configuration” window will open. Select the “Public” tab and set the “Public Identity Mode” to “Identical”. Click “Finish”:

At this point your Primary node is configured and ready to go. The next step is to build out the Secondary node which will be outlined in an upcoming blog post. Stay tuned!

-Jason

Both of those are great features (as well as the many others I didn’t mention), but one I think that flew a little under the radar was the support for multi NIC vMotion. How much time as a VMware Administrator have we waited for VM’s to be evacuated from a host going into maintenance mode? If you were like me and only had 1GB uplinks on your hosts you were limited (IE supported by VMware) to four concurrent vMotions. Set Maintenance Mode and go to Starbucks. If you are one of the cool kids and have 10GB uplinks that number gets doubled to 8 concurrent vMotions. Set Maintenance Mode and run to the break room for a coffee refill.

If these numbers look familiar they should, they are the same as vSphere 4.1. And this is were the joy of Multi NIC vMotion comes in! Since the amount of concurrent vMotions hasn’t changed, how about changing the amount of NIC’s that can be used? You can still only move four VM’s at once, but instead of using a single NIC how about two or four? Same amount VM’s being moved but just more bandwidth to move them. If you are utilizing 1GB NICs the maximum you can combine is 16. For 10GB that number is decreased to four.

If you are not familiar with setting up Multi NIC vMotion both VMware and Duncan Epping have fantastic posts on how to do so. The VMware KB article (KB2007467) is located HERE and the blog post from Duncan is located HERE.

So after following either of the above guides lets do some checking to see if it truly is working. I find the easiest to see this in action is via ESXTOP. Log into one of your hosts (or vMA) launch ESXTOP and press “n” to view the networking screen. Identify your vmk/vmnics that you configured and verify that traffic is crossing each.

To demonstrate this I setup a vSwitch with two uplinks and configured two vMotion portgroups. In the screen shot below you can see traffic crossing over each interface.

A more in depth way to see this as well is to view the /var/log/vmkernel.log. In the screenshot below you will see the kernel binding to the two vmk ports:

For overall performance testing in my lab environment I tested moving four VM’s with only a single vMotion interface and the same four VM’s with multiple vMotion interfaces:

As you can see from the chart there was almost a 50 percent reduction in time when comparing the single interface to the dual interface. If you have upgraded your environment to vSphere 5 be sure to revisit your network design to make sure you are make use of this great new feature.

-Jason

I received the drives on Thursday and brought down one of my hosts and swapped out the existing internal RAID 5 I was running for the SSD’s. I installed ESXi on one SSD and used the second as a dedicated datastore (this config is just for performance testing, each host will end up with a single SSD) to start some testing.

My ESXi hosts are whitebox machines made up of an ASUS P7F-E motherboard, Intel Xeon X3430 Lynnfield 2.4GHZ QC, and 16GBs of RAM. One draw back of my hosts is that the onboard SATA controller is only 3GBs where as the drives are 6GBs capable which will have in effect on my performance numbers.

For testing I created a new VM based on Windows Server 2008 R2 with 2GB of memory and a single vCPU. IOmeter was installed on the VM and I used the configuration provided from the “Unofficial Storage Performance” thread located HERE on the VMware Communities. Each test was performed three times and used the averages to populate the charts below. The tests where performed with the VM running on the same SSD as ESXi was installed (Shared Datastore) and the VM running on a SSD that was configured as a standalone datastore (Dedicated Datastore).

And now the results:

As you can see from the charts above the overall IOPS and MBps are about the same whether the VM was running on the shared or dedicated datastore. However, I was surprised by the changes in the Max IO Response Time numbers.

All in all very happy with the performance numbers I am seeing, and as the main goal was to replace the internal RAID5 on each host (mostly from a power consumption perspective) I am glad I made the purchase.

-Jason

I would have liked to say that branching out and moving my career another step forward was a new years resolution, but in fact it was something that started back last summer. At that time my goal was to learn more and dig deeper into VMware, go beyond my VMware VCP certification and obtain my VCAP-DCA certification. I scheduled the exam for August and set off on my studying. Long story short I had successfully passed the exam and was feeling good about the additional knowledge I had learned. But that was nothing compared to what happened later that month.

Even though I have been working with VMware for the last 4 years or so, I had never had the ability/option of attending VMworld. There had always been talk amongst managers about sending someone, but it never really seemed to pan out. This year that was all going to change, my boss had notified me that I would be attending VMworld 2011! For me this had to be the biggest eye opening event in my career. I had heard on podcasts, and read on blogs about experiencing it, but none of what I read or heard prepared me for what I was about to see. Between all of the sessions, the vendor displays, or just talking to and meeting new folks it was amazing! After soaking up as much as I could, it was at this point I knew I wanted to, better yet needed to take that next step.

So here I am, waiting for that new challenge, that new adventure to start. I got my bags packed and am ready to go!

For this objective I used the following resources:

• vSphere Monitoring and Performance
• vSphere Troubleshooting
• VMware White Paper “The Architecture of VMware ESXi”
• VMware Support Insider Blog
• VMware KB Article 2000988 (Troubleshooting vSphere Auto Deploy)
• VMware KB Article 1011641 (Collecting Diagnostic Information for VMware vCenter Server)
• VMware KB Article 653 (Collecting Diagnostic Information for VMware ESX/ESXi Using the vSphere Client)
• VMware KB Article 1010705 (Collecting Diagnostic Information for VMware ESX/ESXi Using the vm-support Command)
• VMware KB Article 1027932 (Collecting Diagnostic Information for VMware vCenter Server and ESX/ESXi Using the vSphere PowerCLI)

Knowledge

Identify General ESXi Host Troubleshooting Guidelines

The vSphere Troubleshooting guide is the one stop shop for this section

Troubleshoot Common Installation Issues

Refer to Objective 1.3 and make sure your hosts meet the hardware requirements as well as the VMware HCL. If using AutoDeploy refer to pages 20 thru 26 of the vSphere Troubleshooting guide and also VMware KB 2000988 (Troubleshooting vSphere Auto Deploy).

Monitor ESXi System Health

With the release of ESXi back in the VI 3.5 days it provided a new way to manage your hosts, the Common Information Model (CIM). CIM allows for a standard framework to manage computing resources and presents this information via the vSphere Client. For further information read the VMware White Paper “The Architecture of VMware ESXi” as well as this VMware Support Insider blog post.

To actually see how ESXi and vSphere Client leverages CIM read pages 25 thru 28 of the vSphere Monitoring and Performance documentation.

Export Diagnostic Information

There are multiple ways to get at this information, but I will assume the exam is going to be geared more towards using the vSphere Client for this task. For completeness however I have included links that cover using a console session as well as PowerCLI.

Gathering vCenter Server Log Bundles (VMware KB 1011641, Collecting Diagnostic Information for VMware vCenter Server)

To generate a vCenter Server log bundle, select Start > All Programs > Vmware and select either “Generate vCenter Server Log Bundle – Extended” or “Generate vCenter Server Log Bundle” (Be sure to “Run as Administrator”).

Gathering vCenter Server and ESXi Log Bundles (VMware KB 653, Collecting Diagnostic Information for Vmware ESX/ESXi Using the vSphere Client)

From within the vSphere Client connected to vCenter click Administration from the menu bar and select Export System Logs. This will allow you to export either vCenter logs, ESX/ESXi logs, are all of the above in a single zip file.

To use the vm-support command or PowerCLI refer to VMware KB 1010705 (Collecting diagnostic information for VMware ESX/ESXi using the vm-support command) and VMware KB 1027932 (Collecting diagnostic information for VMware vCenter Server and ESX/ESXi using the vSphere PowerCLI)

Tools

• vCenter Server and Host Management Guide
• vSphere Monitoring and Performance Guide
• vSphere Troubleshooting Guide

For this objective I used the following resources:

• VCP 5, Section 2 – Plan and Configure vSphere Networking
• VMware Information Guide “VMware Virtual Networking Concepts”
• VMware KB Article 1003893 (Troubleshooting Virtual Machine Network Connection Issues)
• VMware KB Article 1001938 (ESX/ESXi Host Requirements for Link Aggregation)
• VMware KB Article 1004048 (Sample Configuration of EtherChannel/Link Aggregation with ESX/ESXi and Cisco/HP Switches)
• VMware KB Article 1005577 (What is beacon probing)

Knowledge

Verify Network Configuration

Refer to each objective under Section Two. Focus on the core concepts and configuration of both vNetwork Standard Switches and vNetwork Distributed Switches:

• Port/dvPort Groups
• Load Balancing and Failover Policies
• VLAN Settings
• Security Policies
• Traffic Shaping Policies

For additional information read the VMware Information Guide “VMware Virtual Networking Concepts”. This document is based on VI3 but still does a good job with the core functions of a vStandard Switch.

Verify a Given Virtual Machine is Configured with the Correct Network Resources

Instead of duplicating work, refer to VMware KB 1003893, “Troubleshooting Virtual Machine Network Connection Issues”. More then enough information listed there.

Troubleshoot Virtual Switch and Port Group Configuration Issues

One key aspect to remember is when setting up Port Groups or dvPort Groups, spelling counts (as well as upper/lower case)! If a Port Group is spelled Test on one host and is spelled test on a second host vMotion will fail. Same holds true with Security Policies, if one vSwitch on a host is set to accept Promiscuous Mode and it is set to Reject on the other host, again vMotion will fail. Also, refer to the objectives under Section Two to be sure your switches are configured correctly.

Troubleshoot Physical Network Adapter Configuration Issues

This is pretty straight forward as there is not a lot of configuration done at the physical network layer. Be sure that your physical nics that are assigned to a virtual switch (vSwitch or dvSwitch) are configured the same (speed, vlans, etc) on the physical switch. If using IP Hash as your load balancing method make sure on the physical switch side link aggregation has been enabled. Refer to VMware KB 1001938 and VMware KB 1004048 for further details as well as examples. If using beacon probing for network failover detection it standard practice to use a minimum of three (or more) uplink adapters. See VMware KB 1005577 for further details.

Identify the Root Cause of a Network Issue Based on Troubleshooting Information

Using the above notes as well as the linked VMware KB articles one should be able to isolate issue to one of four areas:

• Virtual Machine
• ESX/ESXi Host Networking (uplinks)
• vSwitch or dvSwitch Configuration
• Physical Switch Configuration

Tools

• vSphere Troubleshooting guide
• vSphere Networking guide

For this objective I used the following resources:

• vSphere Monitoring and Performance
• vSphere Storage
• vSphere Troubleshooting
• VMware KB Article 1008205 (Using ESXTOP to identify storage performance issues)
• VMware KB Article 1003728 (Testing VMkernel network connectivity with the vmkping command)
• VMware KB Article 1020692 (Frequent path state changes indicate storage problem)
• VMware KB Article 1005009 (Analyzing SCSI Reservation conflicts)
• VMware SAN System Design and Deployment Guide
• Duncan Eppings Yellow-Bricks.com Blog

Knowledge

Verify storage configuration

Refer to the vSphere Storage and the SAN System Design and Deployment Guide (not specific to vSphere 5, but worth a read) by VMware. This will cover a lot of areas needed for working with a FC/iSCSI SAN environment with vSphere. Also a good understanding of the hardware you are using on the backend (storage arrays, FC switches, networking, etc) and there “vSphere Best Practices” documents will assist in the proper configuration.

Troubleshoot storage contention issues

When they mention “storage contention” I am taking this as I/O throughput or I/O latency issues. I find the quickest and easiest way of measuring/checking this is via esxtop/resxtop. VMware KB 1008205 and Duncan Eppings esxtop blog post covers this is in more detail.

Metrics to be aware of:

Also see pages 47 thru 50 of the vSphere Troubleshooting documentation for further information.

Troubleshoot storage over-commitment issues

Storage over-commitment can occur in several ways:

• As mentioned above with I/O latency
• Datastore usage (Think Thin Provisioning and over subscription)
• SCSI reservation issues (VMware KB 1005009)
• Path thrashing (VMware KB 1020692)
• LUN queue depth

Troubleshoot iSCSI software initiator configuration issues

Things to look out for:

• Setting up a proper iSCSI Network (pg 74 thru 79 of the vSphere Storage document)
• Configuration of Jumbo Frames (pg 80 of the vSphere Storage document)
• Network connectivity using vmkping (See VMware KB 1003728)

For further information refer to pages 107 thru 112 of the vSphere Storage documentation as well as page 51 of the vSphere Troubleshooting documentation.

Troubleshoot Storage Reports and Storage Maps

See pages 29 thru 32 of the vSphere Monitoring and Performance documentation.

Identify the root cause of a storage issue based on troubleshooting information

The vSphere Troubleshooting document covers several issues that you may run into. See Pages 45 thru 51.

Tools

• vSphere Storage guide
• vSphere Troubleshooting guide