VCP 5 Objective 6.3 – Perform Basic vSphere Storage Troubleshooting

Objective 6.3 – Perform Basic vSphere Storage Troubleshooting

For this objective I used the following resources:

  • vSphere Monitoring and Performance
  • vSphere Storage
  • vSphere Troubleshooting
  • VMware KB Article 1008205 (Using ESXTOP to identify storage performance issues)
  • VMware KB Article 1003728 (Testing VMkernel network connectivity with the vmkping command)
  • VMware KB Article 1020692 (Frequent path state changes indicate storage problem)
  • VMware KB Article 1005009 (Analyzing SCSI Reservation conflicts)
  • VMware SAN System Design and Deployment Guide
  • Duncan Eppings Yellow-Bricks.com Blog

Knowledge

Verify storage configuration

Refer to the vSphere Storage and the SAN System Design and Deployment Guide (not specific to vSphere 5, but worth a read) by VMware. This will cover a lot of areas needed for working with a FC/iSCSI SAN environment with vSphere. Also a good understanding of the hardware you are using on the backend (storage arrays, FC switches, networking, etc) and there “vSphere Best Practices” documents will assist in the proper configuration.

Troubleshoot storage contention issues

When they mention “storage contention” I am taking this as I/O throughput or I/O latency issues. I find the quickest and easiest way of measuring/checking this is via esxtop/resxtop. VMware KB 1008205 and Duncan Eppings esxtop blog post covers this is in more detail.

Metrics to be aware of:

Disk Metric Threshold Description

DAVG

25

This is the average response time in milliseconds per command being sent to the device.

GAVG

25

This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG.

KAVG

2

This is the amount of time the command spends in the VMkernel

Also see pages 47 thru 50 of the vSphere Troubleshooting documentation for further information.

Troubleshoot storage over-commitment issues

Storage over-commitment can occur in several ways:

Troubleshoot iSCSI software initiator configuration issues

Things to look out for:

  • Setting up a proper iSCSI Network (pg 74 thru 79 of the vSphere Storage document)
  • Configuration of Jumbo Frames (pg 80 of the vSphere Storage document)
  • Network connectivity using vmkping (See VMware KB 1003728)

For further information refer to pages 107 thru 112 of the vSphere Storage documentation as well as page 51 of the vSphere Troubleshooting documentation.

Troubleshoot Storage Reports and Storage Maps

See pages 29 thru 32 of the vSphere Monitoring and Performance documentation.

Identify the root cause of a storage issue based on troubleshooting information

The vSphere Troubleshooting document covers several issues that you may run into. See Pages 45 thru 51.

Tools

  • vSphere Storage guide
  • vSphere Troubleshooting guide