Chat now with support
Chat with Support

Foglight for Storage Management Shared 4.8.5 - User and Reference Guide

Getting Started Configuring Agents to Monitor Storage Devices
Brocade SAN Switches Cisco SAN Switches Dell Compellent Arrays Dell EqualLogic PS Series Array Groups EMC CLARiiON CX Series Arrays (CLI) EMC CLARiiON, VNX, or VMAX Storage Arrays (SMI-S) EMC Isilon EMC VPLEX Hitachi Data Systems AMS, USP, and VSP HP EVA Storage Arrays HP 3PAR Arrays NetApp Filers Configuration Procedures
Managing Agents Using Foglight for Storage Management Monitoring Storage Performance Investigating Storage Devices Investigating Storage Components Investigating VPLEX Storage Troubleshooting Storage Performance Managing Data Collection, Rules, and Alarms Understanding Metrics Appendix: Collection Target Support Matrix Online-Only Topics

Troubleshooting Storage Performance

Administrators may receive problem reports from stakeholders about the performance of a VMware virtual machine (VM). When the suspected cause is storage, the Administrator can run an automated analysis using the Storage Troubleshooting dashboard. The analysis can quickly rule out a storage issue, allowing the Administrator to focus on other areas. Conversely, if a storage issue is found to be contributing to poor performance, the results of the analysis clearly highlights the datastores or RDM disk extents that require attention.

This section is intended for Foglight for Storage Management users with the role of Storage Administrator. It describes how to start an investigation, analyze the results, and change latency thresholds. It also summarizes the algorithm used to identify and assess storage performance issues.

This section describes the following topics:

•
Starting a Troubleshooting Investigation
•
Analyzing Storage Issues
•
Analyzing the Pool
•
Changing Latency Thresholds
•
Understanding the Troubleshooting Algorithm

Starting a Troubleshooting Investigation

Before beginning an investigation, you should ask the person reporting the issue the following questions:

•
What is the host name of the affected virtual machine?
•
When did you first notice the performance issues? The reported time frame determines how you set the zonar time range.
To investigate a potential storage performance issue:
1
On the navigation panel, under Dashboards, click the Insights tab, and then click Live Troubleshooter.
2
Set the zonar to encompass the reported time frame, up to a maximum of 8 hours.
The zonar is located in the upper right corner of the dashboard.
3
Type the host name of the virtual machine.
4
The latency threshold values are set automatically based on the latency registry variables. If different values would be more meaningful for your analysis, see Changing Latency Thresholds.
5
Click Perform Analysis.
Depending on the complexity of the analysis request, it may take a little time before the results of the analysis appear on the dashboard.
Recall that a virtual machine can get its storage either from a datastore (connected in turn to one or more disk extents or to a NASVolume) or directly from an RDM disk extent (without a datastore). In the analysis results, each datastore or RDM disk extent that is connected directly to the selected virtual machine is represented by a separate view.
6
Review the icons displayed in the title bars of each datastore/RDM view.
•
If a Normal icon appears in all title bars, the performance issue is not storage-related. This investigation is complete.
 
* 
TIP: In some cases, the Diagnosis summary (right-hand panel) may contain hints about where to look next. For example, increased I/O (as compared to typical I/O for this period) may indicate that an application is behaving differently than before.
•
If the Attention icon appears in one or more title bars, continue your investigation following the workflow described in Analyzing Storage Issues.

Analyzing Storage Issues

If the view for a datastore or RDM disk extent shows the Attention icon, the troubleshooting algorithm has discovered evidence of a performance problem related to storage. The problem may or may not be in the SAN Storage environment. Review the details to determine the cause of the performance issue.

Each datastore/RDM view has three summary panels (from left to right):

•
VM I/O to Datastore/RDM (first panel)
•
Latency for Disk Extents (middle panel)
•
Diagnosis (last panel)

A virtual machine may be connected to multiple datastores and RDM disk extents, each of which may report varying degrees of problems. When a virtual machine has more than one datastore/RDM view, start by scanning the timeline bars in the VM I/O to Datastore/RDM panel to identify a datastore/RDM with consistently slow I/O performance or significant changes from typical performance.

The following workflow describes one way to identify a latency problem in the collected SAN Storage environment. While the details in your investigation may differ, the general workflow should be similar to this one.

 
To analyze storage issues:
1
In a view showing the Attention icon, scan the VM I/O to Datastore/RDM summary (first panel). Look for timeline bars that primarily show colors such as yellow, orange, or pink, that is, any color other than green (which represents acceptable activity).
In this example, the VM Latency vs Threshold timeline is orange, which means the virtual machine is consistently exceeding the default latency thresholds that were specified for the analysis. We should focus our investigation here.
The VM Latency vs Typical timeline is green, which means that the latency is typical for the time period; this behavior has been going on for some time. The typical values are statistical values determined by IntelliProfile from activity within the last 30 days.
2
Now look at the Latency for Disk Extents summary (middle panel) to identify the disk extents that are contributing to the problem.
 
* 
NOTE: When a datastore is connected to a NASVolume, this panel is empty.
In this case, there is only one disk extent attached to the datastore. Its timeline is orange, which means the disk extent is exceeding latency thresholds. The number in brackets indicates that the virtual machine was performing I/O to the disk extent while the VM was experiencing latency. The larger the number, the more I/O was occurring. When this number is zero, no I/O occurred while the VM was experiencing latency.
 
* 
TIP: In your own investigations, you may have more than one disk extent, in which case you may be able to see that one disk extent is slow while the others are normal. Or you may have multiple virtual machines sharing the same disk extent. In this case, the Diagnosis panel may display a message to let you know that the latency may not reflect this virtual machine alone.
3
Next, review the notes in the Diagnosis summary (last panel).
In this case, one of the notes describes a correlation between the virtual machine latency and the disk extent latency. The other note points toward a problem in the SAN Storage or the network, and below is a button that begins an analysis of the SAN Storage.
 
* 
NOTE: If the troubleshooting algorithm determines that the issue is likely in the SAN Storage environment, and if the SAN Storage is provided by an element that the Foglight for Storage Management system is monitoring, the Analyze SAN Storage button appears in the Diagnosis panel.
4
Before analyzing the SAN Storage, you may want to quantify the performance issue by reviewing the metric values that underlie the timeline bars in the summary panels.
a
In the VM I/O to Datastore/RDM summary, click the Chart icon.
The charts show the values of the metrics over the time period. Some charts also contain a baseline range, which shows a statistical range of values encountered over the last 30 days. Spikes outside of the normal range represent a significant change in behavior that may warrant further investigation.
In this case, the top chart shows that the VM I/O latency is hovering around 200 m/s, well above the default 25 m/s and 35 m/s latency thresholds.
b
Close the window.
c
In the Latency for Disk Extents summary, click the Chart icon.
The chart shows the disk extent latency values hovering around 180 m/s. The latency here may reflect the activity of multiple VMs doing I/O or the performance within the ESX itself. It is clear that a significant delay is occurring in the disk extent.
d
Close the window.
5
In the Diagnosis panel, click Analyze SAN Storage.
The Storage-Side Analysis window breaks down performance by the Extent, the LUN to which it belongs, and the pool to which the LUN belongs.
This view shows that the disk extent is slow for all virtual machines and for the selected virtual machine. The disk extent to LUN I/O performance also exceeds latency thresholds, though its current performance is within the baseline range, which shows a statistical range of typical values encountered over the last 30 days. This indicates the high latency has been occurring for some time. If you want to see the metric values, click the Chart icon.
6
The next step in the investigation depends on the state of the pool.
•
If the pool timeline bars are green, the investigation is complete.
•
If the pool timeline bars are a color other than green, you can analyze the changes within the pool and the load on the pool. Continue the investigation by following the workflow in Analyzing the Pool.
In this example, the pool’s Avg Queue Depth timeline bar is yellow. The note beside the bar suggests that this may be a cause of slow I/O to the LUN. The pool warrants further investigation.

Analyzing the Pool

When pool timeline bars show abnormal average queue depth or ops rate, analyze the changes within the pool and the load on the pool.

•
Perform Pool Change Analysis. The Pool Change analyzer identifies the LUNs primarily responsible for increased I/O. It compares LUN activity in the problem time range with LUN activity during the same time range in the past. Changes are reported in terms of average operations rate and change amount.
•
Perform Pool Load Analysis. The Pool Load analyzer identifies the busiest LUNs and ranks them based on their activity during the same time range over the last 30 days (not the current time frame). Activity is measured in operations per second.
 
To analyze the pool:
1
From the Server-Side Analysis window or the Pool Explorer window, click Perform Pool Change Analysis.
The analyzer compares activity for all the LUNs in the pool during the problem time range with their activity a week ago. In this example, the table and pie chart confirm that the LUN with the highest average operations rate and change amount is the LUN that includes the disk extent used by the poor performing virtual machine.
 
* 
TIP: You can change the comparison time range by clicking Change and selecting a new date and time range.
2
Close the window.
3
Click Perform Pool Load Analysis.
The chart and table show the top ten busiest LUNs. The analysis is based on the last 30 days of activity during this time range. In this example, the table confirms that the top two busiest LUNs are connected to the poor performing virtual machine.
4
Close the window.
The investigation is complete.
Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating