How vCenter server performance impact VMware agent collection (4292785)

Foglight VMware agent is designed for collecting infromation from vCenter to provide VMware virtual environment performance data for Foglight and slow/under powered vCenter server can cause collection fail

You will see following error in various log or UI

1. Form Foglight UI | VMware | Administration | Click Alarm box and see some last collection’s last update time are way off (Files collection is only exception as it only collect twice a day by default)

2. Some Virtual object e.g. Virtual Machine’s CPU have gray spinner and no/dotted line history graph

3. You may see consistency alarm Foglight alarm like this (Agent message. . An unexpected issue occurred during data collection from the VirtualCenter.  Please review your environment to ensure that data collection has resumed in a normal fashion. If there is a persistent problem or this message recurs regularly please contact Quest Support {Host Port=443, Host Name=, Host URL=sdk})

You may see following error in VMware agent log

1. ECHO    WARN [FglAM:IncomingMessage[5]-15] com.quest.glue.core.agent.StateChangeRequestProcessor - Could not acquire lock needed to move agent fb8be857-fbcf-4747-973d-295a8570e581/VMwarePerformance//VMwarePerformance/to request state NOT_RUNNING. The agent might be in an unexpected state.

java.util.concurrent.TimeoutException: Failed to acquire the WRITE lock for agent fb8be857-fbcf-4747-973d-295a8570e581/VMwarePerformance//VMwarePerformance/ because it is being held by another thread(s).  One of more of those threads could be hung or deadlocked. This occurs when an agent performs a long running operation in a method that holds the agent lock and declines to return it to the agent manager. Agents are required to respond to InterruptedExceptions and are strongly  encouraged to break long running operations into smaller pieces so that  they can respond in a timely fashion

2. ECHO /VMwarePerformance//  WARN [Quartz[0]-4631] com.quest.vmware.api.util.ServiceRequest - Thread execution timed out after 90 seconds while waiting for VC response during fetchDVPorts call. 

  Here are how you verify and check vCenter performance

1.       As most vCenter are running as virtual machine, so it is sharing the resource with other virtual machine and sometime CPU became a bottleneck and need  check vCenter server CPU contention (CPU ready time).

Here are how:

Via vSphere Client or web client

• Go to vCenter virtual machine and then select performance then advance tab
• Click chart options and select CPU | Real time
• Select Ready from counters box and you may need unselect some other counter is it only allow two distinct unit at a time

Via Foglight VMware cartridge

• Login to Foglight web UI
• Search vCenter name from right top corner global search box
• Select vCenter virtual machine and check CPU ready from CPU utilization chart under Summary Tab

“If you see Ready values of 10% or more per vCPU then you have CPU contention issue. In order to improve VM performance you could try decreasing the vCPU to pCPU ratio in current cluster or move vCenter virtual machine to a less busy cluster

2.       To check if you have too many Foglight/FVE are monitoring your vCenter

3.       If all good but still having issue, you many need checking if vCenter are sizing correctly. Please refer VMware KB2021302 (Configuring Tomcat server settings in VMware vCenter Server 5.1 and 5.5 (2021302) to adjust setting.