VMware scheduled job ran longer than its fixed rate and misfire may be a result of system clock drift (4213760)

The following two messages can be observed from the FglAM (Agent Manager) logs:

WARN    [Quartz[0]-80] com.quest.glue.common.util.scheduler.quartz.QuartzScheduler - The fb8be857-fbcf-4747-973d-295a8570e581/VMwarePerformance/5.8.3/VMwarePerformance/vcenter01.example.com-Collector.esxHardwareCollection scheduled job ran longer [427,995 ms] than its fixed rate of 300,000 ms. 3 scheduled executions were missed.
[...]
WARN    [FglAM_QuartzSchedulerThread] com.quest.glue.common.util.scheduler.quartz.QuartzScheduler - Scheduled job fb8be857-fbcf-4747-973d-295a8570e581/VMwarePerformance/5.8.3/VMwarePerformance/vcenter01.example.com-Collector.esxHardwareCollection misfired by -128,017 ms and will not be run. Cause of misfire may be a result of system clock drift and correction by a time-sync utility.

The most common cause for this issue is that the collection tasks take longer to complete than the collection interval. E.g.: the collection interval for Inventory and Relations is set to 5 minutes (300 seconds) but esxHardwareCollection tasks take 7 minutes (420 seconds).

For Foglight Evolve version 9.3 or 6.0, defect FOG-2822 can lead to longer collection times.

Although this is very rare, this could also potentially be caused by a clock resync event during the collection causing an error in the computed elapsed time. This would generate occasional (non-recurrent) errors as in the description.

Check time sync status from VMware tools command line

Windows:

C:\Program Files\VMware\VMware Tools>VMwareToolboxCmd.exe timesync status

Also, the following Event Viewer, Event IDs will show ntp synchronization messages:

• Event ID 37
• Event ID 35
• Event ID 47

Linux:

vmware-toolbox-cmd timesync status

WORKAROUND

Increase the collection interval for the collection tasks that generate these errors. From the sample error in the description, esxHardwareCollection would require an increase to the Inventory and Relations collector to 8 or 10 minutes.

1. Navigate to Administration │ Agents │ Agent Status
2. Locate the VMWare Performance Agent in question, then click on Edit Properties
3. Click Modify the private properties for this agent
4. Locate Collector Config (in Data Collection Scheduler section); if it's using defaultSchedule, click on Clone and then give it a new name for this agent; if it's already a custom list, click on Edit.
5. Increase the Default Collection Interval of the corresponding Collector that generated the error messages
6. Click Save to save the changes to the list
7. Click Save to save any changes to the selected Collector Config list
8. Deactivate and Activate the VMware Performance agent
9. Let the agent run for a few hours to confirm the collections run normally.

Other considerations

For Evolve 9.3 or 6.0, please see KB 4305677 or consider upgrading to a supported version of Foglight Evolve.

If the FMS (Management Server) and/or FglAM (Agent Manager) run on Virtual Machines, make sure vCenters, FMS and FglAM hosts have correct ntp sync setup.

VMware Timekeeping best practices:

• VMware Timekeeping best practices for Linux and Windows
• Timekeeping best practices for Linux guests
• Timekeeping-In-VirtualMachines

OTHER CHECKS

1. Check your firewall setting to unblock NTP UDP port 123.

2. If running NTP through Windows and if having service "Windows Time" running, please run the following commands from command line to check the status:

w32tm /query /configuration (checks and shows the NTP configuration)
w32tm /query /source (shows the source of the time)
w32tm /query /status (shows the service status, use it to check whether the time is coming from the external NTP servers or the cmos clock)
w32tm /query /status/verbose