-
Title
Observation values are being discarded due to having timestamps outside of the allowable range. -
Description
Observation values are being discarded due to having timestamps outside of the allowable range from the management server time alarm.
How to determine which Agents and Agent Managers are submitting observations with older timestamps?
-
Cause
Observations submitted by one or more FglAMs (Agent Managers) are being submitted with older timestamps.
One possible cause for this issue is due to incorrect time in the system clock of one of the FglAMs or Monitored Host.
Another possible cause is the Collect log events option on WindowsAgent agents can collect records up to one hour old by default. -
Resolution
STATUS: FOGGENERAL-I-23 submitted for consideration for alarm to identify the offending object.
Procedure to identify the offending object.
1) Review the current FMS log such as $FGLHOME/logs/ManagementServer_2019-02-08_093729_001.log
2) Search for messages similar to the following: You may find these repeating each hour. This example shows the object time stamp is 1hr 8min older than the FMS time.
WARN [Data-4-thread-12141] com.quest.nitro.service.data.DataService - Discarded value of observation "failedCollections" on topology object "collect (FglamAgentCollector)" (8434e01c-8054-4681-8617-e55fecc5efc4) because its timestamp is very different from that of the management server (5/31/19 7:24 AM vs 5/31/19 8:32 AM). Please check the server's store and forward configuration and verify the system time of the originating host.]
3A) Look up the objects directly using the Object ID in the Administration | Tooling | Script Console using the input in the lower panel. If it's a FglamAgentCollector object, then clicking on the agent property value should show the properties for the actual agent submitting older metrics as in the example below.
3B) Alternatively, instead of looking for each Object ID individually, use the script findDiscardingAgents.groovy (attached) to get Agent Name, Type and FglAM. Make sure to replace the values in the ids list. -
Defect ID
FOGGENERAL-I-23 -
Attachments