Observation values are being discarded due to having timestamps outside of the allowable range from the management server time alarm.
How to determine which Agents and Agent Managers are submitting observations with older timestamps?
STATUS: FOGGENERAL-I-23 submitted for consideration for alarm to identify the offending object.
Procedure to identify the offending object.
1) Review the current FMS log such as $FGLHOME/logs/ManagementServer_2019-02-08_093729_001.log
2) Search for messages similar to the following: You may find these repeating each hour. This example shows the object time stamp is 1hr 8min older than the FMS time.
WARN [Data-4-thread-12141] com.quest.nitro.service.data.DataService - Discarded value of observation "failedCollections" on topology object "collect (FglamAgentCollector)" (8434e01c-8054-4681-8617-e55fecc5efc4) because its timestamp is very different from that of the management server (5/31/19 7:24 AM vs 5/31/19 8:32 AM). Please check the server's store and forward configuration and verify the system time of the originating host.]
3A) Look up the objects directly using the Object ID in the Administration | Tooling | Script Console using the input in the lower panel. If it's a FglamAgentCollector object, then clicking on the agent property value should show the properties for the actual agent submitting older metrics as in the example below.
3B) Alternatively, instead of looking for each Object ID individually, use the script findDiscardingAgents.groovy (attached) to get Agent Name, Type and FglAM. Make sure to replace the values in the ids list.
