In APM Hit search ( other dashboards as well ) did not show any new results even though we could see traffic coming through the monitoring port and traffic coming through when we run Server auto-discovery. So the traffic is being seen by the Sniffer but never making it to the Archiver.
(In addition to the above and the logs below, at least sometimes this problem also manifests as the shared memory segment of the Relayer--the input queue to the Archiver. This is different from the Java Heap. When the 'Shared Memory' goes to 100%, it indicates that Archiver is not processing the data. In the 'Appliance Health' for the Relayer, we get an alarm message for the Relayer of "the number of hits discarded percent for Transfer Queue for 111.111.111.112 on 111.111.111.111 is High. Hits discarded percent is at 100.00%". Also, we see that the 'Messages Transferred' is 0. In the Summary graph for the Relayer, we saw that the 'Discarded Messages' shoot up from zero a while back and has stayed high.)
env: FMS( 5.6.10) not appliance based + sniffer appliance + archiver appliance (APM version 5.9.3)
In archiver log:
2014-03-07 13:38:37,015 ERROR [com.quest.eu.archiver.core.ShardWriter] Unexpected error when flushing, id=862
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:38:37,015 ERROR [com.quest.eu.archiver.core.WriteSessionCommand] Unexpected error writing to shard862t
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:38:37,018 ERROR [com.quest.eu.archiver.core.WriteSessionCommand] Unexpected error writing to shard862t
java.lang.NullPointerException
2014-03-07 13:39:09,858 ERROR [com.quest.eu.lucene.VisibleTextAnalyzer] Error create Source object from content
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:39:38,517 ERROR [com.quest.eu.lucene.VisibleTextAnalyzer] Error create Source object from content
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:39:53,071 ERROR [com.quest.eu.archiver.core.query.QueryHitDetailsCommand] Unexpected error
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-03-07 13:39:53,071 WARN [com.quest.eu.archiver.core.query.QueryHitDetailsCommand] ERROR: Internal Error
In relayer log:
Mar 7 13:45:00 cxxxfl1 relayer: (6) UploadMgr --> HitsCaptured:0, PagesCaptured:0, Discarded:0, QFullHits:0, QFullPages:0, ShmFull:0, LadTO:1422, MaxSize:0, CapUplErrs:0
The archiver seems to be bumping up against its heap limit
WORKAROUND
Restart your Archiver machine
STATUS
Fixed in patch version 5.9.3_20140324_40. You can download the patch from here:
https://support.quest.com/foglight/kb/122772
© 2025 Quest Software Inc. ALL RIGHTS RESERVED. Terms of Use Privacy Cookie Preference Center