Hit search (other dashboards as well ) find no results in APM (4291264)

Hit search (other dashboards as well ) find no results in APM

In APM Hit search ( other dashboards as well ) did not show any new results  even though we could see traffic coming through the monitoring port and traffic coming through when we run Server auto-discovery. So the traffic is being seen by the Sniffer but never making it to the Archiver.

(In addition to the above and the logs below, at least sometimes this problem also manifests as the shared memory segment of the Relayer--the input queue to the Archiver. This is different from the Java Heap. When the 'Shared Memory' goes to 100%,  it indicates that Archiver is not processing the data. In the 'Appliance Health' for the Relayer, we get an alarm message for the Relayer of  "the number of hits discarded percent for Transfer Queue for 111.111.111.112 on 111.111.111.111 is High. Hits discarded percent is at 100.00%". Also, we see that the 'Messages Transferred' is 0. In the Summary graph for the Relayer, we saw that the 'Discarded Messages' shoot up from zero a while back and has stayed high.)

env: FMS( 5.6.10) not appliance based + sniffer appliance + archiver appliance (APM version 5.9.3)

In archiver log:

2014-03-07 13:38:37,015 ERROR [com.quest.eu.archiver.core.ShardWriter] Unexpected error when flushing, id=862
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:38:37,015 ERROR [com.quest.eu.archiver.core.WriteSessionCommand] Unexpected error writing to shard862t
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:38:37,018 ERROR [com.quest.eu.archiver.core.WriteSessionCommand] Unexpected error writing to shard862t
java.lang.NullPointerException

2014-03-07 13:39:09,858 ERROR [com.quest.eu.lucene.VisibleTextAnalyzer] Error create Source object from content
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:39:38,517 ERROR [com.quest.eu.lucene.VisibleTextAnalyzer] Error create Source object from content
java.lang.OutOfMemoryError: Java heap space
2014-03-07 13:39:53,071 ERROR [com.quest.eu.archiver.core.query.QueryHitDetailsCommand] Unexpected error
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-03-07 13:39:53,071 WARN  [com.quest.eu.archiver.core.query.QueryHitDetailsCommand] ERROR: Internal Error

In relayer log:

Mar  7 13:45:00 cxxxfl1 relayer: (6) UploadMgr --> HitsCaptured:0, PagesCaptured:0, Discarded:0, QFullHits:0, QFullPages:0, ShmFull:0, LadTO:1422, MaxSize:0, CapUplErrs:0

The archiver seems to be bumping up against its heap limit

Archiver java heap can run out of memory on a R720 Archiver-only appliance--even on a "large" R720 with 128 GB of memory. We believe this is caused by having a slightly higher than average heap usage and, when coupled with multiple hit detail queries running, you may run short on heap space. This does _not_ appear to be a memory leak. Tracking number APMDATA-1539 logged to increase the Archiver heap size on Archiver-only appliances

WORKAROUND

Restart your Archiver machine

STATUS

Fixed in patch version 5.9.3_20140324_40. You can download the patch from here:
https://support.quest.com/foglight/kb/122772

 

APMDATA-1539