APM End User Recording an Abnormally High amount of 599 Errors (4291139)

APM End User Recording an Abnormally High amount of 599 Errors

APM has seen a large amount of 599 errors and page with a 599 is showing abnormally long end-to-end times, usually > 2 minutes.

Http 599 errors refer to hits without any response. And it is most probably contributed by external connection resets and external timeouts. Looking at archiver health log in the bundle, it is seen around 25% hits with 599 errors and it appeared the main cause in their case was from connection reset for some reasons. The following is a 5min sample data.

ARCHIVER_HITS_PROCESSED||||||||||:1899.0||||

ARCHIVER_HITS_MISSING_RESPONSE||||||||||:492.0||||

ARCHIVER_HITS_CLIENT_TIMEOUT||||||||||:3.0||||
ARCHIVER_HITS_SERVER_TIMEOUT||||||||||:1.0||||
ARCHIVER_HITS_SERVER_CONNECTION_RESET||||||||||:220.0||||
ARCHIVER_HITS_CLIENT_CONNECTION_RESET||||||||||:256.0||||

By analyzing their traffic trace, it turned out that most of the 599 errors were from their F5s doing some sort of strange keep alive.It pings each second and causes errors in the web and application servers. We looked at the time stamp and saw the same request being sent from the LB each second.  And each second it went to a different web server so we believe this is why we were seeing connection resets.