Incorrect application state in weblogic. (4284108)

Weblogic applications are incorrectly reported down by foglight. On the deployments page in Weblogic, the Status of the applications is Active.

We find this as a bug in WebLogic which is triggered when the foglight agent queries the mbean server.

The foglight agent collects its metrics by querying the WebLogic MBean server. The problem occurs inside WebLogic's MBeanServer implementation and it seems to be related to how WebLogic makes MBean data from all the appservers in the domain accessible through one appserver's MBeanServer. When a query is made to the MBean server, it would see if the object have been registered and scoped earlier. If not it will create(registers) a local copy and caches it. WeakHashMaps are being used to map the local and scoped objects. When the JVM decides to do a GC, it would cause the entries in the weakhashmaps to be dropped. This means that the object has already been registered, but is missing locally.

When Foglight agent does the query later, weblogic mbean server implementation will check its local objects, will not find the object and attempts to register the object which in reality is already been registered. And this operation causes weblogic to throw the AssertionError that is caught by the Foglight agent and it is shutdown. Agent shutdown means there will be no data collected and hence the false alerts they saw in the Foglight console.

1. Restart the adminServer whenever it encounters the error and the agent shutsdown with a FATAL error.
OR
2. Upgrade to more stable WebLogic 10.3

This is happening on Weblogic 9.2.3