Return
-
Title
Unable to Monitor Dell Compellent array due to Getting Current System Storage Usage error -
Description
The environment was very large and there were 32 Storage Centers (SC) monitored by a single Enterprise Manager (EM) host. For this reason, there was a single Storage Collector created to collect topology and performance from the target.Agent Manager log file for the Storage agent had the following error:2016-06-12 15:21:30.515 ERROR [Collections[1]-0] DBLOG - Compellent perf: collection of Storage Center 'SCNAME' has failedcom.compellent.api.communication.CompellentApiMethodException: Error getting system settings from SCNAME2016-06-01 05:49:02.963 ERROR [Collections[1]-0] DBLOG - Compellent perf: collection of Storage Center 'SCNAME' has failedcom.compellent.api.communication.CompellentApiMethodException: Error Getting Current Volume Storage Usage [Storage Center: 11111] : Error getting " Pagepool " from SC [SN: 11111 ] [Name: Storage Center ] (" ")Moreover, EM host was showing native intermittent disconnection errors to the SCs. These disconnection errors were exist when no StorageCollector agent connected to the EM host.Last but not least, all the EM API calls Storage make are sequential. There are no concurrent calls made to Enterprise Manager. -
Cause
EM performance characteristics seemed to be exceeded with the amount of SC's were currently being monitored and SSL threads were being blocked natively. -
Resolution
STATUSBug 52352 has been submitted to Dell Compellent Team.Enable EM Debug logging and deactive the Storage Collector Agent to see the native blocked SSL threads error:...\logs\serverlogs\server_error.log2016-08-14 05:30:50,928 ERROR [org.apache.catalina.connector] (https-pool-threads - 635398) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.IllegalStateException: JBAS018053: No security context foundat org.jboss.as.web.security.SecurityActions$6.run(SecurityActions.java:136) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]at org.jboss.as.web.security.SecurityActions$6.run(SecurityActions.java:130) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_67]at org.jboss.as.web.security.SecurityActions.popRunAsIdentity(SecurityActions.java:130) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]\logs\jvmreports\JvmReport.2016.08.16.02.52.40.864.CDT.xml<thread><group>ServerService ThreadGroup</group><name>pool-27-thread-138188</name><id>15872881</id><daemon>false</daemon><alive>true</alive><state>BLOCKED</state><implType>java.lang.Thread</implType><cpuTime>843</cpuTime><blockedCount>19</blockedCount><blockedTime>87</blockedTime><stackTrace><element>sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:287)</element><element>sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)</element><element>sun.security.ssl.SSLSocketImpl.[init](SSLSocketImpl.java:512)</element><element>sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:73)</element><element>sun.net.www.protocol.https.HttpsClient.createSocket(HttpsClient.java:409)</element><element>sun.net.NetworkClient.doConnect(NetworkClient.java:162)</element><element>sun.net.www.http.HttpClient.openServer(HttpClient.java:432)</element><element>sun.net.www.http.HttpClient.openServer(HttpClient.java:527)</element><element>sun.net.www.protocol.https.HttpsClient.[init](HttpsClient.java:275)</element><element>sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:371)</element><element>sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)</element><element>sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)</element><element>sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)</element><element>sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)</element>WORKAROUND 1- Create another instance of EM host and move one SC at a time to the new EM instance while NO Storage Collector agent is connected to the EM host and monitor for blocked threads.
- Once verified there is no blocked threads happening; delete the current Storage Collector Agent
- Create new Storage Collector Agents with different nameand point it to the new EM host
WORKAROUND 2Based on the amount of the SC's are being monitored such as 32 SCs- Create 3 additional EM hosts and monitor 8 SCs at each EM host.
- Once verified there is no blocked threads happening; delete the current Storage Collector Agent
- Create 4 Storage Collector Agents with different name and point it to the each EM host
-
Additional Information
This problem may be seen when monitoring large systems. Enterprise Manager 2015 R3 [Build: 15.3.1.300]