Thread dumps collected when duplicating the problem indicated that the Users tab was stuck when querying groups:
There were many (2413) external groups within Foglight and this was causing slow security service group queries.
| SEC_GROUP | 2,413 | 524,288 | 458,752 |
| SEC_USER | 71 | 65,536 | 131,072 |
| SEC_USERGROUP | 7,461 | 786,432 | 3,735,552 |
Many of the external groups did not have any roles associated with them, so they weren't needed.
Workaround: Run the delete-external-groups-without-roles.groovy script to clean up groups that do not have any assigned roles.
When the script runs it can cause the response of the Foglight console to be very slow. It is best to run the script after business hours when the number of users is expected to be low.
Following are the steps to run the script from the script console:
1). Navigate to "Administration | Tooling | Script Console".
2). Select the Scripts tab, click the "Add" button, and then paste the contents of the script in the "Enter Script Text" field.
3). Click the "Run" button at the bottom of the "Run Script" window.
We have noted duration of close to one hour to run script to completion.
You can monitor the value of SEC_GROUP via the jmx-console type=DatabaseSpace mbean.
Click DiagnosticSnapshotasString 'Invoke' button. Repeat periodically to see the value reduce.