A popup with the message "Failed to invoke agent callback" appears when adding a new SQL Server database agent.
The SQL Server database agent installer has calls to some code that invokes the installer agent several times and then errors out due to performance issues.
Restart the FMS and the Agent Manager.
Review the sizing of the number of agents on the FMS, particularly the FglAM involved and the total sizing if DB2 agents if they are present on the system.
General FMS performance issues caused by churn. Knowledgebase article 4353149 details how to identify topology churn in the FMS bundle's Diagnostic Snapshot file.