ERROR: "SSL handshake failed: Unsupported curveId: 29" when adding a SQL Server instance (4229724)

ERROR: "SSL handshake failed: Unsupported curveId: 29" when adding a SQL Server instance

The following error message is seen when adding a SQL Server instance to monitoring or testing the connection.

"[[Foglight][SQLServer JDBC Driver]SSL handshake failed: Unsupported curveId: 29- Profile:MSSQLProfile{host='', instance= '>', username='', authType= 'SQL_SERVER', port= '0' useNTLMv2= 'false', socketTimeout= '900', secureConnection= 'REQUIRE' }]. . [SQL-Server Error Code = 0. SQL state : 08001]"

This happens when the "SSL Connection" is set to "Mandatory" in the monitoring agent and the "Force Encryption" option is set to "Yes" at the instance level.

The target SQL Server instance is running on Windows Server 2016.

The encryption handshake fails for the following reasons:

1. Known Java issues with SSL and encryption handshakes prevent the connection between the database agent and the monitored instance to take place.
2. The client (Foglight Agent Manager host) is disabled to work with TLS 1.2 security protocol.

WORKAROUND

This article is for JDK 1.6 only. For any FglAM whose embedded JDK is higher than 1.6 there is no need to set the vmparameter as described below.

1. Enable the TLS 1.2 security on the client (Foglight Agent Manager host). This will require changes to the Windows Registry. Additional information on the required values is available on the following Microsoft documentation: TLS/SSL Settings.
2. Review the TLS 1.2 configuration on the target host. Registry changes should not be needed for Windows Server 2016 as the protocol is enabled by default.
3. Edit the Foglight Agent Manager configuration to override Java configurations:
   1. Navigate to \state\default\config\
   2. Backup the existing "baseline.jvmargs.config file".
   3. Open the file for editing and add the following line to the bottom of the file
      
      vmparameter.x = "-Dcom.sun.net.ssl.enableECC=false";
      
      NOTE: vmparameter.x stands for the next available vmparameter value; for example, vmparameter.3 = ""
      Do not override existing parameters.
4. Restart the Foglight Agent Manager.
5. Test the agent connection.

Support does not provide support for problems that arise from improper modification of the registry. The Windows registry contains information critical to your computer and applications. Make sure you back up the registry before modifying it. For more information on the Windows Registry Editor and how to back up and restore it, refer to Microsoft Article ID 256986 “Description of the Microsoft Windows registry” at Microsoft Support.

STATUS

Please upgrade the SQL Server cartridge to 5.9.3.10 or higher.

In 5.9.3.10, customer need to manually change the corresponding ASP of the installer agent. 

1. Upgrade the database cartridge to version 5.9.3.10
2. Edit the Agent Status properties or for the installer agent to change the installer agent's SSL Connection" to "Mandatory" and "Enforce SSL Version" to "TLSv1.2" 
3. Create a new SQL Server agent with the wizard
4. Edit the Agent Status properties of the new database agent to change the new DB agent's "SSL Connection" to "Mandatory" and "Enforce SSL Version" to "TLSv1.2" 
5. Deactivate and activate the database agent.

Query from SQL Server Management Studio to see if there are any encrypted connections on the SQL Server as well. If 0 is returned, it is most likely that the setting for “Force Encryption” is “No”. SELECT count (encrypt_option) FROM sys.dm_exec_connections WHERE encrypt_option = 'TRUE' 1. Remote desktop to your SQL Server host 2. Open SQL Server Configuration Manager 3. Expand SQL Server Network Configuration section and right-click on the Protocols for and select Properties 4. Force Encryption is under Flags tab. This will also confirmed if Encryption is on or not. Here is a ink about TLS: https://technet.microsoft.com/en-us/library/dn786418(v=ws.11).aspx#BKMK_SchannelTR_SSL30 Review the TLS 1.2 configuration on the target host. Registry changes should not be needed for Windows Server 2016 as the protocol is enabled by default.