Configuring the database agent On-demand Data Port (4308670)

The RMI port is open in both directions in the firewall, but t he RMI, on-demand, dashboards display the following error:

On-demand data retrieval failed. To check connection configuration, click here (more details).

This error is related to the Foglight Agent Manager (FGLAM) and Foglight Management Server (FMS) bi-directional connection on RMI port or FGLAM hostname.

A second more detailed popup provides additional information

Failed to connect to RMI server with host [HOSTNAME], port [XXXX] and server name [oracle_rmi_server]. Reason: Unknown host: HOSTNAMEl nested exception is java.rmi.UnknownHostException: Unknown host: HOSTNAME; nested exception is: java.net.UnknownHostException: HOSTNAME

RMI uses a client-server architecture. The RMI server is on the FglAM itself where the RMI client resides on the FMS.
In some cases this connection may fail due to a blocked port or if the FglAM points to the wrong RMI client machine.

NOTE: default RMI port for Oracle is 3723, default RMI port for SQL Server is 3235, default RMI port for DB2 is 3725

To configure a port for the collection of on-demand data:

1. Select one or more database agents of the same type from the Databases Dashboard
2. Click the Settings button and then Administration
3. Select the "On-demand Data Settings" menu option
4. Select the check box near the names of the requested Foglight Agent Managers.
5. Click Set port.
   The Set On-demand Data Port dialog box opens.
6. Type the name of the requested port.
7. Click Set.
8. Click Validate connectivity.
   The Validate connectivity progress bar appears. At the end of the validation process, the connection status appears on the Status column.
   If the connection failed, take the requested correction measures and try again

IMPORTANT: Changes to the On-demand Data Port view take effect immediately and do not need to be saved.

This should show that the connection succeeded. If there is an error verifying the connection or if the RMI dashboards still show "on-demand data retrieval failed", continue to the section below.

Check the hostname in the error message

In the management logs you can see what the error is when the RMI connection is trying to get established.
If it is referring to the wrong FglAM hostname then you can explicitly set the host on the FglAM for RMI as follows:

1. Stop the FglAM process
2. Edit the vm.config file or baseline.jvmargs.config (which ever is present) in \state\default\config\
3. Under VM Configuration set 
   vmparameter.0 = "-Dquest.host.name=fglamhostname_or_ip"
   (use the next available number, for example vmparameter.1 or vmparameter.2 if there already is a vmparameter.0 line present in the file)
4. Please replace the with your FglAM hostname.
5. Start the FglAM
6. Try verifying the connection again as mentioned previously

Using the RMI Connection Test

If there are still connection issues then use the RMI connection test program to manually try creating a connection.

Download the attached RMI Connection Test utility (rmi-test-3.1.0-tool.zip for Windows or rmi-test-ksh-3.1.0.zip for Unix)

1. Unzip this on both your FMS and FglAM systems
2. On the FglAM system, from command line, run the runServer.bat (on Windows) or runServer.ksh (on Unix)
   Where is the port on which you would like to run the test (3333 is the default).
3. When prompted, enter the JAVA_HOME (for example, \jre\\jre
4. Once this is running go to the FMS machine and run the runClient.bat (on Windows) or runClient.ksh (on Unix)
   Replace the with the hostname of the fglam system
   Replace the with the port set in step 2.

The test should establish a connection over that port between the two systems.

If an error occurs it will print out details of the connection error.
At this point you should contact Quest Support so it can be investigated further.

Resolution 2:

1. Get "netstat –a > desktop\netstat.txt" from the FglAM hostname and another one from the FMS hostname.
2. Get output of all the ports on the FMS and FglAM host, then search on the both txt files for the RMI port (this is the port that appears under On-demand data port window within Administration agent)
3. Go to "Find | Search" on both notepad files and search for the RMI port based on the following:
   FMS: FMS_Home/config/server.config
   server.jndi.rmi.port = "1098";
   
   The RMI port is 1098 which needs to be open bi-directional between the FMS and FglAM host. The network administrator should open the RMI port if it is close on the FMS or FGLAM host.
   
   If the RMI port is changed, this will require to reboot the FMS.