Oracle alarm: DBO Usability OS Connection Availability Failure Due to Cannot Execute Remote Command (4380037)

The "Usability OS Connection Availability" alarm in Foglight for Oracle indicates that the agent failed to execute a remote OS-level command (e.g., checking for the Oracle PMON process or listener status) via SSH on the monitored host. 

An error like the following appears in the Oracle agent log file

WARN    [AGENTNAME-lowPriorityPool-574-[DBO_Usability][]] com.quest.qsi.fason.oracle.collection.processor.DBOListenerControlProcessor - Error occurred when running lsnrctl for listener LISTENER. Submitting the status of the listener process instead.
java.lang.RuntimeException: Failed to run query of "AGENTNAME" instance. Query [
${LOCALE} $ORACLE_HOME_VAR $TNS_ADMIN_VAR $LD_LIBRARY_PATH_VAR $SHLIB_PATH_VAR ${sudo} $oracle_home/bin/lsnrctl status $listener_name]. Params [{LD_LIBRARY_PATH_VAR=export LD_LIBRARY_PATH=$ORACLE_HOME/lib; , listener_name=LISTENER, oracle_home=/apps/oracle/product/19.0.0.0/grid, SHLIB_PATH_VAR=export SHLIB_PATH=$ORACLE_HOME/lib; , ORACLE_HOME_VAR=export ORACLE_HOME=/apps/oracle/product/19.0.0.0/grid; , TNS_ADMIN_VAR=}]
Caused by: com.quest.qsi.fason.framework.connections.common.exceptions.FrameworkQueryException: cannot execute remote command

The root cause is typically a failure in the SSH connection between the Foglight Agent Manager (FglAM) and the monitored host.

Specific causes may include:

• SSH daemon (sshd) not running or misconfigured on the target host.
• Network/firewall issues blocking port 22.
• Incorrect or expired credentials.
• Sudo misconfiguration (e.g., requiring a password or interactive shell) for user accounts that are configured to run commands using sudo. This is uncommon in more user environments.
• SSH session limits
  • MaxSessions too low
  • MaxStartups too agressive
• Open file descriptor limits exceeded
  • The monitoring user or system may hit ulimit -n (number of open files), causing SSH or command execution to fail

 There is no specific single cause to this issue. 

Recommended areas to investigate on the host.

1. Verify SSH Access
   • Ensure sshd is running on the monitored host.
   • Confirm that the Foglight agent user can SSH into the host without interactive prompts by logging into the host using ssh manually from the FglAM. 
   • Intermittent issues may point to flaky network conditions or session limits.
2. Check Agent User Privileges
   • The agent user must have permission to run commands like ps, grep, and lsnrctl 
   • If the agent is configured to require sudo, they confirm that it can be run without a password and use sudo -l to verify allowed commands.
3. Review SSH Configuration
   • Ensure the SSH profile in Foglight is correctly configured (non-keyboard-interactive credentials, correct port, etc.).
   • Recommended settings (because low limits can cause random failures under load)

     Maxsessions 50
     
     MaxStartups 20:50:100

   • Restart the SSH service if needed.

     systemctl restart sshd

   • MIsconfigured SSH profiles (e.g. wrong port, authentication method) can cause both persistent and sporadic failures, especially if the environment changes dynamically (e.g. rotating keys, session timeouts)
4. Check Open File Limits
   • Run ulimit -n
   • If the value is too low (e.g. 1024), increase it in /etc/security/limits.conf or systemd service files
5. Validate Agent Manager Configuration
   • Ensure that the correct SSH progile is used and the host is reachable
   • If using a concentrator setup, ensure the downstream Agent Managers are correctly listed and reachable
   • Intermittent issues are less likely unless there's a load-balancing or routing problem.
6. Test Commands Manually
   • Run the failing command manually on the host by using ssh from the Agent Manager to confirm it works:

     ps -ef | grep -iw ora_pmon_<INSTANCE_NAME> | grep -vc grep
     
     $oracle_home/bin/lsnrctl status $listener_name

7. Check Logs
   • Review Foglight logs for ConnectionBrokenException or SshChannelOpenException for more context.